Video generation method and device, equipment and storage medium

Abstract

The invention provides a video generation method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like. The specific implementation scheme is as follows: inputting an initial feature of a target video, a mask pattern of an original video and a multi-view image of a new object into a video generation model to obtain a to-be-processed feature, a modulation parameter and a three-dimensional feature of the new object; generating a prediction result of the target video based on the to-be-processed feature, the modulation parameter and the three-dimensional feature; wherein the prediction result of the target video comprises the video after the original object in the original video is replaced by the new object.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of computer vision, deep learning, and large models. Background Technology

[0002] Object swapping video generation involves replacing objects, such as merchandise, in a video. By replacing objects in a video within a specific scene, the recorded video can be reused multiple times, achieving the effect of recording once and using it multiple times. Typically, a reference image of the new object is provided, and a video generation model is used to complete and replace the object to be generated. Summary of the Invention

[0003] This disclosure provides a video generation method, apparatus, device, and storage medium.

[0004] According to one aspect of this disclosure, a video generation method is provided, comprising: The initial features of the target video, the mask image of the original video, and the multi-view images of the new object are input into the video generation model to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object. Based on the feature to be processed, the modulation parameter, and the three-dimensional feature, a prediction result for the target video is generated; wherein, the prediction result for the target video includes the video after replacing the original object in the original video with the new object.

[0005] According to another aspect of this disclosure, a method for training a video generation model is provided, comprising: Based on the training samples, the initial features of the target video, the mask image of the original video, and the multi-view images of the new object are input into the video generation model to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object; wherein, the training samples include the original video and the multi-view images of the new object. Based on the feature to be processed, the modulation parameter, and the three-dimensional feature, a prediction result of the target video is generated; wherein, the prediction result of the target video includes the video after replacing the original object in the original video with the new object; Based on the prediction result and the loss determined by the initial feature, the video generation model is trained, and the trained video generation model is then trained.

[0006] According to another aspect of this disclosure, a video generation apparatus is provided, comprising: The feature processing module is used to input the initial features of the target video, the mask image of the original video, and the multi-view image of the new object into the video generation model to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object. The prediction module is used to generate a prediction result of the target video based on the feature to be processed, the modulation parameter, and the three-dimensional feature; wherein the prediction result of the target video includes a video after replacing the original object in the original video with the new object.

[0007] According to another aspect of this disclosure, a training apparatus for a video generation model is provided, comprising: The feature processing module is used to input the initial features of the target video, the mask image of the original video, and the multi-view images of the new object into the video generation model based on the training samples to obtain the features to be processed, modulation parameters, and the three-dimensional features of the new object; wherein, the training samples include the original video and the multi-view images of the new object. The prediction module is used to generate a prediction result of the target video based on the feature to be processed, the modulation parameter, and the three-dimensional feature; wherein, the prediction result of the target video includes the video after replacing the original object in the original video with the new object; The training module is used to train the video generation model based on the prediction result and the loss determined by the initial features, and then train the trained video generation model.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1This is a flowchart illustrating a video generation method according to an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a video generation method according to another embodiment of the present disclosure; Figure 3 This is a flowchart illustrating a video generation method according to another embodiment of the present disclosure; Figure 4 This is a flowchart illustrating a video generation method according to another embodiment of the present disclosure; Figure 5 This is a flowchart illustrating a video generation method according to another embodiment of the present disclosure; Figure 6 This is a flowchart illustrating a training method for a video generation model according to an embodiment of the present disclosure; Figure 7 This is a flowchart illustrating a training method for a video generation model according to another embodiment of the present disclosure; Figure 8 This is a flowchart illustrating a training method for a video generation model according to another embodiment of the present disclosure; Figure 9 This is a flowchart illustrating a training method for a video generation model according to another embodiment of the present disclosure; Figure 10 This is a flowchart illustrating a training method for a video generation model according to another embodiment of the present disclosure; Figure 11 This is a system architecture diagram of a video generation method; Figure 12 This is a schematic diagram of the structure of a video generation apparatus according to an embodiment of the present disclosure; Figure 13 This is a schematic diagram of the structure of a video generation apparatus according to another embodiment of the present disclosure; Figure 14 This is a schematic diagram of the structure of a training device for a video generation model according to an embodiment of the present disclosure; Figure 15 This is a schematic diagram of the structure of a training device for a video generation model according to another embodiment of the present disclosure; Figure 16 This is a block diagram of an electronic device used to implement the methods of the embodiments of this disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] Figure 1 This is a flowchart illustrating a video generation method 100 according to an embodiment of the present disclosure, the method comprising: S110. Input the initial features of the target video, the mask image of the original video, and the multi-view image of the new object into the video generation model to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object.

[0015] S120. Based on the feature to be processed, the modulation parameter, and the three-dimensional feature, generate the prediction result of the target video.

[0016] The prediction result of the target video includes the video after replacing the original object in the original video with the new object.

[0017] In this embodiment, the original video can be referred to as the background video. The original video may include the original object to be replaced. For example, the original object may be a product held by a person. Alternatively, the original object may be a static object, animal, or person in the scene. Masking the original object in the original video yields a mask image of the original video. Each time the original video mask image is input into the video generation model, it can be a mask image of one video frame, multiple video frames, or all video frames. The target video can be the new video to be generated. The initial features of the target video may include initial noise. The video generation model can be built based on a diffusion model, and the video generation model can generate the final target video through iterative noise prediction processes such as adding and removing noise from the initial features of the target video. The original object can also be referred to as the object to be replaced, the substitute object, etc., and the new object can also be referred to as the replacement object, the target object, etc. Multi-view images of the new object can include multiple images of the same new object taken or drawn from multiple angles, fully presenting the spatial morphology and structural details of the new object. Multi-view images can be obtained by camera shooting or computer rendering, etc.

[0018] In this embodiment, the initial features of the target video, the mask image of the original video, and multi-view images of the object can be input into the video generation model. The video generation model can obtain features to be processed based on the initial features of the target video. The video generation model can include multiple layers; except for the last layer, the output features of each layer can be used as input features for the next layer, or can be understood as features to be processed in the next layer. The video generation model can obtain modulation parameters based on one or more frames of the mask image of the original video, and then use the modulation parameters to process the features to be processed in the previous layer before injecting them into the attention layer of the video generation model. The video generation model can extract the three-dimensional features of the new object from the multi-view images of the new object and inject the three-dimensional features into the attention layer of the video generation model. Subsequent processing is then performed through the attention layer until a video is generated after replacing the original object in the original video with the new object.

[0019] According to embodiments of this disclosure, a video generation model is used to process the initial features of the target video, the mask image of the original video, and the multi-view images of the new object to generate a new video after replacing the original object in the original video with the new object. This can make the position of the new object in the new video more reasonable and the details more realistic.

[0020] Figure 2 This is a flowchart illustrating a video generation method 200 according to another embodiment of the present disclosure. The method may include one or more features of the above embodiments. In one implementation, method 200 further includes: S210. Extract a video frame from the original video.

[0021] S220. Superimpose the reference image of the new object onto the original object's location in the video frame to obtain the superimposed image.

[0022] S230. Input the superimposed image into the image editing model using the prompt words to obtain the first frame image.

[0023] S240. Based on the first frame image and Gaussian noise, generate the initial features of the target video.

[0024] In one implementation, the initial features of the target video include first-frame features that are encoded based on the first-frame image, and non-first-frame features that include Gaussian noise.

[0025] In this embodiment, a video frame is the smallest independent image unit constituting a video, and the original video includes multiple video frames. To generate the first frame image of the target video, the first video frame can be extracted from the original video, or a specific video frame can be selected from the original video. A reference image of the new object can be superimposed on the original object from the extracted video frame, thus obscuring the original object. Furthermore, the size of the reference image of the new object can be adjusted based on the size of the original object. Alternatively, the size, position, and dimensions of the reference image of the new object can be adjusted first, and then the adjusted reference image of the new object can be superimposed on the extracted video frame. The above superposition and adjustment process can be performed manually or automatically by software. After superposition and adjustment, a superimposed image of a new object of appropriate size and position can be obtained against the background of the original video.

[0026] In this embodiment, the overlay image can be input into the image editing model via prompts, which may include a task description, such as a task description for generating a composite image from the overlay image. After processing by the image editing model, the first frame of the target video can be generated. An example of generating initial features of the target video based on the first frame may include: converting the first frame image into latent space features using a Variational Auto-Encoder (VAE), and extracting semantic and visual detail features of the first frame image using a pre-encoder such as a Contrastive Language-Image Pre-training (CLIP) model. Furthermore, Gaussian noise conforming to the video frame size is generated, and the noise variance is adjusted according to the diffusion time. The noise is then expanded into a noise sequence corresponding to the video duration. The model can also perform latent space transformation on the noise to match its dimension with the latent space features of the first frame. Latent space features can be called frame latents. Latent space features may include visual features such as texture and structure, and may also carry semantic information, serving as intermediate data for generation tasks such as video diffusion models. The initial features of the target video can include the latent space features of the first frame image and the latent space features of the noise sequence. For example, in the initial features of the target video, the first frame features can include the frame latent vector of the first frame image, and subsequent non-first frame features can include the frame latent vectors of Gaussian noise (or Gaussian noise sequence).

[0027] According to the embodiments of this disclosure, the reference image of the new object is superimposed with the video frame extracted from the original video to generate the first frame image. The first frame image can better control the position and size of the new object in the subsequently generated video frames, making the position and size of the new object in the generated video more reasonable, thereby improving the image quality of the video after the object is replaced.

[0028] In one implementation, the video generation model includes multiple diffusion transformer blocks, with at least one three-dimensional diffusion transformer inserted between the multiple diffusion transformer blocks; the output feature of the diffusion transformer block above the three-dimensional diffusion transformer is the feature to be processed input to the three-dimensional diffusion transformer.

[0029] In the embodiments disclosed herein, such as Figure 3 As shown, the video generation model 300 includes multiple diffusion transformer blocks 301. A diffusion transformer block (DiT Block) is a crucial component of the diffusion transformer (DiT), processing image blocks or sequences of image blocks based on a vision transformer (ViT) architecture. Each diffusion transformer block may include basic modules such as an attention layer and a feedforward network. One or more 3D diffusion transformers (3D DiTs) 302 can be inserted between the multiple diffusion transformer blocks of the video generation model. For example, a 3D diffusion transformer can be inserted after each layer of diffusion transformer blocks. Alternatively, a 3D diffusion transformer can be inserted after every two layers of diffusion transformer blocks. The 3D diffusion transformer (3D DiT) is an extension of the diffusion transformer in the 3D domain, adaptable to the processing of 3D data such as voxelized point clouds and 3D meshes, and used for tasks such as 3D shape generation, part completion, and single-image to 3D asset conversion. A 3D diffusion transformer can use the output features of its previous diffusion transformer block as input features to be processed by the 3D diffusion transformer. The output characteristics of a 3D diffusion transformer can be used as the input characteristics of the diffusion transformer block in the next layer.

[0030] In one implementation, the 3D diffusion transformer may include a partial 3D network 3021 and a 3D feature extraction network 3022. The first diffusion transformer block of the video generation model can output features to be processed based on the initial features of the target video. The video generation model may include multiple layers; except for the last layer, the output features of each layer can be used as the input features of the next layer, or can be understood as the features to be processed in the next layer. The 3D diffusion transformer may include a partial 3D network 3021 that can process the modulation parameters based on one or more frames of the original video's mask image. After processing the features to be processed in the previous layer using the modulation parameters, they can be injected into the attention layer of the 3D diffusion transformer. The 3D feature extraction network 3022 of the 3D diffusion transformer can extract the 3D features of the new object from multi-view images of the new object. The 3D features are injected into the attention layer of the video generation model. Then, subsequent processing is performed through the attention layer to obtain the output features of the 3D diffusion transformer. The next layer of the 3D diffusion transformer block can continue processing based on the output features of the previous layer until a video is generated after replacing the original object in the original video with the new object.

[0031] According to embodiments of this disclosure, by inserting a three-dimensional diffusion transformer between multiple diffusion transformer blocks, the mask image of the original video and the multi-view image of the new object can be injected into the model, so that the generated video can fully present the spatial morphology and structural details of the new object.

[0032] Figure 4 This is a flowchart illustrating a video generation method 400 according to another embodiment of the present disclosure. Method 400 can be used to implement step S110 in video generation method 100. In one embodiment, method 400 includes: inputting initial features of the target video, a mask image of the original video, and a multi-view image of a new object into a video generation model to obtain features to be processed, modulation parameters, and three-dimensional features of the new object; further including: S410. Input the initial features of the target video into the diffusion converter block of the video generation model to obtain the features to be processed; S420. Input the mask image of the original video into the three-dimensional diffusion transformer of the video generation model to obtain the modulation parameters; wherein, the mask image of the original video is the image obtained by masking the target area where the original object is located in the video frame of the original video. S430. Input the multi-view image of the new object into the three-dimensional diffusion transformer of the video generation model to obtain the three-dimensional features of the new object.

[0033] In this embodiment, the initial features of the target video are input to the diffusion transformer block of the video generation model, and then passed sequentially by the diffusion transformer block. One or more frames of the original video's mask image are input to a portion of the 3D network of the 3D diffusion transformer to obtain modulation parameters. There can be one or more modulation parameters. Multi-view images of the new object are input to the 3D feature extraction network of the 3D diffusion transformer to extract the 3D features of the new object. For example, these 3D features include shape information and texture information.

[0034] Assume the video generation model includes diffusion transformer blocks at layers 1 and 2, a 3D diffusion transformer at layer 3, diffusion transformer blocks at layers 4 and 5, and a 6th layer 3D diffusion transformer. The output features obtained from the initial feature processing of the target video by the layer 1 diffusion transformer block are input to the layer 2 diffusion transformer block. The output features obtained from processing the input features by the layer 2 diffusion transformer block are input to the 3D diffusion transformer at layer 3 as the features to be processed in that layer. The partial 3D network of the 3D diffusion transformer at layer 3, based on the mask image of one or more frames of the original video, can obtain modulation parameters. The features to be processed in this layer are modulated based on these parameters and then injected into the attention layer of the 3D diffusion transformer. The 3D feature extraction network of the 3D diffusion transformer at layer 3 extracts the 3D features of the new object based on its multi-view (MV) image and then injects them into the attention layer of the 3D diffusion transformer. The output features of the 3D diffusion transformer at layer 3 are input to the layer 4 diffusion transformer block. The output features obtained from processing the input features by the layer 4 diffusion transformer block are input to the layer 5 diffusion transformer block. The fifth-layer diffusion transformer block processes the input features, and the resulting output features are input to the sixth-layer three-dimensional diffusion transformer as the features to be processed in that layer. The working principle of the sixth-layer three-dimensional diffusion transformer can be found in the third-layer three-dimensional diffusion transformer, and will not be repeated here.

[0035] 3D features are representations of the geometry, topology, and semantic attributes of 3D data (such as meshes, voxels, and 3D models) of objects. They form the basis for tasks such as 3D reconstruction, object recognition, segmentation, and registration. 3D features can include shape and texture information. Shape information describes the geometry and spatial structure of a 3D object, determining its outline, volume, and topological relationships, and is independent of color and material. Texture information describes the color, pattern, and material details attached to the surface of a 3D object, determining its visual appearance and depending on the surface topology of its shape.

[0036] According to embodiments of this disclosure, by generating modulation parameters and extracting the three-dimensional features of the object through the three-dimensional diffusion transformer of the video generation model, the shape and texture consistency of the new object can be maintained, and the fine texture and complex shape of the new object can be more accurately restored. The mask image based on the original video is conducive to obtaining the motion information of the object, making the motion of the new object natural and smooth, and improving the realism of the generated video.

[0037] Figure 5This is a flowchart illustrating a video generation method 500 according to another embodiment of the present disclosure. Method 500 can be used to implement step S120 in video generation method 100. In one embodiment, method 500 includes: generating a prediction result of the target video based on the feature to be processed, the modulation parameters, and the three-dimensional feature, further including: S510. Based on the features to be processed output by the diffusion converter block of the upper layer of the three-dimensional diffusion converter and the modulation parameters of the three-dimensional diffusion converter, features with motion information are obtained. S520. Based on the feature with motion information, the Q matrix of the attention layer is obtained; S530. Based on the three-dimensional features of the new object, obtain the K matrix and V matrix of the attention layer; S540. Based on the Q matrix, K matrix, and V matrix of the attention layer, the output features of the three-dimensional diffusion transformer are obtained; wherein, the output features of the three-dimensional diffusion transformer are the unprocessed features of the diffusion transformer block of the next layer input to the three-dimensional diffusion transformer. S550. Based on the output features of the last layer of the video generation model, generate the prediction result of the target video.

[0038] In this embodiment, modulation parameters may include parameters used to dynamically adjust model feature output, constrain generation logic, and implement conditional control. These parameters precisely control the intensity, distribution, and correlation of features during model operation, ensuring the generated results match specific task requirements (such as co-generation of shape and texture, and ensuring consistency between video frames). Modulation parameters are used to adjust the features to be processed, adding the motion information of the original object in the original video to the features output from the previous layer, thus obtaining features with motion information. These motion-information features are then converted into the Q-matrix of the attention layer of the 3D diffusion transformer. The 3D features of the new object are then converted into the K-matrix and V-matrix of the attention layer of the 3D diffusion transformer. The attention layer performs subsequent processing based on the Q-matrix, K-matrix, and V-matrix, and the output features can be used as input features for the next layer of the diffusion transformer block in this 3D diffusion transformer.

[0039] In this embodiment of the disclosure, the initial features of the target video can be sequentially passed through all layers of the video generation model until the last layer of the video generation model is used to obtain the output features, thereby generating the prediction result of the target video.

[0040] According to embodiments of this disclosure, by alternately processing features through multi-layer diffusion transformer blocks and three-dimensional diffusion transformers, a target video can be generated, which can support the generation of fine textures and complex structures, resulting in higher video detail reproduction and thus improving the quality of video generation.

[0041] Figure 6 This is a flowchart illustrating a training method 600 for a video generation model according to an embodiment of the present disclosure. The method includes: S610. Based on the training samples, input the initial features of the target video, the mask image of the original video, and the multi-view images of the new object into the video generation model to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object; wherein, the training samples include the original video and the multi-view images of the new object. S620. Based on the feature to be processed, the modulation parameter, and the three-dimensional feature, generate a prediction result for the target video; wherein, the prediction result for the target video includes the video after replacing the original object in the original video with the new object; S630. Based on the prediction result and the loss determined by the initial feature, the video generation model is trained to obtain the trained video generation model.

[0042] In this embodiment, the training set may include multiple training samples. The training samples of the video generation model may include multi-view images of the original video and the new object. The training samples may also include masked images of each video frame in the original video after masking processing. The training samples may also include the first frame image generated based on a certain frame image of the original video. During training, the structure and processing principle of the model are similar to those in the video generation method of the above embodiments, and can be found in the relevant descriptions of the above embodiments. Based on the prediction results of the model and the initial features of the target video input to the model, the model loss can be calculated. If the loss function fails to converge based on the loss, the model parameters can be adjusted and the next round of training can continue. If the loss function converges based on the loss and / or the number of training iterations meets the requirements, training can be stopped, and the trained video generation model can be obtained.

[0043] According to embodiments of this disclosure, a video generation model is used to process the initial features of the target video, the mask image of the original video, and the multi-view images of the new object to generate a new video after replacing the original object in the original video with the new object. This can make the position of the new object in the new video more reasonable and the details more realistic.

[0044] Figure 7 This is a flowchart illustrating a training method 700 for a video generation model according to another embodiment of the present disclosure. Method 700 can be used to implement the aforementioned training method for the video generation model. In one embodiment, method 700 further includes: S710. Extract a video frame from the original video; S720. The reference image of the new object is superimposed onto the position of the original object in the video frame to obtain the superimposed image; S730. Input the superimposed image into the image editing model using the prompt words to obtain the first frame image; S740. Based on the first frame image and Gaussian noise, generate initial features of the target video; wherein, the initial features of the target video include features encoded based on the first frame image, and features other than the first frame include Gaussian noise.

[0045] In this embodiment of the disclosure, the specific process of the video generation model generating the first frame of video and generating the initial characteristics of the target video based on the first frame of video can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0046] According to the embodiments of this disclosure, the reference image of the new object is superimposed with the video frame extracted from the original video to generate the first frame image. The first frame image can better control the position and size of the new object in the subsequently generated video frames. The position and size of the new object in the video generated by the trained model are more reasonable, thereby improving the image quality of the video after the object is replaced.

[0047] In one embodiment, the video generation model includes multiple diffusion transformer blocks, with at least one 3D diffusion transformer inserted between these blocks; the output feature of the diffusion transformer block above the 3D diffusion transformer is the feature to be processed input to the 3D diffusion transformer. For the specific structure of the video generation model and the working principle of each part, please refer to [link to relevant documentation]. Figure 3 The details and related descriptions will not be repeated here.

[0048] According to embodiments of this disclosure, by inserting a three-dimensional diffusion transformer between multiple diffusion transformer blocks, the mask image of the original video and the multi-view image of the new object can be injected into the model. The video generated by the trained model can fully present the spatial morphology and structural details of the new object.

[0049] Figure 8 This is a flowchart illustrating a training method 800 for a video generation model according to another embodiment of the present disclosure. Method 800 can be used to implement step S610 in the training method 600 for the video generation model. In one embodiment, method 800 includes: inputting initial features of the target video, a mask image of the original video, and a multi-view image of a new object into a video generation model to obtain features to be processed, modulation parameters, and three-dimensional features of the new object; further including: S810. Input the initial features of the target video into the diffusion transform block of the video generation model to obtain the features to be processed; S820. Input the mask image of the original video into the three-dimensional diffusion transformer of the video generation model to obtain the modulation parameters; wherein, the mask image of the original video is the image obtained by masking the target area where the original object is located in the video frame of the original video. S830. Input the multi-view image of the new object into the three-dimensional diffusion transformer of the video generation model to obtain the three-dimensional features of the new object; wherein, the three-dimensional features include shape information and texture information.

[0050] In the embodiments of this disclosure, the specific functions and examples of the diffusion transformer block and the three-dimensional diffusion transformer of the video generation model can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0051] According to embodiments of this disclosure, by generating modulation parameters and extracting the three-dimensional features of the object through the three-dimensional diffusion transformer of the video generation model, the shape and texture consistency of the new object can be maintained, and the fine texture and complex shape of the new object can be more accurately restored. The mask image based on the original video is conducive to obtaining the motion information of the object, making the motion of the new object natural and smooth, and improving the realism of the generated video.

[0052] Figure 9 This is a flowchart illustrating a training method 900 for a video generation model according to another embodiment of the present disclosure. Method 900 can be used to implement step S620 in the training method 600 for the video generation model. In one embodiment, method 900 includes: generating a prediction result of the target video based on the feature to be processed, the modulation parameters, and the three-dimensional features, further including: S910. Based on the feature to be processed output by the diffusion converter block of the upper layer of the three-dimensional diffusion converter and the modulation parameters of the three-dimensional diffusion converter, a feature with motion information is obtained. S920. Based on the feature with motion information, the Q matrix of the attention layer is obtained; S930. Based on the three-dimensional features of the new object, obtain the K matrix and V matrix of the attention layer; S940. Based on the Q matrix, K matrix, and V matrix of the attention layer, the output features of the three-dimensional diffusion transformer are obtained; wherein, the output features of the three-dimensional diffusion transformer are the unprocessed features of the diffusion transformer block of the next layer input to the three-dimensional diffusion transformer. S950. Based on the output features of the last layer of the video generation model, generate the prediction result of the target video.

[0053] In this embodiment of the disclosure, the specific processing process of the three-dimensional diffusion transformer on the features to be processed output by the diffusion transformer block of the previous layer, the mask map of the original video, and the multi-view image of the new object, as well as the generation process of the Q matrix, K matrix and V matrix of the attention layer of the three-dimensional diffusion transformer, can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0054] According to embodiments of this disclosure, target videos are generated by alternating processing of features through multi-layer diffusion transformer blocks and three-dimensional diffusion transformers. This can support the generation of fine textures and complex structures, and the videos generated by the trained model have higher detail reproduction, thereby improving the quality of the videos generated by the model.

[0055] Figure 10 This is a flowchart illustrating a training method 1000 for a video generation model according to another embodiment of the present disclosure. Method 1000 can be used to implement step S630 in the training method 600 for the video generation model. In one embodiment, method 1000 includes: training the video generation model based on the prediction result and a loss determined by the initial features, and further including: S1010. Determine the loss based on the prediction noise in the prediction result and the Gaussian noise in the initial feature; S1020. Adjust the model parameters of the video generation model based on this loss; S1030. Under the condition that the training stops, the trained video generation model is obtained.

[0056] In the embodiments disclosed herein, the specific functions and examples of loss and Gaussian noise can be described in the relevant descriptions in the above embodiments, and will not be repeated here.

[0057] According to embodiments of this disclosure, the model can output predicted noise, or it can calculate predicted noise based on a new video (where the original object in the original video is replaced with a new object) and the original video. Noise loss can be calculated using the predicted noise and Gaussian noise with added initial features. For example, the mean squared error loss can be calculated using the predicted noise and Gaussian noise. If the predicted noise is changed to a predicted noise sequence, and the Gaussian noise with added initial features is changed to a Gaussian noise sequence, the noise loss can be calculated based on the predicted noise sequence and the Gaussian noise sequence. If the loss function fails to converge based on the noise loss, the model parameters can be adjusted, and the next round of training can continue. If the loss function converges based on the loss and / or the number of training iterations meets the requirement, the training stopping condition can be considered met, training can be stopped, and the trained video generation model is obtained.

[0058] In some applications, image inpainting technology is used to generate swapping videos. This allows video models to replace objects held by people in the video. However, the size of the replaced object is difficult to control, and the interaction with the hand is not natural enough. In addition, if there are no shape constraints for the new object, the object will be distorted when it rotates during video generation.

[0059] This disclosure embodiment can leverage the image editing capabilities of a large model to edit the first frame image and feed it into the video model, making the generated video model more realistic in terms of object size and location, such as the position of a person's hand. The 3D reconstructed large model can extract shape information based on the input multi-view images and inject it into the video generation model to enhance the consistency of the object's shape when the object rotates.

[0060] See Figure 11 The system architecture diagram is shown below. The steps to replace objects in the video are as follows: 1. Using an image editing large model, the first frame of the original video (the baseboard) is input, along with a resized and placed reference image of an object (e.g., a product). A prompt is then input into the model to generate the first frame image. Generating the first frame image allows for greater control over the size of new objects and provides strong control over video generation, enabling the video to follow the information in the first frame. Based on the first frame image and Gaussian noise, frame latent vectors, or latent space features, can be generated.

[0061] 2. Unlike directly extracting image features and injecting them into the video model, multi-view images can provide shape and texture information of objects. Directly extracting image features makes it difficult to extract shape information from the video model. Therefore, a large 3D reconstructed model (which can be simply called a 3D model) is used to extract shape information from multi-view images and then inject it into the video generation model. For example... Figure 11 As shown, 3D DiT modules can be inserted between diffusion transformer blocks in the video generation model to inject information. For example, in a scene where a person is holding an object in the video, the input information of the 3D DiT module includes: x (the output of the previous block); masked control frame latent (e.g., the frame latent vector after the original video mask of the hand area); and MV (multi-view images of the new object).

[0062] The MV can be input into the 3D reconstruction model of the 3D DiT module, such as the Visual Geometry Grounded Transformer (VGGT), to obtain the object's 3D features, such as shape and texture. To obtain the rotational relationships of the object in the original video frame, motion information of the object can be provided using the hand region in the original video. This information is input into the partial 3D model of the 3D DiT module to align with the 3D feature, obtaining modulation signals α and β. The output x of the previous layer is then modulated using α and β to impart motion information. The partial 3D model can possess some functions of the 3D reconstruction model, such as feature extraction. Then, the query (Q) matrix of the attention layer is obtained using the modulated information, and cross-attention operations are performed using the key (K) and value (V) matrices of the attention layer obtained from the 3D feature, completing the information injection. After multi-layer processing by the 3D reconstruction model, the object-swapped video is output.

[0063] Figure 12 This is a schematic diagram of the structure of a video generation apparatus 1200 according to an embodiment of the present disclosure. The apparatus 1200 may include: The feature processing module 1210 is used to input the initial features of the target video, the mask image of the original video, and the multi-view image of the new object into the video generation model to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object. The prediction module 1220 is used to generate a prediction result of the target video based on the feature to be processed, the modulation parameter and the three-dimensional feature; wherein the prediction result of the target video includes a video after replacing the original object in the original video with the new object.

[0064] Figure 13 This is a schematic diagram of the structure of a video generation apparatus 1300 according to another embodiment of the present disclosure. The apparatus 1300 includes a feature processing module 1310 and a prediction module 1320. The functions of the above modules are the same as those of the modules in the video generation apparatus of the above embodiment. In one embodiment, it further includes: Extraction module 1330 is used to extract a video frame from the original video; The overlay module 1340 is used to overlay the reference image of the new object onto the position of the original object in the video frame to obtain an overlay image; Input module 1350 is used to input the superimposed image into the image editing model via prompt words to obtain the first frame image; The generation module 1360 is used to generate initial features of the target video based on the first frame image and Gaussian noise; wherein, the initial features of the target video include features encoded based on the first frame image, and features other than the first frame include Gaussian noise.

[0065] In one implementation, the video generation model includes multiple diffusion transformer blocks, with at least one three-dimensional diffusion transformer inserted between the multiple diffusion transformer blocks; the output feature of the diffusion transformer block above the three-dimensional diffusion transformer is the feature to be processed input to the three-dimensional diffusion transformer.

[0066] In one embodiment, the feature processing module 1310 is used to input the initial features of the target video into the diffusion transformer block of the video generation model to obtain the features to be processed; input the mask image of the original video into the three-dimensional diffusion transformer of the video generation model to obtain modulation parameters; wherein, the mask image of the original video is an image obtained by masking the target area where the original object is located in the video frame of the original video; input the multi-view image of the new object into the three-dimensional diffusion transformer of the video generation model to obtain the three-dimensional features of the new object; wherein, the three-dimensional features include shape information and texture information.

[0067] In one embodiment, the prediction module 1220 is used to obtain features with motion information based on the features to be processed output by the diffusion transformer block of the next layer of the 3D diffusion transformer and the modulation parameters of the 3D diffusion transformer; to obtain the Q matrix of the attention layer based on the features with motion information; to obtain the K matrix and V matrix of the attention layer based on the 3D features of the new object; to obtain the output features of the 3D diffusion transformer based on the Q matrix, K matrix and V matrix of the attention layer; wherein the output features of the 3D diffusion transformer are the features to be processed input to the diffusion transformer block of the next layer of the 3D diffusion transformer; and to generate the prediction result of the target video based on the output features of the last layer of the video generation model.

[0068] Figure 14 This is a schematic diagram of a training apparatus 1400 for a video generation model according to an embodiment of the present disclosure. The apparatus 1400 may include: The feature processing module 1410 is used to input the initial features of the target video, the mask image of the original video, and the multi-view image of the new object into the video generation model based on the training samples to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object; wherein, the training samples include the original video and the multi-view image of the new object. The prediction module 1420 is used to generate a prediction result of the target video based on the feature to be processed, the modulation parameter, and the three-dimensional feature; wherein the prediction result of the target video includes a video after replacing the original object in the original video with the new object; Training module 1430 is used to train the video generation model based on the prediction result and the loss determined by the initial features, and then train the trained video generation model.

[0069] Figure 15 This is a schematic diagram of the structure of a training apparatus 1500 for a video generation model according to another embodiment of the present disclosure. The apparatus 1500 includes a feature processing module 1510, a prediction module 1520, and a training module 1530. The functions of these modules are the same as those of the modules in the video generation model training apparatus of the above embodiment. In one embodiment, it further includes: Extraction module 1540 is used to extract a video frame from the original video; The overlay module 1550 is used to overlay the reference image of the new object onto the position of the original object in the video frame to obtain an overlay image; Input module 1560 is used to input the superimposed image into the image editing model via prompt words to obtain the first frame image; The generation module 1570 is used to generate initial features of the target video based on the first frame image and Gaussian noise; wherein, the initial features of the target video include features encoded based on the first frame image, and features other than the first frame include Gaussian noise.

[0070] In one implementation, the video generation model includes multiple diffusion transformer blocks, with at least one three-dimensional diffusion transformer inserted between the multiple diffusion transformer blocks; the output feature of the diffusion transformer block above the three-dimensional diffusion transformer is the feature to be processed input to the three-dimensional diffusion transformer.

[0071] In one embodiment, the feature processing module 1510 is used to input the initial features of the target video into the diffusion transformer block of the video generation model to obtain the features to be processed; input the mask image of the original video into the three-dimensional diffusion transformer of the video generation model to obtain modulation parameters; wherein, the mask image of the original video is an image obtained by masking the target area where the original object is located in the video frame of the original video; input the multi-view image of the new object into the three-dimensional diffusion transformer of the video generation model to obtain the three-dimensional features of the new object; wherein, the three-dimensional features include shape information and texture information.

[0072] In one embodiment, the prediction module 1520 is used to obtain features with motion information based on the features to be processed output by the diffusion transformer block of the next layer of the 3D diffusion transformer and the modulation parameters of the 3D diffusion transformer; to obtain the Q matrix of the attention layer based on the features with motion information; to obtain the K matrix and V matrix of the attention layer based on the 3D features of the new object; to obtain the output features of the 3D diffusion transformer based on the Q matrix, K matrix and V matrix of the attention layer; wherein the output features of the 3D diffusion transformer are the features to be processed input to the diffusion transformer block of the next layer of the 3D diffusion transformer; and to generate the prediction result of the target video based on the output features of the last layer of the video generation model.

[0073] In one implementation, the training module 1530 is used to determine a loss based on the prediction noise in the prediction result and the Gaussian noise in the initial features; adjust the model parameters of the video generation model based on the loss; and obtain the trained video generation model when the training stopping condition is met.

[0074] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0075] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0076] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0077] Figure 16 A schematic block diagram of an example electronic device 1600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0078] like Figure 16As shown, device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1602 or a computer program loaded into random access memory (RAM) 1603 from storage unit 1608. The RAM 1603 may also store various programs and data required for the operation of device 1600. The computing unit 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Input / output (I / O) interface 1605 is also connected to bus 1604.

[0079] Multiple components in device 1600 are connected to I / O interface 1605, including: input unit 1606, such as keyboard, mouse, etc.; output unit 1607, such as various types of monitors, speakers, etc.; storage unit 1608, such as disk, optical disk, etc.; and communication unit 1609, such as network card, modem, wireless transceiver, etc. Communication unit 1609 allows device 1600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0080] The computing unit 1601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1601 performs the various methods and processes described above, such as video generation methods and / or video generation model training methods. For example, in some embodiments, the video generation methods and / or video generation model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1600 via ROM 1602 and / or communication unit 1609. When the computer program is loaded into RAM 1603 and executed by the computing unit 1601, one or more steps of the video generation methods and / or video generation model training methods described above can be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured by any other suitable means (e.g., by means of firmware) to perform a video generation method and / or a training method for a video generation model.

[0081] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0082] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0083] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0084] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0085] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0086] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0087] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0088] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video generation method, comprising: The initial features of the target video, the mask image of the original video, and the multi-view images of the new object are input into the video generation model to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object. Based on the features to be processed, the modulation parameters, and the three-dimensional features, a prediction result for the target video is generated; wherein, the prediction result for the target video includes a video after replacing the original objects in the original video with the new objects.

2. The method according to claim 1, further comprising: Extract a video frame from the original video; The reference image of the new object is superimposed onto the original object's location in the video frame to obtain the superimposed image; The superimposed image is input into the image editing model using prompt words to obtain the first frame image; Based on the first frame image and Gaussian noise, initial features of the target video are generated; wherein, the initial features of the target video include features encoded based on the first frame image, and features other than the first frame include Gaussian noise.

3. The method according to claim 1 or 2, wherein, The video generation model includes multiple diffusion transformer blocks, with at least one 3D diffusion transformer inserted between the multiple diffusion transformer blocks; the output feature of the diffusion transformer block above the 3D diffusion transformer is the feature to be processed input to the 3D diffusion transformer.

4. The method according to claim 3, wherein, The initial features of the target video, the mask image of the original video, and the multi-view images of the new object are input into the video generation model to obtain the features to be processed, modulation parameters, and the three-dimensional features of the new object, including: The initial features of the target video are input into the diffusion transform block of the video generation model to obtain the features to be processed. The mask image of the original video is input into the three-dimensional diffusion transformer of the video generation model to obtain modulation parameters; wherein, the mask image of the original video is the image obtained by masking the target area where the original object is located in the video frame of the original video; The multi-view images of the new object are input into the three-dimensional diffusion transformer of the video generation model to obtain the three-dimensional features of the new object; wherein, the three-dimensional features include shape information and texture information.

5. The method according to claim 4, wherein, Based on the features to be processed, the modulation parameters, and the three-dimensional features, a prediction result for the target video is generated, including: Based on the features to be processed output by the diffusion converter block of the upper layer of the three-dimensional diffusion converter and the modulation parameters of the three-dimensional diffusion converter, features with motion information are obtained; Based on the features containing motion information, the Q matrix of the attention layer is obtained; Based on the three-dimensional features of the new object, the K matrix and V matrix of the attention layer are obtained; Based on the Q matrix, K matrix, and V matrix of the attention layer, the output features of the three-dimensional diffusion transformer are obtained; wherein, the output features of the three-dimensional diffusion transformer are the unprocessed features of the diffusion transformer block of the next layer input to the three-dimensional diffusion transformer. Based on the output features of the last layer of the video generation model, a prediction result for the target video is generated.

6. A training method for a video generation model, comprising: Based on the training samples, the initial features of the target video, the mask image of the original video, and the multi-view images of the new object are input into the video generation model to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object; wherein, the training samples include the original video and the multi-view images of the new object. Based on the features to be processed, the modulation parameters, and the three-dimensional features, a prediction result for the target video is generated; wherein, the prediction result for the target video includes a video after replacing the original objects in the original video with the new objects; Based on the prediction results and the loss determined by the initial features, the video generation model is trained, and the trained video generation model is then trained.

7. The method according to claim 6, further comprising: Extract a video frame from the original video; The reference image of the new object is superimposed onto the original object's location in the video frame to obtain the superimposed image; The superimposed image is input into the image editing model using prompt words to obtain the first frame image; Based on the first frame image and Gaussian noise, initial features of the target video are generated; wherein, the initial features of the target video include features encoded based on the first frame image, and features other than the first frame include Gaussian noise.

8. The method according to claim 6 or 7, wherein, The video generation model includes multiple diffusion transformer blocks, with at least one 3D diffusion transformer inserted between the multiple diffusion transformer blocks; the output feature of the diffusion transformer block above the 3D diffusion transformer is the feature to be processed input to the 3D diffusion transformer.

9. The method according to claim 8, wherein, The initial features of the target video, the mask image of the original video, and the multi-view images of the new object are input into the video generation model to obtain the features to be processed, modulation parameters, and the three-dimensional features of the new object, including: The initial features of the target video are input into the diffusion transform block of the video generation model to obtain the features to be processed. The mask image of the original video is input into the three-dimensional diffusion transformer of the video generation model to obtain modulation parameters; wherein, the mask image of the original video is the image obtained by masking the target area where the original object is located in the video frame of the original video; The multi-view images of the new object are input into the three-dimensional diffusion transformer of the video generation model to obtain the three-dimensional features of the new object; wherein, the three-dimensional features include shape information and texture information.

10. The method according to claim 9, wherein, Based on the features to be processed, the modulation parameters, and the three-dimensional features, a prediction result for the target video is generated, including: Based on the features to be processed output by the diffusion converter block of the upper layer of the three-dimensional diffusion converter and the modulation parameters of the three-dimensional diffusion converter, features with motion information are obtained; Based on the features containing motion information, the Q matrix of the attention layer is obtained; Based on the three-dimensional features of the new object, the K matrix and V matrix of the attention layer are obtained; Based on the Q matrix, K matrix, and V matrix of the attention layer, the output features of the three-dimensional diffusion transformer are obtained; wherein, the output features of the three-dimensional diffusion transformer are the unprocessed features of the diffusion transformer block of the next layer input to the three-dimensional diffusion transformer. Based on the output features of the last layer of the video generation model, a prediction result for the target video is generated.

11. The method according to any one of claims 6 to 10, wherein, Based on the prediction results and the loss determined by the initial features, the video generation model is trained, and the trained video generation model is then trained, including: The loss is determined based on the prediction noise in the prediction results and the Gaussian noise in the initial features; The model parameters of the video generation model are adjusted based on the loss. Once the training stopping condition is met, the trained video generation model is obtained.

12. A video generation apparatus, comprising: The feature processing module is used to input the initial features of the target video, the mask image of the original video, and the multi-view image of the new object into the video generation model to obtain the features to be processed, the modulation parameters, and the three-dimensional features of the new object. The prediction module is used to generate a prediction result of the target video based on the features to be processed, the modulation parameters, and the three-dimensional features; wherein the prediction result of the target video includes a video after replacing the original objects in the original video with the new objects.

13. A training device for a video generation model, comprising: The feature processing module is used to input the initial features of the target video, the mask image of the original video, and the multi-view images of the new object into the video generation model based on the training samples to obtain the features to be processed, modulation parameters, and the three-dimensional features of the new object; wherein, the training samples include the original video and the multi-view images of the new object. A prediction module is used to generate a prediction result of the target video based on the features to be processed, the modulation parameters, and the three-dimensional features; wherein the prediction result of the target video includes a video after replacing the original objects in the original video with the new objects; The training module is used to train the video generation model based on the prediction results and the loss determined by the initial features, and then train the trained video generation model.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images