Training method of video generation model, and video generation method and device

Abstract

The invention provides a training method of a video generation model and a video generation method and device, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes of digital people, e-commerce and the like. The specific implementation scheme is as follows: acquiring a first source video and a first target image; determining a target interaction part in the first source video, and determining a first reference video and a first mask video based on the target interaction part; determining a first video representation based on the preset noise, the first reference video, and the first masked video; determining a first image representation based on the first target image; inputting the first video representation and the first image representation into a to-be-trained first video generation model, and outputting a predicted video representation by the first video generation model; and training a first video generation model based on the predicted video representation and the first source video to obtain a second video generation model. The interaction effect of the human body and the object in the video can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to the fields of computer vision, deep learning, and large models, and can be applied to scenarios such as digital humans and e-commerce. Background Technology

[0002] In recent years, driven by information digitization, applications such as virtual anchors, online education, e-commerce live streaming, and advertising post-production have emerged, leading to a continuous increase in demand for realistic and believable digital human video content. This is especially true for videos showcasing human-product interactions, where stringent requirements exist. For example, the video must ensure that the contact points between the human body and objects conform to physical laws, making the interaction natural and logical; furthermore, replaced or newly added objects must maintain consistent appearance throughout the video sequence. Meeting these requirements has become a pressing issue in the field of digital human video generation. Summary of the Invention

[0003] This disclosure provides a training method for a video generation model, a video generation method, and an apparatus.

[0004] According to one aspect of this disclosure, a method for training a video generation model is provided, comprising: Acquire the first source video and the first target image; Identify the target interactive part in the first source video, and based on the target interactive part, determine the first reference video and the first mask video; Based on preset noise, the first reference video, and the first masked video, a first video representation is determined; and based on the first target image, a first image representation is determined. The first video representation and the first image representation are input into the first video generation model to be trained, and the first video generation model outputs the predicted video representation. Based on the predicted video representation and the first source video, the first video generation model is trained to obtain the second video generation model.

[0005] According to another aspect of this disclosure, a video generation method is provided, comprising: Acquire the second source video and the second target image; Identify the target interaction area in the second source video, and based on the target interaction area, determine the second reference video and the second mask video; The third video representation is determined using the second reference video and the second masked video; and the third image representation is determined using the second target image. The third video representation and the third image representation are input into a pre-trained second video generation model to obtain the target generated video; The second video generation model is obtained by training the first video generation model using the training method proposed in this disclosure.

[0006] According to another aspect of this disclosure, a training apparatus for a video generation model is provided, comprising: The first acquisition module is used to acquire the first source video and the first target image; The first video determination module is used to determine the target interaction part in the first source video, and based on the target interaction part, determine the first reference video and the first mask video. The first representation determination module is used to determine a first video representation based on preset noise, the first reference video, and the first masked video; and to determine a first image representation based on the first target image. The first input module is used to input the first video representation and the first image representation into the first video generation model to be trained, and the first video generation model outputs the predicted video representation. The training module is used to train the first video generation model based on the predicted video representation and the first source video to obtain the second video generation model.

[0007] According to another aspect of this disclosure, a video generation apparatus is provided, comprising: The second acquisition module is used to acquire the second source video and the second target image; The second video determination module is used to determine the target interaction part in the second source video, and based on the target interaction part, determine the second reference video and the second mask video. The second representation determination module is used to determine a third video representation using the second reference video and the second mask video; and to determine a third image representation using the second target image; The second input module is used to input the third video representation and the third image representation into the pre-trained second video generation model to obtain the target generated video; The second video generation model is obtained by training the first video generation model using the training device proposed in this disclosure.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0011] This disclosure determines a first source video and a first target image, locates the target interaction area in the first source video, and then generates a first reference video and a first mask video, providing accurate guidance information for subsequent processing. Determining a first video representation based on preset noise, the first reference video, and the first mask video, and determining a first image representation based on the first target image, accurately covers various feature information required for video generation. Furthermore, the first video representation and the first image representation are input into a first video generation model, which outputs a predicted video representation. The predicted video representation and the first source video are then used to train the model, resulting in a second video generation model. This second generation model maintains the consistency of the target object's appearance during long-sequence video generation, reducing appearance distortion and accurately integrating the movement of the target object's interactive areas, improving the smoothness of interactive actions. In addition, it maintains a stable state in the background region, reducing artifacts and improving the quality of video generation, ultimately achieving a more realistic and coordinated reproduction of interactive behavior.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure; Figure 2 This is a flowchart illustrating the implementation of a training method for a video generation model according to an embodiment of the present disclosure. Figure 3 This is a schematic diagram of the process for generating a target prediction video according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of a process for determining a predicted video representation according to an embodiment of the present disclosure; Figure 5 This is a flowchart illustrating the implementation of a video generation method according to an embodiment of the present disclosure; Figure 6This is a schematic diagram of the structure of a training device 600 for a video generation model according to an embodiment of the present disclosure; Figure 7 This is a schematic diagram of the structure of a video generation apparatus 700 according to an embodiment of the present disclosure; Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] The term "and / or" in this disclosure indicates that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document means any combination of at least two of a plurality of options, such as including at least one of A, B, and C, which can mean including any one or more elements selected from the set of A, B, and C. The terms "first" and "second" in this document refer to and distinguish multiple similar technical terms, and do not imply a specific order or a limitation to only two. For example, "first feature" and "second feature" refer to two types / two features; the first feature can be one or more, and the second feature can also be one or more.

[0016] With the rapid development of digital technology, emerging application scenarios such as virtual anchors, online education, e-commerce live streaming, and advertising post-production are on the rise, and the market demand for realistic and credible digital human video content is showing a continuous growth trend. Especially in video scenarios involving human-product interaction, the requirements for video quality are quite stringent. On the one hand, it is necessary to ensure that the contact relationship between the human hand and the object conforms to the laws of physics, making the interaction natural, smooth, and realistic; for example, details such as finger grasping and covering must be accurate and reasonable. On the other hand, for objects that are replaced or newly injected into the video, it is necessary to ensure that their appearance remains consistent throughout the entire video sequence, covering aspects such as the object's shape, color, texture, and markings.

[0017] Currently, most mainstream human animation or human-driven methods have limitations. They often focus only on simulating and generating human posture, appearance, or head movements, lacking specialized modeling for the interaction between human hands and objects. This makes it difficult for these methods to accurately maintain details such as finger occlusion and grasping in complex interactive scenarios, resulting in stiff and unnatural interactive actions that fail to meet the needs of practical applications.

[0018] While some methods for reenacting human-object interactions have solved certain problems to some extent, the models used in these methods are typically trained on specific datasets. For example, these methods require assumptions such as a clean and uncluttered background and a limited and fixed variety of objects. However, when these models are directly applied to complex real-world environments, they suffer from insufficient generalization ability, resulting in unnatural generated interactions that cannot adapt to diverse scenario requirements.

[0019] In recent years, the emergence of general-purpose video editing and generation models has brought new ideas to video processing. These models are trained on large-scale internet video data, exhibiting good robustness and controllability, and can handle various complex video editing and generation tasks to a certain extent. However, if simple masks or textures are used to replace objects in these models, a series of problems will arise. For example, when generating object poses, the model tends to tile the objects rather than generating natural and reasonable interactive poses based on hand movements, resulting in unrealistic interactive effects.

[0020] In view of the above problems, this disclosure proposes an innovative method that can specifically enhance the interaction between human body parts (such as hands) and objects without compromising general video generation capabilities, thereby improving the realism of the interaction and the consistency of the object's identity, and meeting the growing demand for high-quality digital human video content.

[0021] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure, such as... Figure 1As shown in the illustration, the application scenario diagram of this disclosure may include, but is not limited to, a model training device 110 and a video generation model 120, which can communicate with each other via any type of wired or wireless network. Specifically, the model training device 110 can be used to receive training samples and train an initial model based on the training samples to obtain a trained video generation model 120. The training samples may include the original video, the video of the object to be interacted with, and the target video for completing the interaction. In this disclosure embodiment, the model training device 110 may include an electronic device or server for providing background management for the video generation model 120. Furthermore, this disclosure embodiment does not impose a specific limitation on the number of model training devices 110; for example, the application scenario diagram of this disclosure embodiment may include one or more model training devices 110.

[0022] Figure 2 This is a flowchart illustrating the implementation of a training method for a video generation model according to an embodiment of the present disclosure, including: S210. Determine the first source video and the first target image; S220. Determine the target interactive part in the first source video, and based on the target interactive part, determine the first reference video and the first mask video; S230. Based on preset noise, the first reference video, and the first masked video, determine a first video representation; and based on the first target image, determine a first image representation; S240. Input the first video representation and the first image representation into the first video generation model to be trained, and output the predicted video representation from the first video generation model. S250. Based on the predicted video representation and the first source video, the first video generation model is trained to obtain the second video generation model.

[0023] In this embodiment of the disclosure, the first source video may be a video showing interaction between a human body part and an original object. Here, the interaction part can be a human body part, such as a hand or foot; or it can be a body part of a robot or digital human, such as a hand or foot. The first target image may include an image of the object (referred to as the target object) that needs to be replaced in the first source video. In one example, the first target image may include an image of a single object.

[0024] In this embodiment of the disclosure, the target interactive part can be an interactive part of a human or digital human in the first source video, such as a body part of a human or digital human. In one example, the target interactive part can be selected from the first source video according to the requirements of the video generation task. For example, in the video generation task, if it is necessary to replace the book in the digital human's hand with a pen, then the digital human's hand can be considered as the target interactive part.

[0025] This disclosure uses the target interactive part as the center and can determine a reference area in the first source video that includes the target interactive part and a certain range around it. This area can be of any shape, such as a circular area or a rectangular area. For example, if the target interactive part is a person's hand, and the person's hand has a linkage relationship with the arm and part of the body, then the reference range may need to include the entire arm and part of the upper body area.

[0026] In one example, if the target interaction part is the hand, the reference area can be the Hand-Object Interaction (HOI) area.

[0027] In this embodiment, a first masked video can be determined based on the target interaction area of ​​the first source video. In one example, this disclosure can determine a region (referred to as a reference region) associated with the target interaction area, then crop the reference region, and convert the pixel values ​​that meet the requirements in the reference region to constant values ​​(i.e., 0 or 1) to obtain the first masked video. Here, this disclosure can use binarization to convert the pixel values ​​in the reference region that meet the binarization threshold to 1, and the pixel values ​​that do not meet the binarization threshold to 0, to obtain the first masked video.

[0028] This disclosure enables the determination of a first reference video based on a first source video and a first mask video. In one example, this disclosure can binarize a reference region based on the first source video to obtain the first reference video. For instance, if the binarization threshold is 127, this disclosure can set pixel values ​​less than 127 in the reference region of each video frame in the first source video to 0, and set the remaining pixel values ​​in the reference region to 1. The remaining pixels in the first source video can also be set to 1 to obtain the first reference video.

[0029] In this way, the first reference video can directly use the background content of the first source video. This design allows the first reference video to provide background semantic information for the first video generation model, while clearly defining the reference region as the target to be reconstructed. It is understandable that the first source video, the first reference video, and the first mask video can have the same number of frames.

[0030] In this embodiment of the disclosure, the preset noise can be a random or regularly occurring interference signal added from an external source. In one example, the preset noise can be a randomly generated sequence of pixel values. This preset noise may not have explicit semantic information, but it possesses randomness and diversity.

[0031] In this disclosure, the first video representation can be a digital description of a first source video generated based on preset noise, a first reference video, and a first mask video. In one example, this disclosure can utilize a deep learning model (such as a convolutional neural network, a recurrent neural network, etc.) to process the preset noise, the first reference video, and the first mask video. Here, the first reference video provides visual information about the target interaction area and its surrounding environment, the first mask video clarifies the location and extent of the target interaction area, and the preset noise can increase the diversity and robustness of the first source video.

[0032] In one example, this disclosure can utilize the same or different deep learning models to convert preset noise, a first reference video, and a first masked video into a fixed-dimensional vector or feature map, which can be considered as a representation of the first video.

[0033] The first image representation is a digital description of the image content generated based on the first target image. Similar to the first video representation, this disclosure can also employ a deep learning model to process the first target image. In one example, the deep learning model can extract key features from the first target image, such as the target object contained in the first target image, the shape of the object, texture, etc., and convert these features into a fixed-dimensional vector or feature map, which can be considered as the first image representation.

[0034] Furthermore, this disclosure allows the first video representation and the first image representation to be input into a first video generation model to be trained, and the first video generation model to output a predicted video representation.

[0035] In one example, the first video generation model could be a diffusion model based on a transformer architecture, such as a Diffusion Transformer (DiT). This first video generation model has feature extraction and generation capabilities, enabling preliminary processing of video data.

[0036] In one example, after receiving the first video representation and the first image representation, the first video generation model can further extract features from these input data (i.e., the first video representation and the first image representation) using internal structures such as convolutional layers and fully connected layers. For example, for the first video representation, the first video generation model can analyze its temporal and spatial features to understand the changing patterns of the target interaction parts in the video; for the first image representation, the first video generation model can focus on the static features of the target object, such as shape and color.

[0037] After feature extraction, the first video generation model can fuse the features of the first video representation and the first image representation. In one example, the fusion method may include concatenation, weighted summation, attention mechanisms, etc., with the aim of organically combining the features of the target object and the features of the video scene.

[0038] In this embodiment of the disclosure, the predicted video representation can be a digital video description generated by a first video generation model based on the input first video representation and first image representation. This predicted video representation can be considered a potential representation of the final generated video, and can include all feature information of the target object predicted by the first video generation model after it is integrated into the video scene, such as the target object's position and pose. The generation process of the predicted video representation will be described in detail later.

[0039] Furthermore, this disclosure can determine the corresponding target predicted video based on the predicted video representation, and then train the first video generation model based on the target predicted video and the first source video. In one example, this disclosure can calculate a loss function for the non-reference region based on the non-reference region of the target predicted video and the non-reference region of the first source video. This loss function can be used to measure the deviation between the content generated by the first video generation model in the non-reference region and the actual situation.

[0040] In another example, this disclosure may employ mean absolute error (MAE) to calculate the loss function for the reference region. Here, MAE can be used to measure the difference at the pixel level between the reference regions of the target prediction video and the reference regions of the first source video by calculating the average of the absolute values ​​between the corresponding pixels.

[0041] In this embodiment of the disclosure, the loss function for the non-reference region and the loss function for the reference region can be fused (e.g., weighted fusion), and then the fused loss function can be used to train the first video generation model to obtain the second video generation model.

[0042] In the embodiments disclosed herein, it is understood that the training objective of the first video generation model may be to reconstruct the first source video under random noise input conditions.

[0043] By employing the above method, this disclosure determines a first source video and a first target image, locates the target interaction area in the first source video, and then generates a first reference video and a first mask video, providing accurate guidance information for subsequent processing. Determining a first video representation based on preset noise, the first reference video, and the first mask video, and determining a first image representation based on the first target image, can accurately cover various feature information required for video generation. Furthermore, the first video representation and the first image representation are input into a first video generation model, which outputs a predicted video representation. The predicted video representation and the first source video are then used to train the model, resulting in a second video generation model. This second generation model can maintain the consistency of the target object's appearance during long-sequence video generation, reducing appearance distortion and accurately integrating the movement of the target interaction area, improving the smoothness of interactive actions. In addition, it can maintain a stable state in the background area, reducing artifacts and improving the quality of video generation, ultimately achieving a more realistic and coordinated reproduction of interactive behavior.

[0044] Figure 3 This is a schematic diagram of the process for generating a target prediction video according to an embodiment of the present disclosure. Figure 3 A method for replaying human-object interaction based on a video diffusion transformer is provided. This method mainly includes three key components: HOI Condition Unit (HCU), a temporally balanced attention mechanism based on Head-Sliding Rotary Position Embedding (Head-Sliding RoPE), and a Spatial Attention Gate. This enables the first video generation model to understand HOI behavior and adapt to the shape and orientation of different objects.

[0045] like Figure 3 As shown, this disclosure can use a first source video and a target interactive area to determine a first reference video and a first mask video. Here, the target interactive area can be a person's hand in the first source video, and thus this disclosure can set the hand and its surrounding area as the HOI region (i.e., the aforementioned reference region).

[0046] This disclosure describes how to construct a first reference video (V_r) and a first mask video (V_mask) based on HOI regions for a given first source video. For example, the HOI regions in the first source video are binarized to obtain the first reference video; and the HOI regions are cropped and binarized to obtain the first mask video.

[0047] To reduce the introduction of branch structures or additional parameters into the first video generation model 340, this disclosure proposes HCU, whose core idea is to directly inject HOI-related guiding information into the latent space input stream to achieve lightweight HOI structure modeling.

[0048] In one example, this disclosure can be based on HCU and, through a unified latent space conditional injection mechanism, enable the first video generation model 340 to parse HOI semantics without adjusting the network structure.

[0049] In some implementations, a first video representation is determined based on preset noise, a first reference video, and a first masked video; and a first image representation is determined based on a first target image, including: Using a pre-trained encoder, preset noise, a first reference video, and a first target image are encoded respectively to obtain noise encoding, first reference video encoding, and first target image encoding; and average pooling is performed on the first mask video to obtain the first mask video features; Using a preset word segmenter, the noise code, the first reference video code, the first target image code, and the first mask video feature are segmented into words respectively; Based on the noise coding after word segmentation, the first reference video coding, and the first mask video features, a first video representation is determined; and based on the first target image coding after word segmentation, a first image representation is determined.

[0050] like Figure 3 As shown, this disclosure can utilize a pre-trained encoder 310 (such as a variational autoencoder (VAE)) to encode preset noise, a first reference video, and a first target image. Here, the encoder 310 can be a deep learning model.

[0051] In one example, encoder 310 can extract features from randomly generated noise signals to obtain statistical characteristics and random patterns of the noise. This disclosure allows encoder 310 to perform multi-level feature extraction on noisy images to generate noise codes.

[0052] This disclosure allows the encoder 310 to perform temporal-spatial joint coding on the first reference video. For example, the encoder 310 can be used to extract motion features and local texture details between video frames to generate a first reference video code that includes a temporal dimension.

[0053] This disclosure allows the encoder 310 to extract visual features from the first target image, capturing static features such as the shape, texture, and color of the target object contained in the first target image, thereby forming the first target image code.

[0054] In this embodiment, the pooling module 320 can be used to perform average pooling (or downsampling) on ​​the first masked video to obtain the features of the first masked video. Here, average pooling is a spatial downsampling technique that calculates the average value of a local region on the video by sliding a fixed-size window, compressing high-resolution two-dimensional data into a low-resolution feature map (i.e., simplifying the spatial information of the binary mask). For example, applying an 8×8 pooling window to the video frames of the 256×256 first masked video outputs a 32×32 feature map, where each pixel value represents the average value of pixels within the original window. In one example, the pooling module 320 can be a deep learning model.

[0055] Furthermore, this disclosure can utilize a preset tokenizer 330 to segment the noise code, the first reference video code, the first target image code, and the first mask video features into words respectively. Here, the preset tokenizer 330 can convert continuous encoded data (i.e., noise code, first reference video code, and first target image code) and feature data (i.e., first mask video features) into discrete token sequences.

[0056] For example, the preset word segmenter 330 can perform sub-word-level segmentation on the noise coding, the first reference video coding, the first target image coding, and the first mask video features to generate corresponding token sequences.

[0057] In one example, the preset word segmenter 330 can be a deep learning model.

[0058] Furthermore, this disclosure can concatenate the segmented noise code, the first reference video code, and the first mask video features to obtain the first video representation Tv; at the same time, the segmented first target image code can be determined as the first image representation Tref.

[0059] By employing the above method, this disclosure encodes preset noise, a first reference video, and a first target image using a pre-trained encoder. This accurately extracts deep features from different data, generating noise codes, first reference video codes, and first target image codes, providing a rich feature foundation for subsequent processing. Furthermore, average pooling is performed on the first masked video, effectively preserving key information of the target interaction region while reducing data dimensionality and improving data processing efficiency. Further, a preset word segmenter is used to segment various codes, achieving a discretized representation of encoded and feature information, enhancing the model's ability to analyze complex features. Based on the segmented encoded and feature information, the first video representation and the first image representation are determined, enabling the fusion and expression of video and image information within a unified semantic space.

[0060] Furthermore, by constructing a latent space representation of the input data, the first generative model can focus on generating HOI regions while maintaining the consistency of the video background.

[0061] Furthermore, such as Figure 3 As shown, this disclosure can input the first video representation Tv and the first image representation Tref into the first video generation model 340 to obtain the predicted video representation, and then train the first video generation model 340 based on the predicted video representation and the first source video.

[0062] In some implementations, the first video generation model is trained based on the predicted video representation and the first source video, including: Using a preset restorer, the predicted video representation is restored to a continuous predicted video code; The predicted video is decoded using a pre-trained decoder to obtain the target predicted video. The target predicted video is compared with the first source video to train the first video generation model.

[0063] like Figure 3 As shown, this disclosure utilizes a preset untokenizer 350 to restore the predicted video representation generated by the first video generation model 340 to obtain continuous predicted video codes. Here, the predicted video representation can be a high-dimensional abstract feature generated by the first video generation model 340, and the preset untokenizer 350 is responsible for restoring the discrete predicted video representation into continuous predicted video codes.

[0064] In one example, the preset restorer 350 may include deep learning models such as deconvolutional neural networks and fully connected layers. For instance, if the predicted video is represented as a 256-dimensional vector, the preset restorer may be progressively expanded to 1024 dimensions through three fully connected layers to meet the input requirements of the pre-trained decoder 360.

[0065] In this embodiment, the pre-trained decoder 360 can decode the predicted video encoding into a visualized target predicted video. The decoder 360 can be a VAE decoder, which learns the mapping pattern between video pixels and encoded features through a large amount of data, and can be directly applied to decoding tasks for the predicted video encoding. For example, the decoder 360 can employ a hierarchical generation strategy, first generating low-resolution video frames according to a preset video encoding, then gradually refining them to high resolution through upsampling, and finally sorting the high-resolution video frames to obtain the target predicted video.

[0066] In this embodiment of the disclosure, the calculation formula for the target prediction video can be: Vpre=D(M(X_rand,E(I_ref),E(V_r),psi(V_mask))) (1) In this formula, Vpre is the target prediction video; X_rand represents the preset noise; I_ref represents the first target image; V_r represents the first reference video; V_mask represents the first masked video; E(.) is the pre-trained encoder; M(.) is the first video generation model (i.e., the model based on the DiT structure, which can also be considered as a Transformer Block); D(.) is the pre-trained decoder; and psi(.) is the pooling module.

[0067] Furthermore, this disclosure can perform pixel-by-pixel or semantic comparison between the target predicted video and the first source video, calculate the loss function, and backpropagate to train the first video generation model to obtain the second video generation model.

[0068] In one example, for non-reference regions, the loss function for the non-reference region can be calculated by comparing pixel information between the target prediction video and the first source video. For reference regions (i.e., regions containing target interaction areas), the loss function for the reference region can be calculated by comparing semantic information, such as action category and contact state, between the target prediction video and the first source video at the target interaction areas. The loss functions for the non-reference regions and the reference regions are then fused to obtain the loss function used to train the first video generation model.

[0069] By employing the above method, this disclosure achieves self-supervised training of the first video generation model through a coordination mechanism between a pre-set restorer and a pre-trained decoder. The pre-set restorer can inversely map the predicted video representation into continuous encoding, preserving key spatiotemporal semantic features. Combined with the generalization ability of the pre-trained decoder, it accurately reconstructs the detailed information of the target predicted video. Furthermore, by performing self-supervised training of the first video generation model using the target predicted video and the first source video, the generalization ability of the second video generation model can be improved, making the generated target interaction part object-swapping video more realistic.

[0070] The following content details the specific structure of the first video generation model 340 and the generation method of the predicted video representation.

[0071] In some implementations, the first video generation model includes a projection layer, a multi-head attention layer, and a spatial attention gating layer; The first video representation and the first image representation are input into the first video generation model to be trained, and the first video generation model outputs a predicted video representation, including: The first video representation and the first image representation are respectively input into the projection layer, and the projection layer outputs the first matrix and the second matrix respectively. The multi-head attention layer and spatial attention gating layer determine the predicted video representation based on the first matrix and the second matrix.

[0072] In the embodiments disclosed herein, such as Figure 3 As shown, projection layer 341 can be a structure used in the first video generation model to generate query vector (Query, Q), key vector (Key, K), and value vector (Value, V).

[0073] In one example, projection layer 341 can essentially be a parameterized linear transformation layer whose core task is to map input features (such as the first video representation and the first image representation) into three independent vector spaces: Q, K, and V.

[0074] In this embodiment of the disclosure, inputting the first video representation into the projection layer 341 yields a first matrix; inputting the first image representation into the projection layer 341 yields a second matrix.

[0075] In some implementations, the first matrix includes a first initial query vector, a first initial key vector, and a first initial value vector for the first video representation; The second matrix contains a second initial query vector, a second initial key vector, and a second initial value vector for the representation of the first image.

[0076] like Figure 3 As shown, by inputting the first video representation Tv into the projection layer 341, a first initial query vector Qv, a first initial key vector Kv, and a first initial value vector Vv can be obtained; by inputting the first image representation Tref into the projection layer 341, a second initial query vector Qref, a second initial key vector Kref, and a second initial value vector Vref can be obtained.

[0077] In this embodiment of the disclosure, the first initial query vector Qv, the first initial key vector Kv, and the first initial value vector Vv can be considered to form a first matrix, and the second initial query vector Qref, the second initial key vector Kref, and the second initial value vector Vref can be considered to form a second matrix.

[0078] Using the above method, this disclosure can utilize a projection layer to map the first video representation and the first image representation into structured first and second matrices containing Q, K, and V, respectively, providing a data foundation for the generation of predicted video representations. Furthermore, this disclosure can utilize a multi-head attention layer and a spatial attention gating layer to process the first and second matrices, enabling the generated video to take into account both temporal coherence and spatial richness, providing a reference for the self-supervised training of the first video generation model, thereby training the model and improving its video generation and generalization capabilities.

[0079] In this embodiment of the disclosure, during the process of generating a predicted video representation by the first video generation model 340, the multi-head attention layer 342 and the spatial attention gating layer 343 stably and accurately fuse the target object in the first target image into the video sequence.

[0080] In some implementations, the multi-head attention layer and the spatial attention gating layer determine the predicted video representation based on a first matrix and a second matrix, including: The multi-head attention layer determines a first predicted query vector and a first predicted key vector based on a first initial query vector and a first initial key vector; and determines a second predicted query vector and a second predicted key vector based on a second initial query vector and a second initial key vector. The spatial attention gating layer determines the predicted video representation based on the first predicted query vector, the first predicted key vector, the second predicted query vector, the second predicted key vector, the first initial value vector, and the second initial value vector.

[0081] In this embodiment, a multi-head attention layer can be used to perform rotational position encoding on the first initial query vector Qv and the first initial key vector Kv using the RoPE algorithm to predict a rotation vector that incorporates relative position information. Subsequently, attention weights are calculated to obtain the first predicted query vector Q'v and the first predicted key vector K'v, which contain spatiotemporal dependencies. Similarly, this disclosure can use the RoPE algorithm to perform rotational position encoding on the second initial query vector Qref and the second initial key vector Kref to predict a rotation vector that incorporates relative position information. Subsequently, attention weights are calculated to obtain the second predicted query vector Q'ref and the second predicted key vector K'ref, which contain spatiotemporal dependencies.

[0082] Taking Qv and Qref as examples, the formulas for calculating Q'v and Q'ref can be: (2) in, The Hadamard product is an operation where corresponding elements of two matrices of the same order are multiplied. Assume... ,Right now A matrix of N×D dimensions; ,Right now A matrix belonging to the M×D dimension; It is the first rotation factor, and ,Right now for A matrix of dimensionality; It is the second twitch factor, and ,Right now for A matrix of dimensionality; It is the third twitch factor, and ,Right now for A matrix of dimensions. In this formula, Qv and Qref need to be encoded in the frame dimension (f), height dimension (h), and width dimension (w) respectively to obtain Q'v and Q'ref, and D1, D2, and D3 can be the same as D respectively.

[0083] In this embodiment of the disclosure, the first initial key vector Kv and the second initial key vector Kref can be input into formula (2) to obtain the first predicted key vector K'v and the second predicted key vector K'ref.

[0084] However, directly applying the RoPE algorithm to the HOI task of this disclosure may have drawbacks. Because the inter-frame distance between the first image representation (token) and the first video representation (token) accumulates over time, the attention weights decay with distance. For example, the response intensity of the first image representation decreases in the latter half of the video frame, leading to frame-by-frame blurring of the target object's appearance and breaks in temporal consistency. Furthermore, the features of the target object decay frame-by-frame in the temporal dimension. As the video plays, the object's appearance in the latter half of the video will differ significantly from the reference image, failing to maintain consistency.

[0085] In addition, the RoPE algorithm can cause object information to be spatially over-diffused into the background area, thereby introducing artifacts, destroying the realism of the original video background, and affecting the overall quality of the video.

[0086] To address the aforementioned issues, this disclosure proposes a temporally balanced attention mechanism based on Head-Sliding RoPE. Specifically, this disclosure slides the virtual frame position of the first image features between different heads of a multi-head attention mechanism, enabling each attention head to capture feature associations of the target object from different temporal ranges. In other words, this disclosure allows the information of the target object to uniformly influence each frame of video in the temporal dimension, ensuring that the object's appearance remains consistent throughout the entire video sequence; and in the spatial dimension, it concentrates object information in the interaction area, reducing information diffusion to the background area.

[0087] In some implementations, the multi-head attention layer includes multiple attention heads; The multi-head attention layer determines a first predicted query vector and a first predicted key vector based on a first initial query vector and a first initial key vector; and determines a second predicted query vector and a second predicted key vector based on a second initial query vector and a second initial key vector, including: The first rotation factor is determined based on the number of multiple attention heads, the number of video frames in the first source video, and the index of the attention heads; Using a first rotation factor, a preset second rotation factor, and a preset third rotation factor, the first initial query vector, the first initial key vector, the second initial query vector, and the second initial key vector are respectively converted into a first predicted query vector, a first predicted key vector, a second predicted query vector, and a second predicted key vector.

[0088] In this embodiment of the disclosure, the attention head can be an independent computing unit operating in parallel. Each attention head projects the input data into a feature subspace using a different subset of parameters, thereby capturing correlation patterns in the data from different angles. Figure 3 The attention head 1, attention head 2, and attention head n are shown.

[0089] This disclosure allows for the determination of a first rotation factor based on the number of attention heads, the number of video frames in the first source video, and the index of the attention heads. The formula for calculating the first rotation factor is: (3) in, The number of video frames in the first source video; For the number of heads; This serves as an index for the attention heads, pointing to a specific attention head within a multi-head attention layer. The first rotation factor, calculated using this formula, allows different attention heads to be assigned the same number of video frames. For example, if the first source video has 20 video frames and the number of attention heads is 4, then 5 video frames can be allocated equally to each attention head, enabling one attention head to focus on processing the first video representation of the first image representation along with the 5 video frames corresponding to that attention head.

[0090] like Figure 3 As shown, for different attention heads (i.e., attention head 1, attention head 2, and attention head n, etc.), the second initial query vector or the second initial key vector is located at different positions in the frame dimension (i.e., the f dimension) of the attention head. That is, as the index of the attention head increases, the second initial query vector or the second initial key vector can slide in the frame dimension. The step size of this sliding operation can be determined by the number of video frames of the first source video and the number of attention heads, which represents the average number of video frames that one attention head needs to process.

[0091] Furthermore, based on formula (2), this disclosure can utilize the first rotation factor obtained by formula (3) and the preset second rotation factor (i.e., the one in formula (2)). ) and the preset third rotation factor (i.e., in formula (2) The first initial query vector is converted into the first predicted query vector (Q'v), the first initial key vector is converted into the first predicted key vector (K'v), the second initial query vector is converted into the second predicted query vector (Q'ref), and the second predicted key vector is converted into the second predicted key vector (K'ref).

[0092] Using the above approach, this disclosure, based on a rotation coding strategy, employs a frame position sliding mechanism across attention heads to make the attention weight distribution of the target object in the first target image more balanced across the entire temporal range of the video, reducing the problem of feature response decay. Secondly, during the rotation coding process, the spatial position coding independence of the first image representation and the first video representation is maintained, ensuring that the spatial continuity between the object and the background is not temporally disrupted. Finally, this disclosure can reduce the phenomenon of target object feature degradation caused by distance accumulation in long sequence modeling using static position coding such as RoPE, thereby improving the temporal consistency of the target object's appearance in long videos.

[0093] In this embodiment, because the HOI region is highly dynamic and has a complex spatial structure, while the background region is relatively stable in time, directly applying the global features of the target object to the entire first source video would result in severe spatiotemporal artifacts in the dynamic region. Therefore, this disclosure proposes a spatial attention gating layer 343, whose core design includes two modules: (1) Hard Mask Gate (HMG), i.e. Figure 3 HMG 3431, as shown, achieves feature decoupling by explicitly constraining the attention flow. Its core rules include that within the HOI region, the first video representation is allowed to access information about the target object to capture dynamic interactive features; the representation of the background region in the first video representation is prohibited from accessing the features of the target object, thereby reducing the interference of dynamic object information on the static background; the target object will not pay attention to the first video representation in return, thus improving the independence of the target object features.

[0094] In this embodiment of the disclosure, HMG 3431 explicitly specifies the feasibility of information propagation through a binary mask, that is, defines the boundary of the channel through which information can flow.

[0095] (2) Soft Flow Gate (SFG), i.e. Figure 3 The SFG 3432 shown is a parameter that adjusts the intensity of information transmission based on dynamic weight allocation, i.e., finely controls the amplitude of the information flow.

[0096] In some implementations, the spatial attention gating layer determines the predicted video representation based on a first predicted query vector, a first predicted key vector, a second predicted query vector, a second predicted key vector, a first initial value vector, and a second initial value vector, including: The target mask matrix is ​​determined based on the first predicted query vector, the first predicted key vector, the second predicted query vector, and the second predicted key vector. The predicted video representation is determined based on the target mask matrix, the first initial value vector, and the second initial value vector.

[0097] In this embodiment of the disclosure, the target mask matrix can be a two-dimensional matrix, where the matrix elements can be either 1 or 0. In one example, the two dimensions of the target mask matrix can be set based on the predicted query vector and the predicted key vector. For example, concatenating the first predicted query vector and the second predicted query vector yields one dimension of the target mask matrix, and concatenating the first initial value vector and the second predicted key vector yields the other dimension of the target mask matrix.

[0098] In one example, a matrix element of 0 in the target mask matrix indicates that information access is not allowed between the predicted query vector and the predicted key vector, meaning that information flow between them is not permitted; a matrix element of 1 indicates that information access is allowed between the predicted query vector and the predicted key vector, meaning that information flow between them is permitted.

[0099] Furthermore, this disclosure can utilize the target mask matrix determined by HMG 3431 and the first initial value vector (i.e., Vv) and the second initial value vector (i.e., Vref) determined according to projection layer 341, in conjunction with SFG 3432, to determine the predicted video representation.

[0100] Using the above method, this disclosure determines the target mask matrix based on the predicted query vector and the predicted key vector, which can explicitly distinguish between dynamic interaction regions and static background regions. This suppresses background interference while enhancing the feature propagation of the target object, thereby reducing motion artifacts and information attenuation. Furthermore, by using the target mask matrix and multi-source initial value vectors, the long-term stability of the target object can be preserved, while also incorporating the spatiotemporal continuity of contextual information. This improves the appearance consistency and semantic integrity of the target object in long video sequences, thus enhancing the quality of video generation.

[0101] The following details how the target mask matrix is ​​determined using the HMG 3431 of the spatial attention gating layer 343.

[0102] In some implementations, the first predicted query vector includes a background query vector and a target query vector; The first predicted key vector includes the background key vector and the target key vector; The target mask matrix contains multiple matrix elements, each of which has a query index and a key index. The query index is associated with the background query vector, the target query vector, and the second predicted query vector, while the key index is associated with the background key vector, the target key vector, and the second predicted key vector.

[0103] In the embodiments disclosed herein, such as Figure 3 As shown, the first predicted query vector Q'v can be divided into a background query vector Q'v_B and a target query vector Q'v_HOI. For example, the first predicted query vector Q'v includes 6 vector elements (or tokens). The first 4 tokens can form the background query vector Q'v_B, and the last 2 tokens can form the target query vector Q'v_HOI.

[0104] The first predicted key vector K'v can be divided into a background key vector K'v_B and a target key vector K'v_HOI. For example, the first predicted key vector K'v includes 6 vector elements (or tokens). The first 4 tokens can form the background key vector K'v_B, and the last 2 tokens can form the target key vector K'v_HOI.

[0105] In this embodiment of the disclosure, the target mask matrix may include a query dimension and a key dimension. The query dimension can be determined based on the background query vector Q'v_B, the target query vector Q'v_HOI, and the second predicted query vector Q'ref. In one example, the token in the background query vector Q'v_B, the target query vector Q'v_HOI, and the second predicted query vector Q'ref can be considered as the query index of the matrix elements of the target mask matrix.

[0106] This disclosure can determine the key dimension based on the background key vector K'v_B, the target key vector K'v_HOI, and the second predicted key vector K'ref. In one example, the tokens of the background key vector K'v_B, the target key vector K'v_HOI, and the second predicted key vector K'ref can be considered as the key indices of the matrix elements of the target mask matrix.

[0107] Based on the above definition, the first predicted query vector and the first predicted key vector are respectively split into background and HOI region dual channels. At the same time, the relationship between the query index and query dimension of the target mask matrix, as well as the relationship between the key index and key dimension, are clarified, and a fine-grained routing of the attention flow between the first source video and the target object is constructed, providing data support for high-quality video generation.

[0108] In some implementations, determining the target mask matrix based on the first predicted query vector, the first predicted key vector, the second predicted query vector, and the second predicted key vector includes: When the query index is associated with the background query vector and the key index is associated with the second predicted key vector, or when the query index is associated with the second predicted index vector and the key index is associated with the background key vector or the target key vector, the matrix elements corresponding to the query index and the key index are set to 0. Set the remaining matrix elements to 1; The target mask matrix is ​​determined based on the matrix elements corresponding to the query index and the key index, as well as the remaining matrix elements.

[0109] In the embodiments disclosed herein, such as Figure 3 As shown, the target mask matrix has two parts in a masked state (i.e., the matrix elements in these two parts are 0). Specifically, when the query index of a matrix element is associated with the background query vector (Q'v_B) (or the query index is considered the background query vector), and the key index is associated with the second predicted key vector (K'ref) (or the key index is considered the second predicted key vector), the matrix element can be set to 0, which is the "mask 1" part in the target mask matrix. When the query index of a matrix element is associated with the second predicted query vector Q'ref (or the query index is considered the second predicted query vector), and the key index is associated with the background key vector K'v_B or the target key vector K'v_HOI (or the key index is considered the background key vector or the target key vector), the matrix element can be set to 0, which is the "mask 2" part in the target mask matrix.

[0110] In this embodiment of the disclosure, the remaining matrix elements may include the "Video to Video (V2V)" part, the "Image to HOI (I2HOI)" part, and the "Image to Image (I2I)" part of the target mask matrix. The "V2V" part indicates that there can be an information flow between the first predicted query vector Q'v and the first predicted key vector K'v. In one example, different spatial regions within a video frame can interact to capture local spatiotemporal dependencies. The "I2HOI" part indicates that there can be an information flow between the target query vector Q'v_HOI of the first predicted query vector and the second predicted key vector K'ref. In one example, the HOI region and the target object can interact to establish a semantic association between human actions and the target object. The "I2I" part indicates that there can be an information flow between the second predicted query vector Q'ref and the second predicted key vector K'ref. In one example, the target object can undergo feature aggregation to enhance global semantic consistency.

[0111] In this embodiment of the disclosure, the matrix elements of the target mask matrix can be expressed by the following formula: (4) In the above formula, The matrix elements representing the target mask matrix; m Indicates the lookup index for matrix elements; n This represents the key index of a matrix element. Furthermore, by using the lookup index and key index of the matrix elements, the numerical value of each element can be determined, thus identifying the target mask matrix.

[0112] By employing the above method, the target mask matrix is ​​determined based on specific settings for the query index and key index. When a specific index condition is met, the corresponding matrix element is set to 0, and the remaining matrix elements are set to 1. This allows for accurate control of information interaction between different regions, enabling the HOI region to acquire the necessary target object injection signal. This ensures that the HOI region can fully focus on and utilize key information related to the target object, laying the foundation for accurate identification and understanding of HOI region interaction behavior. Furthermore, by blocking the information flow between the background region and the target object, interference from the object image to the background region can be avoided, reducing object image contamination of the background region and improving the accuracy and stability of background region reconstruction in the video.

[0113] Furthermore, based on the target mask matrix, a stable, color-free, and artifact-free attention path can be constructed, enhancing the attention of the HOI region to the target object. This leads to a high degree of focus on key information related to the interaction between the HOI region and the target object, thereby improving the first video generation model's perception and understanding of the video generation task. Moreover, by masking the background region and the target object, the confusion between background information and target object information is reduced, further optimizing attention allocation. This enables the first video generation model to accurately distinguish information from different regions, improving the efficiency and accuracy of video generation.

[0114] In some implementations, the predicted video representation is determined based on the target mask matrix, a first initial value vector, and a second initial value vector, including: Based on the target mask matrix, the first prediction query vector, the first prediction key vector, the second prediction query vector, the second prediction key vector, the first initial value vector, and the second initial value vector, the second video representation and the second image representation are determined. The predicted video representation is determined based on the second video representation and the second image representation.

[0115] In this embodiment of the disclosure, the second video representation can be determined using the target mask matrix, the first prediction query vector Q'v, the first prediction key vector K'v, and the first initial value vector Vv. The calculation formula is as follows: (5) In the above formula, T'v is the second video representation; M represents the target mask matrix; represents the dimension of the first predicted key vector K'v; softmax(.) represents the normalized exponential function.

[0116] Similarly, this disclosure can use the target mask matrix, the first prediction query vector Q'ref, the first prediction key vector K'ref, and the first initial value vector Vref to determine the second image representation T'ref, and the calculation formula is as follows: (6) In the embodiments disclosed herein, such as Figure 3 As shown, Vv and Vref can be concatenated, and then the second video representation T'v and the second image representation T'ref can be calculated using the concatenated initial value vector (including the first initial value vector and the second initial value vector), the target mask matrix, the first prediction query vector, the second prediction query vector, the first prediction key vector and the second prediction key vector.

[0117] Furthermore, this disclosure can determine the predicted video representation based on the second video representation T'v and the second image representation T'ref, using the SFG 3432 of the spatial attention gating layer 343.

[0118] By employing the above method, this disclosure determines the second video representation and the second image representation by comprehensively utilizing the target mask matrix, multiple sets of prediction query vectors and key vectors, and initial value vectors, thereby deriving the predicted video representation. In this way, the target mask matrix plays a role in accurately filtering and controlling information, and can guide the effective fusion of information from different sources according to preset rules. This allows the generated second video representation and second image representation to more accurately capture key features, thereby improving the accuracy and comprehensiveness of the determined predicted video representation.

[0119] The following details how to determine the predicted video representation using soft gating (SFG) based on the second video representation and the second image representation.

[0120] Figure 4 This is a schematic diagram of a process for determining a predicted video representation according to an embodiment of the present disclosure.

[0121] In some implementations, the spatial attention gating layer includes a multilayer perceptron and an activation module; Based on the second video representation and the second image representation, the predicted video representation is determined, including: The second video representation and the second image representation are input into the multilayer perceptron, and the multilayer perceptron outputs video features and image features. The video features and image features are input into the activation module to obtain the video activation value and the image activation value; The target video representation is determined based on the video activity value and the second video representation; and the target image representation is determined based on the image activity value and the second image representation. The predicted video representation is determined based on the target video representation and the target image representation.

[0122] like Figure 4 As shown, the soft gating (SFG) may include a multilayer perceptron (MLP) 410 and an activation module 420.

[0123] In one example, MLP 410 can be a feedforward neural network consisting of an input layer, one or more hidden layers, and an output layer. It achieves nonlinear mapping through a fully connected structure and can integrate layer normalization networks and fully convolutional networks (FCNs).

[0124] In this embodiment of the disclosure, the second video representation and the second image representation can be input into the MLP 410 to obtain video features for the second video representation and image features for the second image representation.

[0125] Furthermore, this disclosure allows video features and image features to be input into activation module 420, and activation module 420 to output video activation values ​​and image activation values. Here, activation module 420 may include a sigmoid activation function, video activation values ​​may include weights of video features, and image activation values ​​may include weights of image features.

[0126] For example, for the second video representation T'v, the formula for calculating the video activation value Gv can be: (7) In this formula, This represents the LayerNorm in MLP 410. This represents a fully convolutional network (FCN) in MLP410. This indicates that module 420 is activated, such as the Sigmoid activation function.

[0127] Furthermore, this disclosure can utilize video activation values ​​and a second video representation to obtain the target video representation, and the calculation formula can be: (8) in, The target video is represented.

[0128] Similarly, for the second image representation T'ref, the formula for calculating the image activation value Gref can be: (9) Furthermore, this disclosure can utilize image activation values ​​and a second image representation to obtain the target image representation, and the calculation formula can be: (10) in, This represents the target image.

[0129] In this embodiment of the disclosure, the target video representation and the target image representation can be concatenated to obtain the predicted video representation.

[0130] Using the above method, this disclosure achieves dynamic enhancement and accurate fusion of video and target objects through a semantically adaptive activation control mechanism. Specifically, high-level semantic features (i.e., video features and image features) are first extracted from the second video representation and the second image representation. Then, an activation module generates video activation values ​​and image activation values ​​based on semantic weights, realizing the function of automatically adjusting the feature update intensity according to the semantic type of the region (such as HOI regions and non-HOI regions). For HOI regions where the target object is needed, the activation module can extract the associated features from the second video representation and the second image representation; for semantically stable non-HOI regions, redundant updates are suppressed to maintain the consistency of their original features. This mechanism can further improve the stability and accuracy of video generation.

[0131] Figure 5 This is a flowchart illustrating the implementation of a video generation method according to an embodiment of the present disclosure, including: S510, acquire the second source video and the second target image; S520. Determine the target interaction part in the second source video, and based on the target interaction part, determine the second reference video and the second mask video; S530: Determine a third video representation using a second reference video and a second masked video; and determine a third image representation using a second target image; S540. Input the third video representation and the third image representation into the pre-trained second video generation model to obtain the target generated video.

[0132] The second video generation model is obtained by training the first video generation model using the training method proposed in this disclosure.

[0133] In this embodiment of the disclosure, the second source video may be a video showing interaction between a human body part and the original object. Here, the interaction part can be a human body part, such as a hand or foot; or it can be a body part of a robot or digital human, such as a hand or foot. The second target image may include an image of the object (referred to as the target object) that needs to be replaced in the second source video. Here, the second target image may be an image of a single object.

[0134] In this embodiment of the disclosure, the target interactive part can be an interactive part of a human body or digital human in the second source video, such as a body part of a human or digital human. In one example, the target interactive part can be selected from the second source video according to the video generation requirements.

[0135] This disclosure uses the target interactive part as the center and can determine a reference area in the first source video that includes the target interactive part and a certain range around it. This area can be an area of ​​any shape, such as a circular area or a rectangular area.

[0136] Furthermore, this disclosure can determine the second masked video based on the target interactive area of ​​the second source video. In one example, this disclosure can determine the region (referred to as the reference region) associated with the target interactive area based on the target interactive area, then crop the reference region and convert the pixel values ​​in the reference region to constant values ​​(i.e., 0 or 1) to obtain the second masked video.

[0137] This disclosure allows for the determination of a second reference video based on a second source video and a second mask video. For example, this disclosure allows for the binarization of a reference region based on the second source video to obtain the second reference video.

[0138] Furthermore, this disclosure can utilize a second reference video and a second masked video to determine a third video representation. Here, the third video representation can be a digital description of the second source video generated based on the second reference video and the second masked video. In one example, this disclosure can utilize a deep learning model to process the second reference video and the second masked video to obtain the third video representation.

[0139] Furthermore, this disclosure can utilize the second target image to determine the third image representation. Similar to the third video representation, this disclosure can also employ a deep learning model to process the second target image to obtain the third image representation.

[0140] Furthermore, this disclosure allows the third video representation and the third image representation to be input into the second video generation model to obtain the target generated video.

[0141] Here, the structure of the second video generation model can be the same as that of the first video generation model to be trained, and the parameters of the second video generation model can be determined after training the first video generation model.

[0142] Using the above method, this disclosure dynamically generates a second reference video and a second mask video based on the target interaction part, which can provide spatial constraints for replacing the target object. Then, by jointly encoding the second reference video and the second mask video, a third video representation that retains the dynamics of the original scene is generated. At the same time, the third image representation of the second target image can provide accurate static features. By inputting the two into the second video generation model, high-quality fusion of the object to be replaced (i.e., the target object) and the video can be achieved, improving the reliability and accuracy of generating video for object replacement.

[0143] In some implementations, determining a third video representation using a second reference video and a second masked video; and determining a third image representation using a second target image, including: Using a pre-trained encoder, the second reference video and the second target image are encoded respectively to obtain the second reference video encoding and the second target image encoding; and the second mask video is subjected to average pooling operation to obtain the second mask video features; Using a preset word segmenter, the second reference video encoding, the second target image encoding, and the second mask video features are segmented into words respectively; Based on the second reference video encoding and the second mask video features after word segmentation, a third video representation is determined; and based on the second target image encoding after word segmentation, a third image representation is determined.

[0144] In this embodiment of the disclosure, a pre-trained encoder (such as a VAE) can be used to encode the second reference video and the second target image. Here, the encoder can be a deep learning model.

[0145] In this embodiment of the disclosure, a pooling module can be used to perform average pooling (or downsampling) on ​​the second mask video to obtain the features of the second mask video.

[0146] Furthermore, this disclosure can utilize a pre-defined tokenizer to encode the second reference video, the second target image, and the second mask video features, respectively, and then segment them into tokens. Here, the pre-defined tokenizer can convert continuous encoded data and feature data into discrete token sequences. Here, the pre-defined tokenizer can be a deep learning model.

[0147] Furthermore, this disclosure can concatenate the second reference video code and the second mask video features after word segmentation to obtain a third video representation; at the same time, the second target image code after word segmentation can be determined as the third image representation.

[0148] Using the above method, a pre-trained encoder is used to extract deep features from the second reference video and the second target image, providing a rich feature foundation for subsequent video generation. Simultaneously, average pooling is performed on the second mask video to obtain spatially aware mask features, effectively preserving key information of the target interaction area while reducing data dimensionality and improving data processing efficiency. Subsequently, a pre-defined word segmenter is used to segment the second reference video encoding, the second target image encoding, and the second mask video features, achieving a discretized representation of encoded and feature information, enhancing the model's ability to analyze complex features. Based on the segmented encoded and feature information, a third video representation and a third image representation are determined, enabling video and image information to be fused and expressed in a unified semantic space.

[0149] In some implementations, a third video representation and a third image representation are input into a pre-trained second video generation model to obtain a target generated video, including: The third video representation and the third image representation are input into the pre-trained second video generation model, and the second video generation model outputs the video generation representation. Using a preset restorer, the generated video representation is restored to a continuous target video encoding; The target video is encoded and decoded using a pre-trained decoder to obtain the target generated video.

[0150] In this embodiment, a preset untokenizer can be used to restore the video generation representation output by the second video generation model based on the third video representation and the third image representation. Here, the video generation representation can be a high-dimensional abstract feature generated by the second video generation model, and the preset untokenizer can restore the discrete video generation representation into a continuous target video code.

[0151] Furthermore, this disclosure can utilize a pre-trained decoder to encode and decode the target video into a visualized target-generated video. Here, the pre-trained decoder can be a VAE decoder.

[0152] By using the above method, this disclosure utilizes a preset restorer to inversely map the video generation representation into continuous encoding, preserving key spatiotemporal semantic features. Combined with the generalization ability of the pre-trained decoder, it accurately reconstructs the detailed information of the target generated video, thereby improving the stability and quality of the video output.

[0153] This disclosure also proposes a training device for a video generation model. Figure 6 This is a schematic diagram of the structure of a training apparatus 600 for a video generation model according to an embodiment of the present disclosure, comprising: The first acquisition module 610 is used to acquire the first source video and the first target image; The first video determination module 620 is used to determine the target interaction part in the first source video, and determine the first reference video and the first mask video based on the target interaction part. The first representation determination module 630 is used to determine a first video representation based on preset noise, the first reference video, and the first masked video; and to determine a first image representation based on the first target image. The first input module 640 is used to input the first video representation and the first image representation into the first video generation model to be trained, and the first video generation model outputs the predicted video representation. The training module 650 is used to train the first video generation model based on the predicted video representation and the first source video to obtain the second video generation model.

[0154] In some implementations, the first video generation model includes a projection layer, a multi-head attention layer, and a spatial attention gating layer; the first input module 640 is used for: The first video representation and the first image representation are respectively input into the projection layer, and the projection layer outputs the first matrix and the second matrix respectively; The multi-head attention layer and spatial attention gating layer determine the predicted video representation based on the first matrix and the second matrix.

[0155] In some implementations, the first matrix includes a first initial query vector, a first initial key vector, and a first initial value vector for the first video representation; The second matrix contains a second initial query vector, a second initial key vector, and a second initial value vector for the representation of the first image.

[0156] In some implementations, the first input module 640 is used for: The multi-head attention layer determines a first predicted query vector and a first predicted key vector based on a first initial query vector and a first initial key vector; and determines a second predicted query vector and a second predicted key vector based on a second initial query vector and a second initial key vector. The spatial attention gating layer determines the predicted video representation based on the first predicted query vector, the first predicted key vector, the second predicted query vector, the second predicted key vector, the first initial value vector, and the second initial value vector.

[0157] In some implementations, the multi-head attention layer includes multiple attention heads; the first input module 640 is used for: The first rotation factor is determined based on the number of multiple attention heads, the number of video frames in the first source video, and the index of the attention heads; Using a first rotation factor, a preset second rotation factor, and a preset third rotation factor, the first initial query vector, the first initial key vector, the second initial query vector, and the second initial key vector are respectively converted into a first predicted query vector, a first predicted key vector, a second predicted query vector, and a second predicted key vector.

[0158] In some implementations, the first input module 640 is used for: The target mask matrix is ​​determined based on the first predicted query vector, the first predicted key vector, the second predicted query vector, and the second predicted key vector. The predicted video representation is determined based on the target mask matrix, the first initial value vector, and the second initial value vector.

[0159] In some implementations, the first predicted query vector includes a background query vector and a target query vector; The first predicted key vector includes the background key vector and the target key vector; The target mask matrix contains multiple matrix elements, and each matrix element has a query index and a key index in the target mask matrix. The query index is associated with the background query vector, the target query vector, and the second predicted query vector, and the key index is associated with the background key vector, the target key vector, and the second predicted key vector.

[0160] In some implementations, the first input module 640 is used for: When the query index is associated with the background query vector and the key index is associated with the second predicted key vector, or when the query index is associated with the second predicted index vector and the key index is associated with the background key vector or the target key vector, the matrix elements corresponding to the query index and the key index are set to 0. Set the remaining matrix elements to 1; The target mask matrix is ​​determined based on the matrix elements corresponding to the query index and the key index, as well as the remaining matrix elements.

[0161] In some implementations, the first input module 640 is used for: Based on the target mask matrix, the first prediction query vector, the first prediction key vector, the second prediction query vector, the second prediction key vector, the first initial value vector, and the second initial value vector, the second video representation and the second image representation are determined. The predicted video representation is determined based on the second video representation and the second image representation.

[0162] In some implementations, the spatial attention gating layer includes a multilayer perceptron and an activation module; the first input module 640 is used for: The second video representation and the second image representation are input into the multilayer perceptron, and the multilayer perceptron outputs video features and image features. The video features and image features are input into the activation module to obtain the video activation value and the image activation value; The target video representation is determined based on the video activity value and the second video representation; and the target image representation is determined based on the image activity value and the second image representation. The predicted video representation is determined based on the target video representation and the target image representation.

[0163] In some implementations, the first representation determining module 630 is used for: Using a pre-trained encoder, preset noise, a first reference video, and a first target image are encoded respectively to obtain noise encoding, first reference video encoding, and first target image encoding; and average pooling is performed on the first mask video to obtain the first mask video features; Using a preset word segmenter, the noise code, the first reference video code, the first target image code, and the first mask video feature are segmented into words respectively; Based on the noise coding after word segmentation, the first reference video coding, and the first mask video features, a first video representation is determined; and based on the first target image coding after word segmentation, a first image representation is determined.

[0164] In some implementations, the training module 650 is used for: Using a preset restorer, the predicted video representation is restored to a continuous predicted video code; The predicted video is decoded using a pre-trained decoder to obtain the target predicted video. The target predicted video is compared with the first source video to train the first video generation model.

[0165] This disclosure also proposes a video generation apparatus. Figure 7 This is a schematic diagram of the structure of a video generation apparatus 700 according to an embodiment of the present disclosure, including: The second acquisition module 710 is used to acquire the second source video and the second target image; The second video determination module 720 is used to determine the target interaction part in the second source video, and determine the second reference video and the second mask video based on the target interaction part. The second representation determination module 730 is used to determine a third video representation using a second reference video and a second mask video; and to determine a third image representation using a second target image. The second input module 740 is used to input the third video representation and the third image representation into the pre-trained second video generation model to obtain the target generated video; The second video generation model is obtained by training the first video generation model using the training device proposed in this disclosure.

[0166] In some implementations, the second representation determination module 730 is used for: Using a pre-trained encoder, the second reference video and the second target image are encoded respectively to obtain the second reference video encoding and the second target image encoding; and the second mask video is subjected to average pooling operation to obtain the second mask video features; Using a preset word segmenter, the second reference video encoding, the second target image encoding, and the second mask video features are segmented into words respectively; Based on the second reference video encoding and the second mask video features after word segmentation, a third video representation is determined; and based on the second target image encoding after word segmentation, a third image representation is determined.

[0167] In some implementations, the second input module 740 is used for: The third video representation and the third image representation are input into a pre-trained second video generation model, which outputs a video generation representation. Using a preset restorer, the generated video representation is restored to a continuous target video encoding; The target video is encoded and decoded using a pre-trained decoder to obtain the target generated video.

[0168] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0169] The acquisition, storage, and application of personal information by users involved in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0170] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0171] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0172] like Figure 8As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0173] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0174] The computing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the training method for a video generation model and the video generation method. For example, in some embodiments, the training method for a video generation model and the video generation method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the training method for a video generation model and the video generation method described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured in any other suitable manner (e.g., by means of firmware) to perform a training method for the video generation model and a video generation method.

[0175] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0177] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0178] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0179] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0180] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0181] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0182] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a video generation model, comprising: Acquire the first source video and the first target image; The target interactive part in the first source video is determined, and based on the target interactive part, a first reference video and a first mask video are determined; A first video representation is determined based on preset noise, the first reference video, and the first masked video. And based on the first target image, determine the first image representation; The first video representation and the first image representation are input into the first video generation model to be trained, and the first video generation model outputs the predicted video representation. Based on the predicted video representation and the first source video, the first video generation model is trained to obtain the second video generation model.

2. The method according to claim 1, wherein, The first video generation model includes a projection layer, a multi-head attention layer, and a spatial attention gating layer; The step of inputting the first video representation and the first image representation into a first video generation model to be trained, and having the first video generation model output a predicted video representation, includes: The first video representation and the first image representation are respectively input into the projection layer, and the projection layer outputs the first matrix and the second matrix respectively. The multi-head attention layer and spatial attention gating layer determine the predicted video representation based on the first matrix and the second matrix.

3. The method according to claim 2, wherein, The first matrix includes a first initial query vector, a first initial key vector, and a first initial value vector for the representation of the first video; The second matrix contains a second initial query vector, a second initial key vector, and a second initial value vector for the representation of the first image.

4. The method according to claim 3, wherein, The multi-head attention layer and spatial attention gating layer determine the predicted video representation based on the first matrix and the second matrix, including: The multi-head attention layer determines a first predicted query vector and a first predicted key vector based on the first initial query vector and the first initial key vector; and determines a second predicted query vector and a second predicted key vector based on the second initial query vector and the second initial key vector. The spatial attention gating layer determines the predicted video representation based on the first predicted query vector, the first predicted key vector, the second predicted query vector, the second predicted key vector, the first initial value vector, and the second initial value vector.

5. The method according to claim 4, wherein, The multi-head attention layer includes multiple attention heads; The multi-head attention layer determines a first predicted query vector and a first predicted key vector based on the first initial query vector and the first initial key vector; and determines a second predicted query vector and a second predicted key vector based on the second initial query vector and the second initial key vector, including: A first rotation factor is determined based on the number of the plurality of attention heads, the number of video frames of the first source video, and the index of the attention heads; Using the first rotation factor, the preset second rotation factor, and the preset third rotation factor, the first initial query vector, the first initial key vector, the second initial query vector, and the second initial key vector are respectively converted into the first predicted query vector, the first predicted key vector, the second predicted query vector, and the second predicted key vector.

6. The method according to claim 4, wherein, The spatial attention gating layer determines the predicted video representation based on the first predicted query vector, the first predicted key vector, the second predicted query vector, the second predicted key vector, the first initial value vector, and the second initial value vector, including: Based on the first predicted query vector, the first predicted key vector, the second predicted query vector, and the second predicted key vector, the target mask matrix is ​​determined. The predicted video representation is determined based on the target mask matrix, the first initial value vector, and the second initial value vector.

7. The method according to claim 6, wherein, The first predicted query vector includes a background query vector and a target query vector; The first predicted key vector includes a background key vector and a target key vector; The target mask matrix contains multiple matrix elements, and each matrix element has a query index and a key index in the target mask matrix; the query index is associated with the background query vector, the target query vector and the second predicted query vector, and the key index is associated with the background key vector, the target key vector and the second predicted key vector.

8. The method according to claim 7, wherein, The step of determining the target mask matrix based on the first predicted query vector, the first predicted key vector, the second predicted query vector, and the second predicted key vector includes: When the query index is associated with the background query vector and the key index is associated with the second predicted key vector, or when the query index is associated with the second predicted index vector and the key index is associated with the background key vector or the target key vector, the matrix elements corresponding to the query index and the key index are set to 0. Set the remaining matrix elements to 1; The target mask matrix is ​​determined based on the matrix elements corresponding to the query index and the key index, as well as the remaining matrix elements.

9. The method according to any one of claims 6-8, wherein, The step of determining the predicted video representation based on the target mask matrix, the first initial value vector, and the second initial value vector includes: Based on the target mask matrix, the first prediction query vector, the first prediction key vector, the second prediction query vector, the second prediction key vector, the first initial value vector, and the second initial value vector, the second video representation and the second image representation are determined. The predicted video representation is determined based on the second video representation and the second image representation.

10. The method according to claim 9, wherein, The spatial attention gating layer includes a multilayer perceptron and an activation module; Determining the predicted video representation based on the second video representation and the second image representation includes: The second video representation and the second image representation are input into the multilayer perceptron, and the multilayer perceptron outputs video features and image features. The video features and the image features are input into the activation module to obtain the video activation value and the image activation value; Based on the video activity value and the second video representation, a target video representation is determined; and based on the image activity value and the second image representation, a target image representation is determined. The predicted video representation is determined based on the target video representation and the target image representation.

11. The method according to claim 1, wherein, The first video representation is determined based on preset noise, the first reference video, and the first masked video. And based on the first target image, determine the first image representation, including: Using a pre-trained encoder, the preset noise, the first reference video, and the first target image are encoded respectively to obtain noise encoding, first reference video encoding, and first target image encoding; and average pooling is performed on the first masked video to obtain the first masked video features. Using a preset word segmenter, the noise code, the first reference video code, the first target image code, and the first masked video feature are segmented into words respectively. Based on the noise coding after word segmentation, the first reference video coding, and the first mask video features, the first video representation is determined; and based on the first target image coding after word segmentation, the first image representation is determined.

12. The method according to claim 1, wherein, The step of training the first video generation model based on the predicted video representation and the first source video includes: Using a preset restorer, the predicted video representation is restored into a continuous predicted video code; The predicted video is decoded using a pre-trained decoder to obtain the target predicted video. The target predicted video is compared with the first source video to train the first video generation model.

13. A video generation method, comprising: Acquire the second source video and the second target image; The target interaction area in the second source video is determined, and based on the target interaction area, a second reference video and a second mask video are determined. The third video representation is determined using the second reference video and the second masked video; The third image representation is determined using the second target image; The third video representation and the third image representation are input into a pre-trained second video generation model to obtain the target generated video; The second video generation model is obtained by training the first video generation model using any of the training methods described in claims 1-12.

14. The method according to claim 13, wherein, The third video representation is determined using the second reference video and the second masked video; And using the second target image to determine the representation of the third image, including: Using a pre-trained encoder, the second reference video and the second target image are encoded respectively to obtain the second reference video encoding and the second target image encoding; and the second mask video is subjected to average pooling operation to obtain the second mask video features; Using a preset word segmenter, the second reference video encoding, the second target image encoding, and the second masked video features are segmented into words respectively; The third video representation is determined based on the second reference video encoding and the second mask video features after word segmentation; and the third image representation is determined based on the second target image encoding after word segmentation.

15. The method according to claim 13, wherein, The third video representation and the third image representation are input into a pre-trained second video generation model to obtain the target generated video, including: The third video representation and the third image representation are input into a pre-trained second video generation model, and the second video generation model outputs a video generation representation. Using a preset restorer, the generated video representation is restored into a continuous target video encoding; The target video is encoded and decoded using a pre-trained decoder to obtain the target generated video.

16. A training device for a video generation model, comprising: The first acquisition module is used to acquire the first source video and the first target image; The first video determination module is used to determine the target interaction part in the first source video, and determine the first reference video and the first mask video based on the target interaction part; The first representation determination module is used to determine the first video representation based on preset noise, the first reference video, and the first masked video; And based on the first target image, determine the first image representation; The first input module is used to input the first video representation and the first image representation into the first video generation model to be trained, and the first video generation model outputs the predicted video representation. The training module is used to train the first video generation model based on the predicted video representation and the first source video to obtain a second video generation model.

17. A video generation apparatus, comprising: The second acquisition module is used to acquire the second source video and the second target image; The second video determination module is used to determine the target interaction part in the second source video, and based on the target interaction part, determine the second reference video and the second mask video. The second representation determination module is used to determine a third video representation using the second reference video and the second mask video; and to determine a third image representation using the second target image. The second input module is used to input the third video representation and the third image representation into a pre-trained second video generation model to obtain the target generated video; The second video generation model is obtained by training the first video generation model using the training device described in claim 16.

18. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-15.

19. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-15.

20. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-15.

Images