Video generation method and device, electronic equipment and readable storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a video generation method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring an appearance reference video and a motion reference video; inputting the appearance reference video, the motion reference video and the noise map into a trained video generation model to obtain a target video output by the video generation model; wherein the video generation model comprises a motion processing branch, an appearance processing branch and a diffusion module, the motion processing branch is used for extracting the motion features of the motion reference video, the appearance processing branch is used for extracting the appearance features of the appearance reference video, and the diffusion module is used for fusing the motion features, the appearance features and the noise map to obtain a fused image; and generating a target video. The motion information of the object can be accurately controlled, the integrity of the appearance and the background of the object is kept, the appearance and the background of the object do not change along with the change of the motion information, and therefore the quality of the generated video is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, device, electronic device and readable storage medium. Background Art

[0002] Video generation refers to the process of creating or editing video content using computer technology and algorithms, especially artificial intelligence technology. This field covers a wide range of aspects, from simple video editing to complex dynamic scene generation. Current diffusion-based video editing models have made significant progress in editing video material properties, but diffusion-based video editing technology still has certain limitations in its ability to maintain the original appearance of the video subject and background when modifying the motion information in the video. Existing technologies mainly focus on motion alignment, but are still insufficient in the strict constraints of keeping the background and object content unchanged, resulting in the generated video deviating from expectations. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a video generation method, device, electronic device and readable storage medium to solve the problem in the prior art that it is difficult to accurately control the movement of an object and retain the appearance and background of the object at the same time.

[0004] In a first aspect of an embodiment of the present application, a video generation method is provided, the method comprising:

[0005] Obtain an appearance reference video and a motion reference video; obtain a noise map, input the appearance reference video, the motion reference video and the noise map into a trained video generation model for video generation, and obtain a target video output by the video generation model; wherein the video generation model includes a motion processing branch, an appearance processing branch and a diffusion module, the motion processing branch includes a motion adapter, the motion adapter is used to extract the global motion features and local motion features of the motion reference video, and determine the motion features of the motion reference video according to the global motion features and the local motion features, the appearance processing branch is used to extract the appearance features of the appearance reference video, the diffusion module is used to fuse the motion features, the appearance features and the noise map, and diffuse the fusion results to generate the target video.

[0006] According to a second aspect of the embodiments of the present application, a video generating device is provided, the device comprising:

[0007] The acquisition module is configured to acquire an appearance reference video and a motion reference video; the generation module is configured to acquire a noise map, input the appearance reference video, the motion reference video and the noise map into a trained video generation model for video generation, and obtain a target video output by the video generation model; wherein the video generation model includes a motion processing branch, an appearance processing branch and a diffusion module, the motion processing branch includes a motion adapter, the motion adapter is used to extract the global motion features and local motion features of the motion reference video, and determine the motion features of the motion reference video based on the global motion features and the local motion features, the appearance processing branch is used to extract the appearance features of the appearance reference video, the diffusion module is used to fuse the motion features, the appearance features and the noise map, and diffuse the fusion results to generate the target video.

[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0009] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0010] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0011] Obtain an appearance reference video and a motion reference video; obtain a noise map, input the appearance reference video, the motion reference video and the noise map into a trained video generation model for video generation, and obtain a target video output by the video generation model; wherein the video generation model includes a motion processing branch, an appearance processing branch and a diffusion module, the motion processing branch includes a motion adapter, the motion adapter is used to extract the global motion features and local motion features of the motion reference video, and determine the motion features of the motion reference video according to the global motion features and the local motion features, the appearance processing branch is used to extract the appearance features of the appearance reference video, the diffusion module is used to fuse the motion features, the appearance features and the noise map, and diffuse the fusion results to generate the target video. The dual-branch structure can simultaneously obtain the appearance features of the appearance reference video and the motion features of the motion reference video. At the same time, the global motion features and local motion features of the motion reference video can be extracted through the configuration of the motion adapter, so that the obtained motion features can maintain the global motion coherence while maintaining the precise control of local details. Then, the diffusion module is used to fuse the motion features, appearance features and noise map, so that the motion features are accurately used for motion control in the video generation process, and the appearance features are used to maintain the integrity of the appearance background, so as to generate high-quality and highly coherent videos, avoiding the problem in the prior art that it is difficult to accurately control the motion of objects and retain the original protagonist appearance and background at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 It is a flowchart of a video generation method provided in an embodiment of the present application;

[0014] Figure 2 is a structural schematic diagram of a motion adapter provided in an embodiment of the present application;

[0015] Figure 3 It is a structural schematic diagram of an appearance adapter provided in an embodiment of the present application;

[0016] Figure 4 is a structural schematic diagram of a diffusion submodule provided in an embodiment of the present application;

[0017] Figure 5 is a structural schematic diagram of a video generating device provided in an embodiment of the present application;

[0018] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0020] A video generation method, device, electronic device, and readable storage medium according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0021] Figure 1 FIG. 1 is a flow chart of a video generation method provided in an embodiment of the present application. Figure 1 As shown, the method includes:

[0022] S101, obtaining an appearance reference video and a motion reference video;

[0023] S102, obtaining a noise map, inputting an appearance reference video, a motion reference video and a noise map into a trained video generation model for video generation, and obtaining a target video output by the video generation model; wherein the video generation model includes a motion processing branch, an appearance processing branch and a diffusion module, the motion processing branch includes a motion adapter, the motion adapter is used to extract global motion features and local motion features of the motion reference video, and determine the motion features of the motion reference video based on the global motion features and the local motion features, the appearance processing branch is used to extract appearance features of the appearance reference video, the diffusion module is used to fuse the motion features, the appearance features and the noise map, and perform diffusion processing on the fusion result to generate a target video.

[0024] Specifically, the above-mentioned appearance reference video contains background information and appearance information of the target object (such as a person, animal, plant), wherein the background information includes environmental features, geographical location, weather conditions, etc., while the appearance information covers details such as the shape, color, texture, size, etc. of the target object. Through this information, the background information and appearance information of the appearance reference video can be more accurately identified. For example, in a video, the target object may be a specific bird. The background information includes the forest environment, season, time, etc. where the bird appears, while the appearance information includes the color of the bird, the texture of the feathers, the size of the body, etc. The motion reference video includes motion information. Exemplarily, the motion information can be the position change, speed, acceleration, and motion trajectory of the target object at different time points. For example, if the target object is a flying bird, the motion information will include its position change, flight speed, and direction between different frames. These motion information are crucial for generating models because they can help the model understand the dynamic behavior of the target object and reproduce these motion features when generating the target video.

[0025] It is understandable that noise maps can provide additional randomness to the model, making the generated videos more diverse. Noise maps are usually randomly generated images that are added to the model during training to increase the richness and unpredictability of the generated videos. In practical applications, noise maps can be designed to have different intensities and patterns to meet different generation requirements.

[0026] It can be understood that the video generation model includes a motion processing branch, an appearance processing branch, and a diffusion module. After the appearance reference video, the motion reference video, and the noise map are input into the video generation model, the motion processing branch of the model first extracts the global motion features and local motion features of the motion reference video through the motion adapter. The global motion features describe the overall motion trend of the target object, for example, the overall direction and speed of a bird flying in the forest. The local motion features focus on the motion details of the target object at a specific part, for example, the flapping frequency and amplitude of the bird's wings. These motion features are used to determine the motion features of the motion reference video, providing a motion basis for generating the target video. The appearance processing branch focuses on extracting the appearance features of the appearance reference video. These features cover details such as the color, shape, texture, size, and the background of the scene in which the target object is located, such as environmental features, geographical location, weather conditions, etc. For example, for a bird and the environmental features, geographical location, and weather conditions of the scene in which it is located. These appearance features are crucial to generating the visual effect of the target object in the target video because they ensure that the appearance of the target object in the generated video is consistent with the target object in the appearance reference video.

[0027] It can be understood that the diffusion module can fuse motion features, appearance features and noise maps, and perform iterative fusion. Diffusion processing is a process of gradually increasing noise and finally generating a target video. During the diffusion process, the model gradually combines the randomness in the noise map with motion features and appearance features, and generates a realistic video frame through a series of diffusion steps. The motion features, appearance features and noise map are fused by the diffusion module, and finally, the target video output by the video generation model will show the dynamic behavior of the target object under a specific background, with a high degree of realism and richness of details. That is, the motion information (motion features) of the motion reference video can be fused into the appearance reference video, so that the appearance reference video keeps the appearance information (appearance features) of the appearance reference video unchanged while fusing the motion information of the motion reference video, avoiding the problem that it is difficult to accurately control the motion of the object and retain the original protagonist appearance and background in the prior art.

[0028] According to the technical solution provided in the embodiment of the present application, an appearance reference video and a motion reference video are obtained; a noise map is obtained, and the appearance reference video, the motion reference video and the noise map are input into a trained video generation model for video generation, so as to obtain a target video output by the video generation model; wherein the video generation model includes a motion processing branch, an appearance processing branch and a diffusion module, the motion processing branch includes a motion adapter, the motion adapter is used to extract global motion features and local motion features of the motion reference video, and determine the motion features of the motion reference video based on the global motion features and the local motion features, the appearance processing branch is used to extract appearance features of the appearance reference video, the diffusion module is used to fuse the motion features, the appearance features and the noise map, and diffuse the fusion result to generate the target video. The dual-branch structure can simultaneously obtain the appearance features of the appearance reference video and the motion features of the motion reference video. At the same time, the global motion features and local motion features of the motion reference video can be extracted through the configuration of the motion adapter, so that the obtained motion features can maintain the global motion coherence while maintaining the precise control of local details. Then, the diffusion module is used to fuse the motion features, appearance features and noise map, so that the motion features can be accurately used for motion control in the video generation process, and the appearance features can be used to maintain the integrity of the appearance background, so as to generate high-quality and highly coherent videos, avoiding the problem that the existing technology is difficult to accurately control the motion of objects and retain the original protagonist appearance and background at the same time.

[0029] In some embodiments, before the appearance reference video, the motion reference video and the noise map are input into the trained video generation model for video generation, the method further includes: obtaining the appearance description text corresponding to the sample appearance reference video and the motion description text corresponding to the sample motion reference video, and determining the appearance text features corresponding to the appearance description text and the motion text features corresponding to the motion description text; extracting a skeleton map from the sample motion reference video using the motion processing branch to be trained, and performing feature processing on the motion text features and the skeleton map to obtain sample motion features; extracting a stroke map and a depth map from the sample appearance reference video using the appearance processing branch to be trained, and performing feature processing on the appearance text features, the stroke map and the depth map to obtain sample appearance features; obtaining a sample noise map, and inputting the appearance text features, the motion text features, the sample noise map, the sample motion features and the sample appearance features into the diffusion module for diffusion processing to obtain a predicted video frame output by the diffusion module; calculating the loss value between the predicted video frame and the real video frame, and using the loss value to update the parameters of the video generation model to be trained until the parameters of the video generation model to be trained converge, and using the video generation model to be trained with converged parameters as the video generation model.

[0030] Specifically, the sample appearance reference video and the sample motion reference video are samples in the sample training set, wherein the appearance description text is the text description of the sample appearance reference video, and the motion description text is the text description of the sample motion reference video. Determining the appearance description text and the motion description text can enhance the model's understanding ability for the sample appearance reference video and the sample motion reference video. The process of determining the appearance text features and the motion text features enables the model to better capture the key information in the video, which can be specifically obtained by encoding the appearance description text and the motion description text respectively using the CLIP text encoder.

[0031] It can be understood that the skeleton map can be used to characterize the key action postures in the sample motion reference video, while the stroke map and depth map are used to capture the contour details and spatial depth information in the sample appearance reference video, respectively. The skeleton map can effectively represent the motion state and action mode of the target object by extracting key points and the lines connecting these key points. The stroke map can highlight the outline of the target object and retain important shape information. The depth map can provide the model with the spatial relationship between the target object and the background, enhancing the three-dimensional sense and realism of the generated video. Among them, the skeleton map can be extracted using a skeleton extraction network, such as the PoseC3D model, the stroke map can be extracted using a stroke extraction network, such as the EPFL-VILAB / 4M edge extraction model, and the depth map can be extracted based on a depth map extraction network, such as the marigold depth model, which is not specifically limited in this example.

[0032] Furthermore, the motion text features and skeleton map are processed to obtain sample motion features, and the appearance text features, stroke map and depth map are processed to obtain sample appearance features, so as to further fuse the motion text features and skeleton map, and further fuse the appearance text features, stroke map and depth map, thereby generating richer and more accurate sample features. After obtaining the sample motion features and sample appearance features, they are combined with the appearance text features, motion text features and sample noise. Figure 1 The image is input into the diffusion module. The diffusion module generates a predicted video frame by gradually adding noise and performing a reverse diffusion process. The loss value between the predicted video frame and the real video frame is reconstructed to guide the parameter update of the video generation model. Specifically, the mean square error loss can be used to calculate the difference between the predicted image and the real image, which is used as a guide to update the model parameters. After multiple iterative training, the parameters of the video generation model to be trained gradually converge, and finally a model is obtained that can generate a video based on the appearance reference video, the motion reference video and the noise map, so that the motion features are accurately used for motion control in the video generation process, and the appearance features are used to maintain the integrity of the appearance background, so as to generate high-quality and highly coherent videos.

[0033] In some embodiments, feature processing is performed on motion text features and skeleton maps to obtain sample motion features, including: fusing the motion text features with the skeleton map to obtain motion control features; using at least one motion adapter to extract features from the motion control features to obtain sample motion features; feature processing is performed on appearance text features, stroke maps, and depth maps to obtain sample appearance features, including: fusing the appearance text features, stroke maps, and depth maps to obtain appearance control features; using at least one appearance adapter to extract features from the appearance control features to obtain sample appearance features.

[0034] Specifically, the motion text features are fused with the skeleton map to obtain the motion control features, and the appearance text features, the stroke map and the depth map are fused to obtain the appearance control features. In this example, the Controlnet network is preferably used for fusion, which can ensure the efficiency and accuracy of feature fusion. The Controlnet network is an advanced neural network architecture specially designed to handle multimodal data fusion tasks. Through the Controlnet network, the feature information of different modes can be effectively integrated to generate richer and more accurate control features.

[0035] It can be understood that after obtaining the motion control features and appearance control features, the next step is to use at least one motion adapter (preferably 5 in this example) and at least one appearance adapter (preferably 5 in this example) to extract features from these control features. By repeating the same unit structure, it can be ensured that the processing of motion features or appearance features remains consistent and stable throughout the adaptation process, which helps to reduce errors caused by structural differences and improve overall performance. The motion adapter and appearance adapter can be deep learning models, such as convolutional neural networks or recurrent neural networks, which can extract key information from the control features to form the final sample motion features and sample appearance features.

[0036] In addition, in order to further improve the effect of feature extraction, an attention mechanism can be used. The attention mechanism enables the model to pay more attention to the important parts of the input data, thereby improving the accuracy and robustness of feature extraction. For example, in the motion adapter, the attention mechanism can help the model focus on the motion-related parts of the skeleton map, while in the appearance adapter, the attention mechanism helps the model focus on the areas in the stroke map and depth map that have a greater impact on the appearance features.

[0037] According to the technical solution provided in the embodiment of the present application, the motion text feature is fused with the skeleton map to obtain the motion control feature; the motion control feature is extracted using at least one motion adapter to obtain the sample motion feature; and the appearance text feature, the stroke map and the depth map are fused to obtain the appearance control feature; the appearance control feature is extracted using at least one appearance adapter to obtain the sample appearance feature. The problem of insufficient motion and appearance feature extraction in traditional methods can be effectively solved, thereby obtaining richer motion control features and more refined appearance control features, thereby providing a basis for refined motion control and appearance control.

[0038] In some embodiments, the motion adapter includes a first global modeling branch and a first local modeling branch, and uses at least one motion adapter to extract features of motion control features to obtain sample motion features, including: using the cross-attention layer, the first time convolution layer and the first time attention layer successively set in the first global modeling branch to process the motion control features to obtain the first global feature, and adding the processing result of the cross-attention layer to the first global feature to obtain the global motion feature; using the second time convolution layer, the third time convolution layer and the second time attention layer successively set in the first local modeling branch to process the motion control features to obtain the first local feature, and adding the motion control features to the first local feature to obtain the local motion feature; adding the global motion feature and the local motion feature to obtain the sample motion feature.

[0039] In order to better understand this embodiment, Figure 2 This embodiment is described as follows. Figure 2 As shown:

[0040] Specifically, the motion adapter includes a first global modeling branch and a first local modeling branch, wherein the first global modeling branch includes a cross-attention layer, a first temporal convolution layer, and a first temporal attention layer that are continuously set. After the motion control feature is input into the first global modeling branch, the processing of the cross-attention layer can focus on the key motion clues in the sample motion video and enhance the control accuracy. Then, the output feature of the cross-attention layer passes through the first temporal convolution layer to further refine the feature representation. Finally, the first temporal attention layer optimizes the feature to obtain the first global feature, ensuring the coherence and consistency of the global feature. Furthermore, the output feature of the cross-attention layer is enhanced by the feature fusion with the first global feature through a residual connection. In this way, the first global modeling branch can extract features related to the overall motion pattern to form a global motion feature.

[0041] It can be understood that the first local modeling branch focuses on capturing detailed motion in the local area. The motion control features are further refined through the second and third temporal convolutional layers that are set consecutively to extract local motion features. The second temporal attention layer is used to optimize these local features to ensure that they can accurately reflect the key action details in the motion. Finally, the initial motion control features are added to the first local features to obtain local motion features to retain the information of the original motion control features and enhance the expressiveness of the local features.

[0042] In addition, in order to further improve the comprehensive performance of the sample motion features, the global motion features and local motion features are fused by addition, so that the global and local information can complement each other, thereby obtaining a more comprehensive sample motion feature. This fusion strategy not only retains the coherence of the global motion, but also enhances the expression of the details of the local motion, so that the final sample motion features can maintain the coherence of the global motion while maintaining the precise control of the local details. Finally, the final feature output of a single module is formed, which can be used as the input of the next module for further extraction.

[0043] In this way, more comprehensive sample motion features can be obtained. This method of combining global and local information can not only capture the overall trend of the motion, but also focus on the key details in the motion, thus providing richer and more accurate feature information for subsequent motion control.

[0044] In some embodiments, the appearance adapter includes a second global modeling branch and a second local modeling branch, and uses at least one appearance adapter to extract features of the appearance control features to obtain sample appearance features, including: using the spatial convolution layer, the fourth temporal convolution layer and the third temporal attention layer successively set in the second global modeling branch to process the appearance control features to obtain second global features, and adding the processing result of the spatial convolution layer to the second global features to obtain global appearance features; using the fifth temporal convolution layer, the sixth temporal convolution layer and the fourth temporal attention layer successively set in the second local modeling branch to process the motion control features to obtain second local features, and adding the appearance control features to the second local features to obtain local appearance features; adding the global appearance features and the local appearance features to obtain sample appearance features.

[0045] In order to better understand this embodiment, Figure 3 This embodiment is described as follows. Figure 3 As shown:

[0046] Specifically, the appearance adapter includes a second global modeling branch and a second local modeling branch, wherein the second global modeling branch includes a continuously arranged spatial convolution layer, a fourth temporal convolution layer, and a third temporal attention layer, and the second global modeling branch focuses on capturing appearance features from a macroscopic perspective. After the appearance control feature is input into the second global modeling branch, the appearance control feature is modeled through the spatial convolution layer, which can capture the spatial distribution and structural information of the appearance control feature, and further passes through the fourth temporal convolution layer to capture the feature changes in the time series, and then passes through the third temporal attention layer to identify and respond to key appearance changes in the video. Through the combination of these layers, the appearance control feature is processed to extract features related to the overall appearance pattern to form a second global feature. The processing result of the spatial convolution layer is further fused with the second global feature through a residual connection to achieve enhanced fusion of features, further enhance the expressiveness of the features, and obtain the final global appearance feature.

[0047] In addition, the second local modeling branch focuses on capturing the appearance details in the local area. Through the fifth and sixth temporal convolutional layers set consecutively, the appearance control features are further refined, focusing on extracting the temporal changes of local appearance features. The fourth temporal attention layer is used to optimize these local appearance features, ensure the accuracy of the local appearance, and ensure that the obtained second local features can accurately reflect the key details in the appearance. Finally, the appearance control features are added to the second local features to obtain the local appearance features, so as to retain the original appearance control feature information while enhancing the expressiveness of the local appearance features.

[0048] It can be understood that further adding the global appearance features and the local appearance features can obtain more comprehensive sample appearance features. This method of combining global and local information can not only capture the overall trend of the appearance, but also focus on the key details in the appearance, thereby providing richer and more accurate feature information for subsequent appearance control.

[0049] In this way, a more comprehensive sample appearance feature can be obtained. This method of combining global and local information can not only capture the overall appearance information, but also focus on the key details in the appearance, thus providing richer and more accurate feature information for subsequent appearance control.

[0050] In some embodiments, the diffusion module includes a plurality of diffusion sub-modules arranged in series; the appearance text features, motion text features, sample noise maps, sample motion features and sample appearance features are input into the diffusion module for diffusion processing to obtain a predicted video frame output by the diffusion module, including: inputting the appearance text features, motion text features and sample noise maps into the first diffusion sub-module for diffusion processing, and inputting the sample motion features and sample appearance features into each diffusion sub-module for diffusion processing; using the output of the first diffusion sub-module as the output of the next diffusion sub-module for diffusion processing until the predicted video frame output by the last diffusion sub-module is obtained.

[0051] Specifically, each diffusion submodule in the diffusion module can gradually fuse and diffuse the input feature information into the entire video frame. In the first diffusion submodule, the appearance text features, motion text features and sample noise maps are first input, and a series of diffusion processes are performed to capture the global features of appearance and motion. Subsequently, the sample motion features and sample appearance features are input into each diffusion submodule for further diffusion fusion to ensure that the appearance details and dynamic characteristics of the video frame are fully expressed. The connection between the diffusion submodules is designed as a cascade structure, so that each submodule can use the output results of the previous submodule during the processing process, thereby gradually optimizing and refining the quality of the predicted video frame, wherein each diffusion submodule can adopt a U-Net structure. If the U-Net structure is adopted, the spatial Transformer in the U-Net is expanded to a 3D Transformer by attaching a temporal self-attention layer, so as to enhance the model's ability to capture temporal information in the video sequence.

[0052] In some embodiments, each diffusion submodule includes a spatial convolution layer, a self-attention layer, an appearance cross-attention layer, a motion cross-attention layer, and a temporal attention layer; the appearance text features, motion text features, and noise map are input into the first diffusion submodule for iterative fusion, and the motion features and appearance features are input into each diffusion submodule for iterative fusion, including: using continuously set spatial convolution layers, self-attention layers, appearance cross-attention layers, motion cross-attention layers, and temporal attention layers to perform diffusion processing on the appearance text features, motion text features, and sample noise maps, and during the diffusion processing, the sample appearance features are input into the appearance cross-attention layer of each diffusion submodule for diffusion fusion, and the sample motion features are input into the motion cross-attention layer of each diffusion submodule for diffusion fusion.

[0053] In order to better understand this embodiment, Figure 4 This embodiment is described as follows. Figure 4 As shown:

[0054] Continuing from the above example, each diffusion submodule includes a spatial convolution layer, a self-attention layer, an appearance cross-attention layer, a motion cross-attention layer, and a temporal attention layer. In each diffusion submodule, the spatial convolution layer is responsible for capturing spatial feature information, extracting local features through convolution operations, and downsampling feature maps. The self-attention layer is used to enhance the model's perception of global information. By calculating the relationship between each position in the feature map, the model can better understand the correlation between different regions. The appearance cross-attention layer and the motion cross-attention layer are used to interact the sample appearance features and sample motion features with the features of the current diffusion submodule, respectively, so that the model can combine appearance and motion information for more accurate predictions. The temporal attention layer focuses on capturing feature changes in the time series to ensure that the model can capture the dynamic characteristics between video frames.

[0055] It can be understood that in the iterative fusion process, each diffusion submodule first receives the output feature map from the previous submodule, and then performs feature extraction and downsampling through the spatial convolution layer. Next, the self-attention layer enhances the feature map with global information, so that the model can better understand the contextual information of the entire feature map. The appearance cross-attention layer and the motion cross-attention layer fuse the sample appearance features and sample motion features with the current feature map respectively, and the attention mechanism enables the model to focus on the key information of appearance and motion. Finally, the temporal attention layer processes the fused feature map in the temporal dimension to capture the dynamic changes between video frames.

[0056] In this way, through this iterative fusion method, each diffusion submodule can gradually integrate the appearance and motion information into the entire video frame, so that the final generated predicted video frame not only has accurate appearance details, but also can show natural dynamic characteristics. The cascade structure of each diffusion submodule ensures that the feature information can be transmitted and optimized layer by layer, so that the entire diffusion module can generate high-quality predicted video frames.

[0057] It should be noted that the training process of the video generation model to be trained can be divided into two stages. In the first stage, the temporal attention layer of each diffusion submodule is frozen, and the spatial convolution layer, self-attention layer, appearance cross-attention layer, motion cross-attention layer, appearance adapter and motion adapter are trained; in the second stage, the spatial convolution layer, self-attention layer, appearance cross-attention layer and appearance adapter of each diffusion submodule are frozen, and the motion cross-attention layer, temporal attention layer and motion adapter are trained; in this way, by freezing and training different modules in stages, computing resources can be used more efficiently, because each stage can focus on optimizing a specific part of the network. In addition, the instability and potential gradient vanishing or exploding problems in the early stage of training can be reduced. At the same time, each module is fully trained at different stages, which helps the model better capture motion features and appearance features when processing video generation tasks, thereby improving the quality of generated videos.

[0058] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of the present application.

[0059] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.

[0060] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0061] Figure 5 Schematic diagram of the structure of a video generation device provided in an embodiment of the present application. Figure 5 As shown, the device comprises:

[0062] An acquisition module 501 is configured to acquire an appearance reference video and a motion reference video;

[0063] The generation module 502 is configured to obtain a noise map, input the appearance reference video, the motion reference video and the noise map into a trained video generation model for video generation, and obtain a target video output by the video generation model; wherein the video generation model includes a motion processing branch, an appearance processing branch and a diffusion module, the motion processing branch includes a motion adapter, the motion adapter is used to extract the global motion features and local motion features of the motion reference video, and determine the motion features of the motion reference video based on the global motion features and the local motion features, the appearance processing branch is used to extract the appearance features of the appearance reference video, the diffusion module is used to fuse the motion features, the appearance features and the noise map, and diffuse the fusion results to generate the target video.

[0064] In some embodiments, the video generation device also includes a training module 503, which is configured to obtain an appearance description text corresponding to a sample appearance reference video and a motion description text corresponding to a sample motion reference video, and determine an appearance text feature corresponding to the appearance description text and a motion text feature corresponding to the motion description text; use a motion processing branch to be trained to extract a skeleton map from the sample motion reference video, and perform feature processing on the motion text feature and the skeleton map to obtain a sample motion feature; use an appearance processing branch to be trained to extract a stroke map and a depth map from the sample appearance reference video, and perform feature processing on the appearance text feature, the stroke map and the depth map to obtain a sample appearance feature; obtain a sample noise map, input the appearance text feature, the motion text feature, the sample noise map, the sample motion feature and the sample appearance feature into a diffusion module for diffusion processing, and obtain a predicted video frame output by the diffusion module; calculate a loss value between the predicted video frame and the real video frame, and use the loss value to update the parameters of the video generation model to be trained until the parameters of the video generation model to be trained converge, and use the video generation model to be trained with converged parameters as the video generation model.

[0065] In some embodiments, the training module 503 is also configured to fuse the motion text features with the skeleton map to obtain motion control features; use at least one motion adapter to extract features of the motion control features to obtain sample motion features; fuse the appearance text features, the stroke map and the depth map to obtain appearance control features; use at least one appearance adapter to extract features of the appearance control features to obtain sample appearance features.

[0066] In some embodiments, the training module 503 is also configured to process the motion control feature using the cross-attention layer, the first temporal convolution layer and the first temporal attention layer that are successively set in the first global modeling branch to obtain a first global feature, and add the processing result of the cross-attention layer to the first global feature to obtain a global motion feature; process the motion control feature using the second temporal convolution layer, the third temporal convolution layer and the second temporal attention layer that are successively set in the first local modeling branch to obtain a first local feature, and add the motion control feature to the first local feature to obtain a local motion feature; add the global motion feature and the local motion feature to obtain a sample motion feature.

[0067] In some embodiments, the training module 503 is also configured to process the appearance control feature using the spatial convolution layer, the fourth temporal convolution layer and the third temporal attention layer that are successively arranged in the second global modeling branch to obtain a second global feature, and add the processing result of the spatial convolution layer to the second global feature to obtain a global appearance feature; process the motion control feature using the fifth temporal convolution layer, the sixth temporal convolution layer and the fourth temporal attention layer that are successively arranged in the second local modeling branch to obtain a second local feature, and add the appearance control feature to the second local feature to obtain a local appearance feature; add the global appearance feature and the local appearance feature to obtain a sample appearance feature.

[0068] In some embodiments, the training module 503 is also configured to input the appearance text features, motion text features and sample noise map into the first diffusion sub-module for diffusion processing, and input the sample motion features and sample appearance features into each diffusion sub-module for diffusion processing; the output of the first diffusion sub-module is used as the output of the next diffusion sub-module for diffusion processing until the predicted video frame output by the last diffusion sub-module is obtained.

[0069] In some embodiments, the training module 503 is also configured to perform diffusion processing on the appearance text features, motion text features, and sample noise maps using continuously arranged spatial convolution layers, self-attention layers, appearance cross-attention layers, motion cross-attention layers, and temporal attention layers, and during the diffusion processing, the sample appearance features are input into the appearance cross-attention layer of each diffusion sub-module for diffusion fusion, and the sample motion features are input into the motion cross-attention layer of each diffusion sub-module for diffusion fusion.

[0070] Figure 6 Schematic diagram of an electronic device 6 provided in an embodiment of the present application. Figure 6As shown, the electronic device 6 of this embodiment includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 601 executes the computer program 603, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0071] The electronic device 6 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 6 may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will appreciate that Figure 6 The electronic device 6 is merely an example and does not limit the electronic device 6 . The electronic device 6 may include more or less components than those shown in the figure, or different components.

[0072] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0073] The memory 602 may be an internal storage unit of the electronic device 6, for example, a hard disk or memory of the electronic device 6. The memory 602 may also be an external storage device of the electronic device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. The memory 602 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0074] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0075] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0076] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A video generation method, characterized in that: include: Get appearance reference video and motion reference video; Obtain a noise map, input the appearance reference video, the motion reference video, and the noise map into a trained video generation model to perform video generation, and obtain a target video output by the video generation model; Among them, the video generation model includes a motion processing branch, an appearance processing branch and a diffusion module. The motion processing branch includes a motion adapter. The motion adapter is used to extract the global motion features and local motion features of the motion reference video, and determine the motion features of the motion reference video based on the global motion features and the local motion features. The appearance processing branch is used to extract the appearance features of the appearance reference video. The diffusion module is used to fuse the motion features, the appearance features and the noise map, and perform diffusion processing on the fusion results to generate the target video.

2. The method according to claim 1, characterized in that Before inputting the appearance reference video, the motion reference video and the noise map into the trained video generation model for video generation, the method further includes: Acquire an appearance description text corresponding to a sample appearance reference video and a motion description text corresponding to a sample motion reference video, and determine an appearance text feature corresponding to the appearance description text and a motion text feature corresponding to the motion description text; Extracting a skeleton graph from the sample motion reference video using the motion processing branch to be trained, and performing feature processing on the motion text feature and the skeleton graph to obtain a sample motion feature; Extracting a stroke map and a depth map from the sample appearance reference video using the to-be-trained appearance processing branch, and performing feature processing on the appearance text features, the stroke map, and the depth map to obtain sample appearance features; Acquire a sample noise map, input the appearance text feature, the motion text feature, the sample noise map, the sample motion feature and the sample appearance feature into a diffusion module for diffusion processing, and obtain a predicted video frame output by the diffusion module; The loss value between the predicted video frame and the real video frame is calculated, and the parameters of the video generation model to be trained are updated using the loss value until the parameters of the video generation model to be trained converge, and the video generation model to be trained with converged parameters is used as the video generation model.

3. The method according to claim 2, characterized in that The step of performing feature processing on the motion text feature and the skeleton graph to obtain a sample motion feature includes: The motion text feature is fused with the skeleton graph to obtain a motion control feature; Using at least one motion adapter to extract the motion control feature to obtain the sample motion feature; The performing feature processing on the appearance text feature, the stroke map, and the depth map to obtain the sample appearance feature includes: Fusing the appearance text feature, the stroke map, and the depth map to obtain an appearance control feature; At least one appearance adapter is used to extract the appearance control feature to obtain the sample appearance feature.

4. The method according to claim 3, characterized in that The motion adapter includes a first global modeling branch and a first local modeling branch. The step of extracting the motion control feature using at least one motion adapter to obtain the sample motion feature includes: Processing the motion control feature using a cross attention layer, a first temporal convolution layer, and a first temporal attention layer that are successively arranged in the first global modeling branch to obtain a first global feature, and adding a processing result of the cross attention layer to the first global feature to obtain a global motion feature; Processing the motion control feature using a second temporal convolution layer, a third temporal convolution layer, and a second temporal attention layer that are successively arranged in the first local modeling branch to obtain a first local feature, and adding the motion control feature to the first local feature to obtain a local motion feature; The global motion feature and the local motion feature are added together to obtain the sample motion feature.

5. The method according to claim 3, characterized in that: The appearance adapter includes a second global modeling branch and a second local modeling branch, and the use of at least one appearance adapter to extract the appearance control feature to obtain the sample appearance feature includes: Processing the appearance control feature using the spatial convolution layer, the fourth temporal convolution layer, and the third temporal attention layer that are continuously arranged in the second global modeling branch to obtain a second global feature, and adding the processing result of the spatial convolution layer to the second global feature to obtain a global appearance feature; Processing the motion control feature using a fifth temporal convolution layer, a sixth temporal convolution layer, and a fourth temporal attention layer that are successively arranged in the second local modeling branch to obtain a second local feature, and adding the appearance control feature to the second local feature to obtain a local appearance feature; The global appearance feature and the local appearance feature are added to obtain the sample appearance feature.

6. The method according to claim 2, characterized in that The diffusion module includes a plurality of diffusion submodules arranged in series; the appearance text feature, the motion text feature, the sample noise map, the sample motion feature and the sample appearance feature are input into the diffusion module for diffusion processing to obtain a predicted video frame output by the diffusion module, including: Input the appearance text feature, the motion text feature and the sample noise map into a first diffusion submodule for diffusion processing, and input the sample motion feature and the sample appearance feature into each diffusion submodule for diffusion processing; The output of the first diffusion submodule is used as the output of the next diffusion submodule for diffusion processing until the predicted video frame output by the last diffusion submodule is obtained.

7. The method according to claim 6, characterized in that Each of the diffusion submodules includes a spatial convolution layer, a self-attention layer, an appearance cross-attention layer, a motion cross-attention layer, and a temporal attention layer; The step of inputting the appearance text feature, the motion text feature and the sample noise map into a first diffusion submodule for diffusion processing, and inputting the sample motion feature and the sample appearance feature into each diffusion submodule for diffusion processing comprises: The appearance text features, the motion text features, and the sample noise map are diffused using the continuously arranged spatial convolution layer, the self-attention layer, the appearance cross-attention layer, the motion cross-attention layer, and the temporal attention layer. During the diffusion processing, the sample appearance features are input into the appearance cross-attention layer of each diffusion submodule for diffusion fusion, and the sample motion features are input into the motion cross-attention layer of each diffusion submodule for diffusion fusion.

8. A video generating device, characterized in that: include: An acquisition module, configured to acquire an appearance reference video and a motion reference video; A generation module is configured to obtain a noise map, input the appearance reference video, the motion reference video and the noise map into a trained video generation model for video generation, and obtain a target video output by the video generation model; wherein the video generation model includes a motion processing branch, an appearance processing branch and a diffusion module, the motion processing branch includes a motion adapter, the motion adapter is used to extract the global motion features and local motion features of the motion reference video, and determine the motion features of the motion reference video according to the global motion features and the local motion features, the appearance processing branch is used to extract the appearance features of the appearance reference video, and the diffusion module is used to fuse the motion features, the appearance features and the noise map, and diffuse the fusion result to generate the target video.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images