Video generation method and device, electronic equipment and readable storage medium
By combining style and motion noise graphs, the parallel processing mechanism of multiple subnets is used to solve the problem of insufficient spatial and temporal continuity of videos in existing text-to-video generation, and the generated videos are smoother and more natural, improving the overall quality.
Patent Information
- Application Number
- CN202411022528.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-29
AI Technical Summary
The existing text-to-video generation algorithms are insufficient in the space-time continuity of the video, resulting in the generated videos often flickering and incoherent, and it is difficult to capture the motion details and rhythm in the video, resulting in the lack of fluency and realism in the generated videos.
Using the diffusion model, by obtaining style reference diagrams and text descriptions, extracting style text descriptions and image features, combining video frame noise diagrams and motion noise diagrams, and using the parallel processing mechanism of multiple subnetworks to enhance the content consistency between video frames and Movement changes, generate predicted video frames, reduce flicker and artifacts.
The generated video has stronger visual continuity, more natural movement, and improved overall quality, maintains video style consistency, and reduces flicker and artifact phenomena.
Smart Images

Figure CN119052529A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation method, device, electronic device and readable storage medium. Background Art
[0002] With the rapid development of diffusion models, text-to-video generation technology has become a hot topic of research and is widely used in entertainment, advertising, education, surveillance and other fields. Although the current text-to-video generation algorithm has made some progress in the reproduction of visual content, it is still insufficient in the spatiotemporal continuity of the video, resulting in flickering and incoherence in the generated video. At the same time, most of the existing text-to-video generation methods focus on extracting visual information from text descriptions and trying to convert them into video frames. These methods often ignore the coherence of video content in the temporal dimension and the rationality of dynamic changes. In addition, when processing videos, image-based diffusion models often have difficulty capturing the motion details and rhythm in the video, resulting in the lack of smoothness and realism in the generated video. Summary of the invention
[0003] In view of this, the embodiments of the present application provide a video generation method, device, electronic device and readable storage medium to solve the problems of flickering and incoherence in the videos generated in the prior art, and the lack of visual smoothness and realism in the generated videos.
[0004] In a first aspect of an embodiment of the present application, a video generation method is provided, the method comprising:
[0005] A style reference image and a text description are obtained; based on the style reference image, a style text description and a style image feature are obtained; based on the style text description and the text description, a target text description is obtained, and feature processing is performed on the target text description to obtain target text features; a video frame noise map and a motion noise map for generating a target video are determined, and the style image features, the target text features, the video frame noise map and the motion noise map are input into a diffusion model to obtain a predicted video frame output by the diffusion model, and a target video is obtained based on the predicted video frame; wherein the diffusion model includes a plurality of sub-networks arranged in succession, each sub-network includes a content processing branch, a motion processing branch and a fusion module, the content processing branch performs diffusion processing based on the style image features and the video frame noise map, the motion processing branch performs diffusion processing based on the motion noise map, and the fusion module is used to fuse the diffusion processing results of the content processing branch and the motion processing branch with the target text features, and use the fusion results as the input of the two branches in the next sub-network.
[0006] According to a second aspect of the embodiments of the present application, a video generation device is provided, the method comprising:
[0007] The acquisition module is configured to acquire a style reference image and a text description; the feature module is configured to obtain a style text description and a style image feature based on the style reference image, obtain a target text description based on the style text description and the text description, and perform feature processing on the target text description to obtain a target text feature; the generation module is configured to determine a video frame noise map and a motion noise map for generating a target video, input the style image feature, the target text feature, the video frame noise map and the motion noise map into a diffusion model, obtain a predicted video frame output by the diffusion model, and obtain a target video based on the predicted video frame; wherein the diffusion model includes a plurality of sub-networks arranged in succession, each sub-network includes a content processing branch, a motion processing branch and a fusion module, the content processing branch performs diffusion processing based on the style image feature and the video frame noise map, the motion processing branch performs diffusion processing based on the motion noise map, and the fusion module is used to fuse the diffusion processing results of the content processing branch and the motion processing branch with the target text feature, and use the fusion result as the input of the two branches in the next sub-network.
[0008] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.
[0009] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0010] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0011] According to the technical solution provided in the embodiment of the present application, a style reference image and a text description are obtained; based on the style reference image, a style text description and a style image feature are obtained; based on the style text description and the text description, a target text description is obtained, and feature processing is performed on the target text description to obtain target text features; a video frame noise map and a motion noise map for generating a target video are determined, and the style image features, the target text features, the video frame noise map, and the motion noise map are input into a diffusion model to obtain a predicted video frame output by the diffusion model, and based on the predicted video frame, a target video is obtained. Through the parallel processing mechanism of the content processing branch and the motion processing branch in each subnetwork of the diffusion model, the content consistency between video frames can be enhanced, and the motion changes in the video can be enriched, thereby generating a video with stronger visual continuity and more natural motion. At the same time, through the parallel processing mechanism of the content processing branch and the motion processing branch, the video flicker and artifact phenomenon can be effectively reduced while maintaining the consistency of the video style, thereby improving the overall quality of the generated video. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0013] Figure 1 It is a flowchart of a video generation method provided in an embodiment of the present application;
[0014] Figure 2 It is a flowchart of another video generation method provided in an embodiment of the present application;
[0015] Figure 3 It is a schematic diagram of a sub-network workflow provided by an embodiment of the present application;
[0016] Figure 4 is a structural schematic diagram of a video generating device provided in an embodiment of the present application;
[0017] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0018] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0019] A video generation method, device, electronic device, and readable storage medium according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0020] Figure 1 FIG. 1 is a flow chart of a video generation method provided in an embodiment of the present application. Figure 1 As shown, the method includes:
[0021] S101, obtaining a style reference image and a text description;
[0022] S102, obtaining a style text description and a style image feature based on the style reference image, obtaining a target text description based on the style text description and the text description, and performing feature processing on the target text description to obtain a target text feature;
[0023] S103, determining a video frame noise map and a motion noise map for generating a target video, inputting the style image features, the target text features, the video frame noise map and the motion noise map into a diffusion model, obtaining a predicted video frame output by the diffusion model, and obtaining a target video based on the predicted video frame;
[0024] Among them, the diffusion model includes multiple sub-networks set up continuously, each sub-network includes a content processing branch, a motion processing branch and a fusion module. The content processing branch performs diffusion processing based on style image features and video frame noise maps, and the motion processing branch performs diffusion processing based on motion noise maps. The fusion module is used to fuse the diffusion processing results of the content processing branch and the motion processing branch with the target text features, and use the fusion results as the input of the two branches in the next sub-network.
[0025] It can be understood that the above-mentioned style reference image is an image containing the style or subject that the user hopes to imitate in the generated target video. The style reference image can be one or more paintings, one or more photos or any other form of image, and the text description can be one or more text paragraphs used to describe the content, scenes, actions, etc. of the target video.
[0026] In some examples, after obtaining the style reference map, the style reference map can be input into the BLIP2 image encoder, and the Q-Former in the BLIP2 image encoder is used to ask the image style of the image to obtain the style text description, and the style reference map is input into the image style encoder (preferably the VIT image encoder) for processing to obtain the corresponding style image features, and after obtaining the style text description, it is spliced with the text description to obtain the target text description, and the CLIP text encoder is used to encode the target text description to obtain the target text features;
[0027] It should be noted that after obtaining the target text features and style image features, it is necessary to determine the video frame noise map and motion noise map used to generate the target video. The video frame noise map is an image obtained by adding noise to the video frame, which is used to simulate or analyze the noise interference that may be generated by the video signal during transmission or recording, while the motion noise map is an image representation of the noise or artifacts generated by motion in the video frame. This noise is caused by factors such as rapid motion, scene switching, and changes in lighting, and may appear as blur, ghosting, flickering, and other phenomena. By extracting and analyzing the motion noise map, we can better understand the impact of motion on video quality, and how to reduce or eliminate this noise through algorithms or techniques;
[0028] In some examples, after determining the video frame noise map and the motion noise map, the style image features, the target text features, the video frame noise map, and the motion noise map are combined. Figure 1The data are input into the diffusion model at the same time. The diffusion model simulates the data generation process by gradually adding noise, and learns how to recover the original data from the noise during the training process. Each sub-network in the diffusion model can be regarded as a stage of the model, and each stage will learn how to gradually generate clearer video frames from more blurred predicted video frames. In this embodiment, the diffusion model is designed as a deep neural network containing multiple continuous sub-networks. At the same time, the diffusion model adopts a 3D-UNet structure. After multiple iterations, the diffusion model gradually removes noise and generates a set of predicted video frames. In this embodiment, the structure of each sub-module in the 3D-UNet structure is transformed so that the sub-networks of this application all include content processing branches, motion processing branches and fusion modules.
[0029] Continuing from the above example, the content processing branch is mainly responsible for processing style image features and video frame noise maps. The motion processing branch focuses on processing motion noise maps. The fusion module is responsible for fusing the diffusion processing results of the two branches with the target text features. The fusion result is used as the input of the next sub-network to continue the diffusion processing. Among them, the content processing branch and the motion processing branch adopt a parallel network structure, which can simultaneously process the spatiotemporal dimensions of the video, improve the consistency of motion and content between video frames, and enhance the diversity of the video. After continuous processing by multiple sub-networks, the diffusion model finally outputs a predicted video frame sequence. This predicted video frame sequence not only conforms to the text description provided by the user, but also incorporates the style characteristics of the style reference map provided by the user, and includes the motion changes introduced by the motion noise map. By arranging and combining these predicted video frames in chronological order, a complete target video can be obtained.
[0030] According to the technical solution provided in the embodiment of the present application, a style reference image and a text description are obtained; based on the style reference image, a style text description and a style image feature are obtained; based on the style text description and the text description, a target text description is obtained, and feature processing is performed on the target text description to obtain target text features; a video frame noise map and a motion noise map for generating a target video are determined, and the style image features, the target text features, the video frame noise map, and the motion noise map are input into a diffusion model to obtain a predicted video frame output by the diffusion model, and based on the predicted video frame, a target video is obtained. Through the parallel processing mechanism of the content processing branch and the motion processing branch in each subnetwork of the diffusion model, the content consistency between video frames can be enhanced, and the motion changes in the video can be enriched, thereby generating a video with stronger visual continuity and more natural motion. At the same time, through the parallel processing mechanism of the content processing branch and the motion processing branch, the video flicker and artifact phenomenon can be effectively reduced while maintaining the consistency of the video style, thereby improving the overall quality of the generated video.
[0031] In some embodiments, the style image features, the target text features, the video frame noise map, and the motion noise map are input into the diffusion model to obtain the predicted video frame output by the diffusion model, including:
[0032] The style image features and the target text features are input into each sub-network of the diffusion model; the video frame noise map is input into the content processing branch of the first sub-network for feature processing and added to the style image features to obtain the content features, and the motion noise map is input into the motion processing branch of the first sub-network for feature processing to obtain the motion features; based on the target text features, content features and motion features, a fusion module is used to obtain content fusion features and motion fusion features; the content fusion features and motion fusion features are respectively input into the content processing branch and motion processing branch of the next sub-network, and the feature processing steps and the processing steps of the fusion module are repeated until the content fusion features and motion fusion features output by the last sub-network are obtained, and the predicted video frame is obtained based on the content fusion features and motion fusion features output by the last sub-network.
[0033] Specifically, the style image features and the target text features will be simultaneously input into each sub-network of the diffusion model. That is, these two features will be processed in each sub-network. The video frame noise map is input into the content processing branch of the first sub-network for feature processing. During the output processing of the content processing branch, the initial result obtained will be added to the style image features, and the final output result of the content processing branch is the content feature. The motion noise map is input into the motion processing branch of the first sub-network for feature processing to obtain motion features. After obtaining the content features, motion features and target text features, they are fused through the fusion module. The obtained content fusion features and motion fusion features will be used as the input of the next sub-network, and the same processing flow will be continued. For example, after the first sub-network inputs the content fusion features and motion fusion features, the content fusion features will be used as the input of the content processing branch in the second sub-network, and the motion fusion features will be used as the input of the motion processing branch in the second sub-network. The content processing branch of the second sub-network processes the content fusion features and adds them to the style image features to obtain the content features. The motion processing branch of the second sub-network processes the motion fusion features to obtain the motion features. The content features, motion features and target text features are then fused through the fusion module, and the fusion results are used as the input of the third sub-network. This continues until the content fusion features and motion fusion features output by the last sub-network are obtained.
[0034] In each sub-network of the diffusion model, content features, motion features, and target text features are fused through the fusion module. The function of the fusion module is to fuse features from different branches to generate a more comprehensive and accurate predicted video frame. Through the fusion module, content features, motion features, and target text features can complement and enhance each other, so that the predicted video frame can maintain content consistency and style consistency while also including the information in the text description provided by the user and the motion changes introduced by the motion noise map.
[0035] After continuous processing by multiple sub-networks, the diffusion model finally outputs a predicted video frame sequence. This predicted video frame sequence not only conforms to the text description provided by the user, but also incorporates the style characteristics of the style reference map provided by the user, and includes the motion changes introduced by the motion noise map. By arranging and combining these predicted video frames in chronological order, a complete target video can be obtained. This target video performs well in terms of visual continuity, natural motion, style consistency, and content accuracy, and can meet the needs and expectations of users.
[0036] In some embodiments, both the content processing branch and the motion processing branch include a spatial convolution layer and a self-attention layer; the video frame noise map is input into the content processing branch of the first subnetwork for feature processing, and is added to the style image feature to obtain a content feature, including: performing convolution processing on the video frame noise map using a spatial convolution layer; processing the convolved video frame noise map using a self-attention layer to obtain an attention feature, and adding the attention feature to the style image feature to obtain a content feature; the motion noise map is input into the motion processing branch of the first subnetwork for feature processing to obtain a motion feature, including: performing convolution processing on the motion noise map using a spatial convolution layer; processing the convolved motion noise map using a self-attention layer to obtain a motion feature.
[0037] Specifically, the design of the content processing branch and the motion processing branch both take into account the spatial characteristics and temporal dependencies of the video frames. The spatial convolution layer is used to capture local spatial information in the video frame, such as the edges and textures of objects. The self-attention layer is used to model long-distance dependencies in the video frame, helping the model understand the overall structure and dynamic changes in the video frame. This allows the two branches to consider both local details and global structures when processing video frames, thereby generating more accurate and natural predicted video frames.
[0038] It can be understood that in the content processing branch, the video frame noise map is first convolved through the spatial convolution layer to extract the basic features in the video frame. Then, these features are sent to the self-attention layer for further processing. The self-attention layer weights the features, emphasizes important features and suppresses unimportant features, thereby obtaining a more accurate and meaningful feature representation. Finally, these features processed by the self-attention layer (i.e., attention features) are added to the style image features to obtain content features. This addition operation incorporates the features of the style image into the video frame, so that the predicted video frame not only maintains content consistency, but also incorporates the style characteristics provided by the user.
[0039] In the motion processing branch, the motion noise map is also first convolved through the spatial convolution layer to extract the motion information in the video frame. Then, this motion information is sent to the self-attention layer for further processing to obtain motion features. These motion features describe the motion patterns and dynamic changes in the video frame and are the key to generating predicted video frames with smooth motion effects.
[0040] According to the technical solution provided by the embodiment of the present application, by combining the spatial convolution layer and the self-attention layer in the two branches, each branch can more comprehensively understand and utilize the spatial information and temporal dynamics of the video frame during the processing process, thereby improving the accuracy of the model and enhancing the adaptability of the model to different video contents.
[0041] In some embodiments, based on the target text feature, the content feature and the motion feature, a fusion module is used to obtain a content fusion feature and a motion fusion feature, including:
[0042] Use temporal cross attention to fuse the target text features with the content features and motion features respectively to obtain the first fused feature and the second fused feature; use cross attention to fuse the first fused feature and the second fused feature to obtain the content fused feature and the motion fused feature.
[0043] Specifically, the temporal cross-attention mechanism allows the model to consider the dependencies between different time points when processing sequence data. In the context of video frames, temporal cross-attention enables the model to capture the correspondence between text descriptions and content or motion at different moments in video frames. By fusing the target text features with content features and motion features through temporal cross-attention, the model is able to generate two intermediate fused features: the first fused feature combines the relationship between the text description and the content of the video frame; the second fused feature combines the relationship between the text description and the motion of the video frame.
[0044] Furthermore, the two intermediate fusion features are further fused using the cross-attention mechanism. Cross-attention is a bidirectional attention mechanism that allows features from two different sources (here, the first fusion feature and the second fusion feature) to interact with each other and generate the final feature representation. Through cross-attention fusion, the model can comprehensively consider the information between the target text features, content features, and motion features to generate more comprehensive and accurate fusion features: content fusion features and motion fusion features. The content fusion features and motion fusion features not only contain the original content information of the video frame, but also incorporate the guidance of the target text description on the content, so that the predicted video frame is more in line with the requirements of the text description in terms of content. At the same time, it also combines the motion information of the video frame and the expectations of the text description for motion, so that the predicted video frame is more in line with the scene and emotion described in the text in terms of dynamic performance.
[0045] The technical solution provided by the embodiment of the present application can more comprehensively understand and utilize the information between target text features, content features, and motion features when processing video frames through a multi-level attention fusion mechanism. It can not only improve the accuracy of the model in the video frame prediction task, but also enhance the adaptability of the model to different text descriptions and video contents. At the same time, the interaction and fusion of the feature information of the two branches make the generated predicted video frames smoother.
[0046] In some embodiments, the diffusion model further includes a convolutional fusion network, which obtains a predicted video frame based on the content fusion feature and the motion fusion feature output by the last sub-network, including:
[0047] The first convolution layer of the convolutional fusion network is used to reduce the number of channels of the content fusion features and the motion fusion features; the second convolution layer of the convolutional fusion network is used to enhance the content fusion features and the motion fusion features after the number of channels is reduced; the fusion layer of the convolutional fusion network is used to fuse the enhanced content fusion features and the motion fusion features to obtain convolutional fusion features; the third convolution layer of the convolutional fusion network is used to restore the number of channels of the convolutional fusion features to obtain restored features; the restored features are input into a preset model for processing to obtain target fusion features; based on the target fusion features, a predicted video frame is obtained.
[0048] Specifically, in the convolutional fusion network, the role of the first convolutional layer (e.g., 1×1 convolutional layer) is to reduce the dimension of features by reducing the number of channels in order to reduce the amount of computation and extract more critical information. After reducing the number of channels, the content fusion features and motion fusion features become more compact while retaining key information, which is convenient for subsequent feature processing. Then, the second convolutional layer (e.g., 3×3 convolutional layer) performs feature enhancement on the features after reducing the number of channels. This layer further mines and enhances the detail information in the content fusion features and motion fusion features by increasing the diversity and depth of the convolution kernel. This detail information can reflect the subtle changes in the content and motion of the video frames.
[0049] Further, the fusion layer fuses the content fusion feature and the motion fusion feature after feature enhancement. This layer fuses the two features together through a specific fusion strategy (such as addition, multiplication or splicing, etc.) to generate a convolution fusion feature that contains both content and motion information. This fusion feature not only contains the original information of the video frame, but also incorporates the guidance of the target text description on the content and motion, so it can more accurately reflect the characteristics that the predicted video frame should have. Then, the third convolution layer (such as a 1×1 convolution layer) restores the number of channels of the convolution fusion feature. This layer restores the dimension of the feature by increasing the number of channels to match the dimension of the original video frame. Maintain the consistency and integrity of the feature in subsequent processing. Finally, the restored feature is input into the preset model for processing to obtain the target fusion feature. This preset model can be any deep learning model suitable for processing video frames, such as a convolutional neural network (CNN), a recurrent neural network (RNN) or a transformer (Transformer), etc. The preset model of this embodiment preferably uses a multilayer perceptron (MultilayerPerceptron, MLP).
[0050] In some embodiments, determining a video frame noise map and a motion noise map for generating a target video includes:
[0051] A preset number of video frame images are obtained, and the video frame images are subjected to noise processing to obtain a video frame noise map; the video frame images are subjected to motion decomposition to obtain motion decomposition features; and the motion decomposition features are subjected to noise processing to obtain a motion noise map.
[0052] Specifically, a preset number of video frame images are obtained. The video frame images can be randomly selected from the existing video data, or can be screened according to a specific rule (such as a high correlation with the target text description). By performing noise processing on these video frame images, a video frame noise map can be obtained. The noise processing can adopt various methods, such as adding Gaussian noise, salt and pepper noise, etc., to increase the diversity and randomness of the noise map.
[0053] Furthermore, the video frame image is subjected to motion decomposition. The purpose of motion decomposition is to extract the motion information in the video frame and separate it from the original image so as to focus on the motion change information of the video in subsequent processing. Motion decomposition can be achieved by using methods such as optical flow method and inter-frame difference method. After obtaining the motion decomposition features, it is subjected to noise processing to generate a motion noise map. Similar to the video frame noise map, the noise processing of the motion noise map is also to increase diversity and randomness to support the subsequent generation process.
[0054] After obtaining the video frame noise map and motion noise map, we can use them as input, combine them with the target text features, perform iterative optimization through the diffusion model, and gradually generate video frames that meet the target text description. In this process, content fusion features and motion fusion features will play a key role. They will guide the model on how to fuse the text description with the content and motion information of the video frame during the generation process to generate predicted video frames that better meet user expectations.
[0055] In some embodiments, performing motion decomposition on a video frame image to obtain motion decomposition features includes:
[0056] A preset encoder is used to perform feature encoding on two adjacent video frame images of a video frame image to obtain image features of the two adjacent video frame images; the fourth convolutional layer is used to reduce the number of channels of the image features; the fifth convolutional layer is used to perform convolution processing on the image features of the latter video frame image of the two video frame images after the number of channels is reduced to obtain video frame convolution features; based on the image features of the previous video frame image of the two video frame images and the video frame convolution features, a differential feature is obtained, and the sixth convolutional layer is used to enhance the differential feature to obtain a target differential feature; an additional frame feature is obtained, and a motion decomposition feature is obtained based on the additional feature and the target differential feature.
[0057] Specifically, in one example, the first frame and the l+1th frame of the video sequence can be input into the VIT image encoder to extract the image features F and L respectively. l and F l+1 ; After that, use the 1X1 convolution layer (the fourth convolution layer) to independently reduce F l and F l+1 The number of channels is reduced to reduce the computational complexity. Then, F l+1 Perform a 3X3 two-dimensional convolution operation (the fifth convolution layer) to obtain feature F l ‘ +1 ; Then, the transformed F l Subtract F l ‘ +1 , and obtain the differential characteristic ΔF l, to enhance the representation of motion features;
[0058] Furthermore, the 1X1 convolution layer (the sixth convolution layer) is used again to process ΔF l , repeat the above steps with the target differential feature, operate on all frames to obtain the motion features of the entire video sequence, and finally obtain the features of the L-1 frame. In addition, in order to keep the video length L, an additional frame of features is added to ensure that the number of video frames remains unchanged. Finally, the features of the L-1 frame and the additional frame of features are added to obtain the motion decomposition features.
[0059] Figure 2 is a flow chart of another video generation method provided in an embodiment of the present application, such as Figure 2 As shown:
[0060] First, based on the acquired style reference image, the style text description and style image features can be obtained, wherein the image style of the Q-Former question image in the BLIP2 image encoder can be used to obtain the style text description, and the style reference image is processed by the VIT image encoder to obtain the style image features. After the style text description is obtained, it is spliced with the text description to obtain the target text description, and the target text description is encoded by the CLIP text encoder to obtain the target text features. After that, a preset number of video frame images are obtained, and the video frame images are subjected to noise processing to obtain a video frame noise map; the video frame images are subjected to motion decomposition to obtain motion decomposition features; the motion decomposition features are subjected to noise processing to obtain a motion noise map.
[0061] Then, the style image features and target text features are input into each sub-network of the diffusion model; the video frame noise map is input into the content processing branch of the first sub-network for feature processing, and the motion noise map is input into the motion processing branch of the first sub-network for feature processing.
[0062] The processing results of the two branches are fused by the fusion module, and the fusion results are input into the content processing branch and motion processing branch of the next sub-network. The processing steps of the branch network and the processing steps of the fusion module are repeated until the output result of the last sub-network is obtained.
[0063] Finally, the convolutional fusion network of the diffusion model is used to perform convolutional fusion on the output of the last sub-network to obtain a set of predicted video frames output by the diffusion model.
[0064] Figure 3 is a schematic diagram of a sub-network workflow provided by an embodiment of the present application, such as Figure 3 As shown:
[0065] Both the content processing branch and the motion processing branch contain spatial convolution layers and self-attention layers. After the input data is input into each sub-network, the content processing branch of each sub-network will use the spatial convolution layer to convolve the input data; use the self-attention layer to process the video frame noise map after convolution to obtain the attention feature, add the attention feature to the style image feature to obtain the content feature, and use the spatial convolution layer to convolve the motion noise map; use the self-attention layer to process the motion noise map after convolution to obtain the motion feature.
[0066] In the fusion module, the target text features are fused with the content features and the motion features respectively using temporal cross attention to obtain the first fused features and the second fused features; the first fused features and the second fused features are fused using cross attention to obtain the content fused features and the motion fused features.
[0067] Need to explain, Figure 3 The processing flow of the first sub-network is given as an example, and its input data is a video frame noise map and a motion noise map. The processing flows of the remaining sub-networks can refer to the processing flow of the first sub-network, and will not be described in detail here.
[0068] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of the present application.
[0069] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described in detail here.
[0070] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0071] Figure 4 Schematic diagram of the structure of a video generation device provided in an embodiment of the present application. Figure 4 As shown, the device comprises:
[0072] An acquisition module 401 is configured to acquire a style reference image and a text description;
[0073] The feature module 402 is configured to obtain a style text description and a style image feature based on the style reference image, obtain a target text description based on the style text description and the text description, and perform feature processing on the target text description to obtain a target text feature;
[0074] The generation module 403 is configured to determine the video frame noise map and the motion noise map for generating the target video, input the style image features, the target text features, the video frame noise map and the motion noise map into the diffusion model, obtain the predicted video frame output by the diffusion model, and obtain the target video based on the predicted video frame; wherein the diffusion model includes a plurality of sub-networks arranged in succession, each sub-network includes a content processing branch, a motion processing branch and a fusion module, the content processing branch performs diffusion processing based on the style image features and the video frame noise map, the motion processing branch performs diffusion processing based on the motion noise map, and the fusion module is used to fuse the diffusion processing results of the content processing branch and the motion processing branch with the target text features, and use the fusion results as the input of the two branches in the next sub-network.
[0075] In some embodiments, the generation module 403 is also used to input the style image features and the target text features into each sub-network of the diffusion model; input the video frame noise map into the content processing branch of the first sub-network for feature processing, and add it to the style image features to obtain content features, and input the motion noise map into the motion processing branch of the first sub-network for feature processing to obtain motion features; based on the target text features, content features and motion features, use the fusion module to obtain content fusion features and motion fusion features; input the content fusion features and motion fusion features into the content processing branch and motion processing branch of the next sub-network respectively, repeat the feature processing steps and the processing steps of the fusion module until the content fusion features and motion fusion features output by the last sub-network are obtained, and based on the content fusion features and motion fusion features output by the last sub-network, a predicted video frame is obtained.
[0076] In some embodiments, both the content processing branch and the motion processing branch include a spatial convolution layer and a self-attention layer, and the generation module 403 is also used to perform convolution processing on the video frame noise map using the spatial convolution layer; use the self-attention layer to process the video frame noise map after convolution processing to obtain attention features, add the attention features to the style image features to obtain content features; use the spatial convolution layer to perform convolution processing on the motion noise map; use the self-attention layer to process the motion noise map after convolution processing to obtain motion features.
[0077] In some embodiments, the generation module 403 is also used to use temporal cross-attention to fuse the target text feature with the content feature and the motion feature respectively to obtain a first fused feature and a second fused feature; and use cross-attention to fuse the first fused feature and the second fused feature to obtain a content fused feature and a motion fused feature.
[0078] In some embodiments, the diffusion model also includes a convolutional fusion network, and the generation module 403 is also used to use the first convolutional layer of the convolutional fusion network to reduce the number of channels of the content fusion features and the motion fusion features; use the second convolutional layer of the convolutional fusion network to enhance the content fusion features and the motion fusion features after the number of channels is reduced; use the fusion layer of the convolutional fusion network to fuse the enhanced content fusion features and the motion fusion features to obtain convolutional fusion features; use the third convolutional layer of the convolutional fusion network to restore the number of channels of the convolutional fusion features to obtain restored features; input the restored features into a preset model for processing to obtain target fusion features; and obtain a predicted video frame based on the target fusion features.
[0079] In some embodiments, the generation module 403 is also used to obtain a preset number of video frame images, perform noise processing on the video frame images to obtain a video frame noise map; perform motion decomposition on the video frame images to obtain motion decomposition features; perform noise processing on the motion decomposition features to obtain a motion noise map.
[0080] In some embodiments, the generation module 403 is also used to use a preset encoder to perform feature encoding on two adjacent video frame images of a video frame image to obtain image features of the two adjacent video frame images; use the fourth convolutional layer to reduce the number of channels of the image features; use the fifth convolutional layer to convolve the image features of the latter video frame image in the two video frame images after reducing the number of channels to obtain video frame convolution features; based on the image features of the previous video frame image in the two video frame images and the video frame convolution features, differential features are obtained, and the differential features are enhanced using the sixth convolutional layer to obtain target differential features; obtain a frame of additional features, and obtain motion decomposition features based on the additional features and the target differential features.
[0081] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present application. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.
[0082] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.
[0083] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0084] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.
[0085] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.
[0086] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. The readable storage medium may include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0087] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A video generation method, characterized in that: include: Get style reference images and text descriptions; Based on the style reference graph, a style text description and a style image feature are obtained, based on the style text description and the text description, a target text description is obtained, and feature processing is performed on the target text description to obtain target text features; Determine a video frame noise map and a motion noise map for generating a target video, input the style image feature, the target text feature, the video frame noise map and the motion noise map into a diffusion model, obtain a predicted video frame output by the diffusion model, and obtain the target video based on the predicted video frame; Among them, the diffusion model includes a plurality of sub-networks arranged in succession, each of the sub-networks includes a content processing branch, a motion processing branch and a fusion module, the content processing branch performs diffusion processing based on the style image features and the video frame noise map, the motion processing branch performs diffusion processing based on the motion noise map, and the fusion module is used to fuse the diffusion processing results of the content processing branch and the motion processing branch with the target text features, and use the fusion results as inputs of the two branches in the next sub-network.
2. The method according to claim 1, characterized in that The step of inputting the style image feature, the target text feature, the video frame noise map, and the motion noise map into a diffusion model to obtain a predicted video frame output by the diffusion model includes: Inputting the style image features and the target text features into each sub-network of the diffusion model; Input the video frame noise map into the content processing branch of the first sub-network for feature processing, and add it to the style image feature to obtain a content feature, and input the motion noise map into the motion processing branch of the first sub-network for feature processing to obtain a motion feature; Based on the target text feature, the content feature and the motion feature, using the fusion module to obtain a content fusion feature and a motion fusion feature; The content fusion feature and the motion fusion feature are respectively input into the content processing branch and the motion processing branch of the next sub-network, and the feature processing steps and the processing steps of the fusion module are repeatedly executed until the content fusion feature and the motion fusion feature output by the last sub-network are obtained, and the predicted video frame is obtained based on the content fusion feature and the motion fusion feature output by the last sub-network.
3. The method according to claim 2, characterized in that The content processing branch and the motion processing branch both include a spatial convolution layer and a self-attention layer; The step of inputting the video frame noise map into the content processing branch of the first sub-network for feature processing and adding the feature to the style image feature to obtain the content feature comprises: Performing convolution processing on the video frame noise map using the spatial convolution layer; Using the self-attention layer to process the video frame noise map after convolution processing to obtain an attention feature, and adding the attention feature to the style image feature to obtain the content feature; The step of inputting the motion noise map into the motion processing branch of the first sub-network for feature processing to obtain motion features comprises: Using the spatial convolution layer to perform convolution processing on the motion noise map; The motion noise map after convolution processing is processed by using the self-attention layer to obtain the motion feature.
4. The method according to claim 2, characterized in that: The step of obtaining content fusion features and motion fusion features by using the fusion module based on the target text features, the content features and the motion features includes: Using temporal cross attention to fuse the target text feature with the content feature and the motion feature respectively, to obtain a first fused feature and a second fused feature; The first fusion feature and the second fusion feature are fused by using cross attention to obtain the content fusion feature and the motion fusion feature.
5. The method according to claim 2 is characterized in that: The diffusion model further includes a convolutional fusion network, and the predicted video frame is obtained based on the content fusion feature and the motion fusion feature output by the last sub-network, including: Using the first convolutional layer of the convolutional fusion network to reduce the number of channels of the content fusion feature and the motion fusion feature; Using the second convolutional layer of the convolutional fusion network, the content fusion feature and the motion fusion feature after the number of channels is reduced are enhanced; Using the fusion layer of the convolutional fusion network to fuse the content fusion feature and the motion fusion feature after feature enhancement, to obtain a convolutional fusion feature; Using the third convolution layer of the convolutional fusion network to restore the number of channels of the convolutional fusion feature, to obtain a restored feature; Inputting the restored features into a preset model for processing to obtain target fusion features; Based on the target fusion features, the predicted video frame is obtained.
6. The method according to claim 1, characterized in that The determining of a video frame noise map and a motion noise map for generating a target video comprises: Acquire a preset number of video frame images, perform noise processing on the video frame images, and obtain the video frame noise map; Performing motion decomposition on the video frame image to obtain motion decomposition features; The motion decomposition features are subjected to noise addition processing to obtain the motion noise map.
7. The method according to claim 6, characterized in that The performing motion decomposition on the video frame image to obtain motion decomposition features includes: Using a preset encoder to perform feature encoding on two adjacent video frame images of the video frame image to obtain image features of the two adjacent video frame images; Using a fourth convolutional layer to reduce the number of channels of the image features; Using a fifth convolutional layer, convolution processing is performed on the image features of the latter video frame image of the two video frame images after the number of channels is reduced to obtain a video frame convolution feature; Based on the image features of the previous video frame image of the two video frame images and the video frame convolution features, a differential feature is obtained, and the differential feature is enhanced by using the sixth convolution layer to obtain a target differential feature; A frame of additional features is obtained, and the motion decomposition features are obtained based on the additional features and the target differential features.
8. A video generating device, characterized in that: include: An acquisition module, configured to acquire a style reference image and a text description; A feature module is configured to obtain a style text description and a style image feature based on the style reference image, obtain a target text description based on the style text description and the text description, and perform feature processing on the target text description to obtain a target text feature; A generation module is configured to determine a video frame noise map and a motion noise map for generating a target video, input the style image features, the target text features, the video frame noise map and the motion noise map into a diffusion model, obtain a predicted video frame output by the diffusion model, and obtain the target video based on the predicted video frame; wherein the diffusion model includes a plurality of sub-networks arranged in succession, each of the sub-networks includes a content processing branch, a motion processing branch and a fusion module, the content processing branch performs diffusion processing based on the style image features and the video frame noise map, the motion processing branch performs diffusion processing based on the motion noise map, and the fusion module is used to fuse the diffusion processing results of the content processing branch and the motion processing branch with the target text features, and use the fusion results as inputs of the two branches in the next sub-network.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Image generation method and device, electronic equipment and readable storage medium
CN119338951A
Image generation method, device, electronic device, and readable storage medium
CN119338951B
Consistency video generation method and device based on wavelet transform and frequency decomposition
CN120264095A
Consistent video generation method and device based on wavelet transform and frequency decomposition
CN120264095B
Video generation method and device, equipment, storage medium and program product
CN120434478A
