Fine-grained regulation and control and multi-scene coherence video generation method based on diffusion model

Abstract

The invention discloses a fine-grained regulation and control and multi-scene coherence video generation method based on a diffusion model, and the method comprises the following steps: automatically generating a simple video script through a large language model, and carrying out the detailed processing of a language and a split mirror through manual work; utilizing a stable diffusion model to generate images containing different scene groups; fine-grained control of image details and depth of field is realized by adjusting model parameters and algorithms; through a character consistency algorithm, it is ensured that character images generated under different scenes are kept coherent; background images generated under different scenes are kept consistent through a background consistency algorithm; simulating a lens rotation effect by adopting a visual angle transformation algorithm, and generating images at different angles; and taking the generated image as a video key frame, and generating coherent video content by optimizing inter-frame transition processing and dynamic frame insertion operation.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation in computer vision, and in particular to a method for fine-grained control and multi-view coherence video generation based on a diffusion model. Background Art

[0002] With the rapid development of digital media and entertainment industries, the generation and editing of video content has become a growing demand. Traditional video production relies on actual shooting and complex post-editing, which not only consumes a lot of time and resources, but also requires a high level of technical skills from the creators. In recent years, diffusion models have opened up new possibilities for video creation with their powerful data simulation and generation capabilities. However, despite the remarkable achievements in this field, existing technologies still expose many defects that need to be addressed in practical applications.

[0003] First, a key challenge is how to achieve smooth and natural switching and transitions between different shot sizes. The shot size, that is, the distance between the camera and the subject, determines the range of content that the audience sees on the screen. From long shots to close-ups, each shot size has its own unique narrative function and visual experience. In video production, the change of shot size can not only guide the audience's attention, but also enhance the dynamics and emotional expression of the story. However, when dealing with such complex scenes, current models often find it difficult to maintain the natural and smooth flow of video content, causing discomfort to the audience during the viewing process. This problem not only affects the viewing experience of the video, but also limits the depth and breadth of video content creation.

[0004] Secondly, the consistency between characters and background is another technical challenge. In multi-angle videos, the character's posture, expression, and movement need to be consistent in different shots to avoid the audience feeling abrupt. At the same time, the background elements also need to be consistent in different shots to maintain the overall sense and realism of the video. Automatically generated character images and backgrounds may be inconsistent due to the randomness of the diffusion model or parameter adjustments.

[0005] In addition, the simulation of lens rotation effect is an important means to improve the viewing experience of videos. In actual shooting, by rotating the camera to capture different perspectives and dynamics, it can add rich visual effects to the video. However, existing video generation technology has limitations in simulating lens rotation effects, and it is difficult to generate rotating perspective images with realism and dynamics. This makes it difficult for users to obtain video content that meets their personalized requirements when facing specific needs.

[0006] In summary, existing video generation technologies have obvious defects in fine-grained multi-view coherence, consistency between characters and backgrounds, and simulation of lens rotation effects. Summary of the invention

[0007] The purpose of the present invention is to provide a method for fine-grained control and multi-view coherence video generation based on a diffusion model to address the above-mentioned problems that currently exist. The purpose is to automatically produce high-quality video content based on the generated video script with minimal human intervention.

[0008] The technical solution of the present invention is as follows:

[0009] The fine-grained control and multi-view coherence video generation method based on the diffusion model includes the following steps:

[0010] A simple video script is automatically generated through a large language model, and the language and storyboards are processed manually to obtain the final video script;

[0011] Use the stable diffusion model to generate images with different scene resolutions; adjust model parameters and algorithm layers to achieve fine-grained control of image details and depth of field;

[0012] The character consistency algorithm is used to ensure that the character images generated under different scenes remain consistent;

[0013] The background consistency algorithm is used to ensure that the background images generated under different scenes are consistent;

[0014] Use perspective transformation algorithm to simulate lens rotation effect and generate images at different angles;

[0015] The generated images are used as video key frames, and coherent video content is generated by optimizing the transition between frames.

[0016] Furthermore, the background consistency algorithm comprises the following steps:

[0017] Collect a series of high-quality panoramic videos, either including only the background scene or covering both foreground and background;

[0018] Select key frames including complete background information from panoramic videos;

[0019] For videos including foreground, the SAM image segmentation algorithm is used to separate the background from the foreground in the key frame, and the background is filled in by the remaining key frames and the diffusion model;

[0020] ControlNet is used to predict the depth information of the background area, and four views of each key frame are drawn by combining all key frames; the coarse prediction module including multi-layer perceptron and low-frequency position coding is used to capture the large-scale structure in the background, so as to construct the preliminary shape of the background; the refinement module including high-frequency position coding and multi-layer perceptron is used to capture the background surface changes and details to construct a more refined background shape; the optimized background structure is represented by neural implicit surface to obtain the 3D modeling of the background.

[0021] Furthermore, the three-dimensional modeling of the background is three-dimensional modeling under a panoramic lens, and the diffusion model is trained by different scene types so that the diffusion model can adaptively scale the three-dimensional modeling according to the segmentation algorithm and the lens prompt words.

[0022] Furthermore, the character consistency algorithm includes the following steps:

[0023] Collect motion videos of multiple individuals, where the videos show the activities of the individuals in the same or different scenarios;

[0024] DWPose motion capture technology is used to extract skeleton features from the video to form a human skeleton sequence P; specific character shots are abstracted;

[0025] Combine the abstracted scene shots with the character shots to further abstract a rich lens language;

[0026] Use SAM image segmentation algorithm to extract key feature areas from the video;

[0027] The feature Q of the region is further encoded through the variational autoencoder model;

[0028] A conditional embedding module is designed based on the diffusion model to embed the human skeleton sequence P and the regional feature layer Q into the diffusion model.

[0029] Furthermore, the character consistency algorithm further comprises the following steps:

[0030] Use the face detection model to extract the portrait face region from the pose image; use the portrait face set M i Adaptively replace the extracted facial region.

[0031] Furthermore, the face detection model finely adjusts the facial region in the portrait generated by the diffusion model through cropping, splicing, redirection and fusion, and generates a set M of faces of the same portrait with various expressions through expression prompt words and fixed random seeds. i .

[0032] Furthermore, embedding the human skeleton sequence P into the diffusion model comprises the following steps:

[0033] The human skeleton sequence P is first enhanced through a double-layer sparse self-attention mechanism layer to enhance the expressiveness of the sequence features. Subsequently, the sequence images are processed using ConvNeXt blocks to ensure that their resolution matches the input requirements of the diffusion model backbone. Finally, the processed sequence is cascaded with the input of the diffusion model backbone to form a new input FP, providing the model with richer contextual information;

[0034] The regional feature layer Q is embedded in the diffusion model, which includes the following steps:

[0035] The regional feature layer Q is added to FP in the form of residual after passing through the feature adaptation module. At the same time, in order to enhance the model's ability to capture the appearance and position information of local portraits, Q is cross-attended with the backbone of the diffusion model.

[0036] Furthermore, the perspective transformation algorithm comprises the following steps:

[0037] The diffusion model and specific prompt words are used to generate the same person in different scenes, and the scene changes are cleverly used to adaptively embed these person into the generated background, so as to construct a key frame of the final video.

[0038] The same method is used to generate different key frames under multiple lenses, and the diffusion model is used for relighting to enhance the light and shadow effects and visual impact of the picture;

[0039] After manual selection and sorting, all the key frames required for the final video are determined;

[0040] Through frame transition optimization and interpolation operations, a smooth and coherent final video with perspective changes is produced.

[0041] Furthermore, the different shot levels include distant shot, panoramic shot, mid shot, close shot, and close-up.

[0042] Furthermore, the key frame includes a foreground and a background, wherein the foreground is a character image generated by a character consistency algorithm, and the background is a background image generated based on a background consistency algorithm.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The fine-grained control and multi-view coherence video generation method based on the diffusion model combines the cutting-edge open source language large model and the stable diffusion model, and can automatically generate video scripts and images with only light human intervention, greatly improving the efficiency of video production. The proposed character consistency algorithm and background consistency algorithm ensure the coherence of images under different views, thereby improving the overall quality of the video content. In addition, by simulating the lens rotation effect and optimizing the transition between frames, the richness and viewing experience of the video are enhanced. More importantly, this method lowers the technical threshold for video production, allowing non-professionals to easily create professional videos, promoting the vigorous development of personalized and creative content. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Flowchart of this application.

[0046] Figure 2 This is a schematic diagram of the generation principle of this application.

[0047] Figure 3 This is a scene representation diagram of this application.

[0048] Figure 4 This is the overall flow chart of this application.

[0049] Figure 5 This is the character consistency algorithm architecture diagram of this application.

[0050] Figure 6 This is the result diagram of the character consistency algorithm of this application.

[0051] Figure 7 This is the background consistency algorithm architecture diagram of this application.

[0052] Figure 8 This is the background consistency algorithm result diagram of this application. DETAILED DESCRIPTION

[0053] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0054] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0055] See also Figure 1-8 , fine-grained control and multi-view coherent video generation method based on diffusion model, such as Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, the following steps are included:

[0056] Automatically generate simple video scripts through a large language model, and perform manual refinement, including language and storyboard optimization, to obtain the final video script;

[0057] Use the stable diffusion model to generate images with different scene resolutions, and adjust the model parameters and algorithm layers to achieve precise control of image details and depth of field.

[0058] The diffusion model is retrained through the character consistency algorithm to ensure that the characters in the images generated under different scenes remain consistent. ;

[0059] The background consistency algorithm assists the diffusion model to generate images to ensure the consistency of the background of images under different scenes;

[0060] The perspective transformation algorithm is used to simulate the lens rotation effect. Its principle is to process key frames with different perspectives through smooth interpolation to generate images at different angles.

[0061] The generated images are used as video key frames, which are manually selected and sorted, and then processed by optimizing the transition between frames to generate coherent video content.

[0062] like Figure 5 As shown, the character consistency algorithm includes the following steps:

[0063] Collect motion videos of multiple individuals. These videos should show the activities of individuals in the same or different scenes. The individuals in the video should be clearly visible, avoid occlusion, and include repeated actions in the same scene and changes in different scenes. The video can be obtained through the Internet, camera or mobile phone, and can be animated or real-life. The length is limited to between 5 and 30 seconds.

[0064] Using DWPose motion capture technology, skeleton features are extracted from these videos and a human skeleton sequence P is constructed. This sequence is used to identify and abstract specific character shots. In this process, a set of discrete posture categories are defined, represented by letters A to E: A represents the front, B represents a 45-degree right turn, C represents the side, D represents a 135-degree turn, and E represents the back. Its purpose is to convert the character's posture information into control signals while effectively eliminating interference factors such as background and appearance.

[0065] Combining the abstracted scene shots with the character shots can construct a rich and detailed lens language. For example, A1 represents a frontal portrait in a panoramic shot, and C3 represents a side portrait in a medium shot.

[0066] The SAM image segmentation algorithm is used to extract key regional features from human videos, including the person's clothing, hairstyle, skin color, etc. Subsequently, these specific regional features are re-encoded through the variational autoencoder (VAE) model.

[0067] In the final stage of feature processing, a feature set Q containing information of multiple specific regions is constructed by cascading channel dimensions. Furthermore, based on the conditional embedding module, the human skeleton sequence P and the regional feature layer Q are embedded in the diffusion model. After careful training and optimization, the diffusion model is named "multi-lens portrait model", which can accurately generate posture images of the same person under different scene scales through the cross combination of AE and 1-5.

[0068] In order to enhance the accuracy of facial regional features, a face-guided expression diffusion model is developed. Combining expression prompt words and fixed random seeds, it can generate a set M of faces with different expressions of the same portrait. i .

[0069] First, a face detection algorithm is used to accurately extract the portrait face region from the pose image. Then, the pre-built portrait face set M is used to extract the portrait face region. i , through operations such as cropping, splicing, redirection and fusion, the extracted facial areas of the portrait are adaptively replaced. This process significantly improves the consistency of the facial features of the characters, ensuring that the facial expressions and features of the characters can remain highly consistent in different video scenes.

[0070] The process of embedding the human skeleton sequence P into the diffusion model follows the following steps:

[0071] The human skeleton sequence P is first processed by a dual sparse self-attention layer, which enhances the expressiveness of sequence features by optimizing attention allocation.

[0072] Subsequently, the sequence images are processed using ConvNeXt blocks to ensure that their resolution matches the input requirements of the diffusion model backbone, thereby maintaining the integrity of the image information.

[0073] Finally, the processed sequence is concatenated with the input of the diffusion model backbone to form a new input FP, providing the model with richer contextual information;

[0074] The process of embedding the regional feature layer Q into the diffusion model follows the following steps:

[0075] The regional feature layer Q is processed by the feature adaptation module and then added to FP in the form of residual.

[0076] At the same time, in order to further enhance the model’s ability to capture local portrait appearance and position information, Q is cross-attentionally operated with the diffusion model backbone.

[0077] Figure 6 The result graph generated by the character consistency algorithm provided in this application can generate an image with the same character template in a refined manner after determining the character skeleton sequence.

[0078] like Figure 7 As shown, the background consistency algorithm includes the following steps:

[0079] Collect multiple high-quality panoramic videos that can include only the background or both foreground and background. The video length should be between 10 and 30 seconds.

[0080] From these panoramic videos, select key frames that can fully represent the background features of the entire video. These key frames should contain complete background information.

[0081] For videos containing foreground, the SAM image segmentation algorithm is used to separate the background from the foreground in the key frame. Then, the background is filled using the remaining key frames and the diffusion model; for videos that do not contain foreground, no processing is required.

[0082] ControlNet is used to predict the depth information of the background area and is combined with the diffusion model to draw four views of each key frame, including the front view, back view, top view and left view.

[0083] First, a coarse prediction module combining a shallow multi-layer perceptron (MLP) and a low-frequency position encoding is used to capture the large-scale structure of the background, thereby constructing a preliminary shape of the background. Then, a refinement module containing a deep multi-layer perceptron and a high-frequency position encoding is used to capture the background surface changes and details to construct a more refined background shape. Finally, Neural Implicit Surfaces is used to represent the optimized background structure to achieve 3D modeling of the background.

[0084] To ensure the consistency of background features under different scenes, the diffusion model is trained in combination with numbers 1-5, so that it can adaptively scale the 3D model according to the segmentation algorithm and the lens prompt words to achieve the effect of the framing frame. Specifically, the number 1 represents the panoramic view, 2 represents the distant view, 3 represents the mid-range view, 4 represents the close-up view, and 5 represents the close-up view.

[0085] Figure 8 The result diagram generated by the background consistency algorithm provided in this application, wherein the first image is a background key frame, and the remaining images show different direction views of the 3D background model.

[0086] The perspective transformation algorithm includes the following steps:

[0087] The multi-lens portrait model accurately generates the same person in different scenes through the combination of AE and 1-5. As the scene changes, these people are adaptively embedded into the generated background through prompt words to form the key frames of the video.

[0088] The keyframes generated from multiple shots are relighted using a diffusion model to simulate different lighting conditions, thereby improving the dynamic range and depth of the image.

[0089] The key frame sequence of the final video is selected and sorted by professionals. Through dynamic interpolation and inter-frame transition optimization, smooth and coherent video content with perspective changes is produced.

[0090] Different shot types include long shot, panoramic shot, mid shot, close shot and close-up.

[0091] The key frame includes a foreground and a background. The foreground is a person after the generated image is processed by a person consistency algorithm, and the background is a scene after the generated image is processed by a background consistency algorithm.

[0092] The present application provides a method for fine-grained control and multi-angle coherence video generation based on a diffusion model, which generates coherence videos of different angles and lenses through a given video script.

[0093] Using open source language models, such as the GPT series, we can automatically construct storylines, dialogues, and scene descriptions based on given keywords or topics to generate a first draft of the video script.

[0094] Based on the automatically generated script, professionals refine the storyboard to ensure the script's feasibility and expressiveness. At the same time, professionals will design the perspective, action and duration of each shot based on the script content, providing detailed visual guidance for subsequent image generation.

[0095] By carefully fine-tuning the existing stable diffusion model, it can be enabled to recognize specific combinations of prompt words and accurately generate images of different scene scales based on them.

[0096] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. Fine-grained control and multi-view coherence video generation method based on diffusion model, characterized in that: The following steps are involved: A simple video script is automatically generated through a large language model, and the language and storyboards are processed manually to obtain the final video script; Use the stable diffusion model to generate images with different scene resolutions; adjust model parameters and algorithm layers to achieve fine-grained control of image details and depth of field; The character consistency algorithm is used to ensure that the character images generated under different scenes remain consistent; The background consistency algorithm is used to ensure that the background images generated under different scenes are consistent; Use perspective transformation algorithm to simulate lens rotation effect and generate images at different angles; The generated images are used as video key frames, and coherent video content is generated by optimizing the transition between frames.

2. According to the diffusion model-based fine-grained control and multi-view coherence video generation method of claim 1, it is characterized in that: The background consistency algorithm comprises the following steps: Collect a series of high-quality panoramic videos, either including only the background scene or covering both foreground and background; Select key frames including complete background information from panoramic videos; For videos including foreground, the SAM image segmentation algorithm is used to separate the background from the foreground in the key frame, and the background is filled in by the remaining key frames and the diffusion model; ControlNet is used to predict the depth information of the background area, and four views of each key frame are drawn by combining all key frames. A coarse prediction module including multi-layer perceptron and low-frequency position coding is used to capture the large-scale structure of the background, thereby constructing the preliminary shape of the background. A refinement module including high-frequency position coding and multi-layer perceptron is used to capture the background surface changes and details to construct a more refined background shape. The optimized background structure is represented using a neural implicit surface to obtain a three-dimensional model of the background.

3. The method for fine-grained control and multi-view coherence video generation based on diffusion model according to claim 2 is characterized in that: The three-dimensional modeling of the background is three-dimensional modeling under a panoramic lens. The diffusion model is trained by different scene types so that the diffusion model can adaptively scale the three-dimensional modeling according to the segmentation algorithm and the lens prompt words.

4. The method for fine-grained control and multi-view coherence video generation based on a diffusion model according to claim 1, characterized in that: The character consistency algorithm comprises the following steps: Collect motion videos of multiple individuals. The videos should show the individuals' activities in the same or different scenarios. DWPose motion capture technology is used to extract skeleton features from the video to form a human skeleton sequence P; specific character shots are abstracted; Combine the abstracted scene shots with the character shots to further abstract a rich lens language; Use SAM image segmentation algorithm to extract key feature areas from human body videos; The specific region feature Q is further encoded through the variational autoencoder (VAE) model; A conditional embedding module is designed based on the diffusion model to embed the human skeleton sequence P and the regional feature layer Q into the diffusion model.

5. The method for fine-grained control and multi-view coherence video generation based on diffusion model according to claim 4 is characterized in that: The character consistency algorithm also includes the following steps: Use the face detection model to extract the portrait face region from the pose image; use the portrait face set M i Adaptively replace the extracted facial region.

6. The method for fine-grained control and multi-view coherence video generation based on a diffusion model according to claim 5, characterized in that: The face detection model fine-tunes the facial region in the portrait generated by the diffusion model through cropping, splicing, redirection and fusion, and generates a set M of faces of the same portrait with various expressions through expression prompt words and fixed random seeds. i .

7. The method for fine-grained control and multi-view coherence video generation based on diffusion model according to claim 4, 5 or 6, characterized in that: The embedding of the human skeleton sequence P into the diffusion model comprises the following steps: The human skeleton sequence P first enhances the expressiveness of sequence features through a double-layer sparse self-attention mechanism layer. Then, the sequence image is processed using the ConvNeXt block to ensure that its resolution matches the input requirements of the diffusion model backbone. Finally, the processed sequence is cascaded with the input of the diffusion model backbone to form a new input FP, providing the model with richer context information; The regional feature layer Q is embedded in the diffusion model, which includes the following steps: The regional feature layer Q is added to FP in the form of residual after passing through the feature adaptation module. At the same time, in order to enhance the model's ability to capture the appearance and position information of local portraits, Q is cross-attended with the backbone of the diffusion model.

8. The method for fine-grained control and multi-view coherence video generation based on a diffusion model according to claim 1, characterized in that: The perspective transformation algorithm comprises the following steps: The diffusion model and specific prompt words are used to generate the same person in different scenes, and the scene changes are cleverly used to adaptively embed these person into the generated background, so as to construct a key frame of the final video. The same method is used to generate different key frames under multiple lenses, and the diffusion model is used for relighting to enhance the light and shadow effects and visual impact of the picture; After manual selection and sorting, all the key frames required for the final video are determined; Through frame transition optimization and interpolation operations, a smooth and coherent final video with perspective changes is produced.

9. The method for fine-grained control and multi-view coherence video generation based on a diffusion model according to claim 1, characterized in that: The different shot types include long shot, full shot, medium shot, close shot and close-up.

10. The method for fine-grained control and multi-view coherence video generation based on diffusion model according to claim 1, characterized in that: The key frames are composed of two parts: foreground and background, wherein the foreground is a character image generated by a character consistency algorithm, and the background is a background image generated based on a background consistency algorithm.

Images