Dynamic split mirror generation system and method based on controllable diffusion model

Abstract

The invention discloses a dynamic split mirror generation system and method based on a controllable diffusion model, and belongs to the technical field of film and television production. The implementation method comprises the following steps of: 1, setting a keyword text by a user, and inputting the keyword text into ChatGPT to generate a script; 2, converting the script into a scene category, a camera motion mode, a character role position and action and image description in a shot language by utilizing ChatGPT; 3, using a CLIP model to carry out contrast training on the image encoder and the text encoder; 4, performing an OpenPose model on the action reference image to obtain skeleton key points of the character role, and converting the skeleton key points into image character actions; 5, performing action fine-grained control on the character action of the image by using a Stable Diffusion model and a ControlNet model, and generating a split image; 6, utilizing a Pika Labs model to generate a dynamic video from the split image; compared with the prior art, the accuracy of user role action matching under the scene based on multi-text and high-complexity actions is improved.

Description

Technical Field

[0001] The present invention relates to a dynamic storyboard generation system and method based on a controllable diffusion model, belongs to the technical field of film and television production, and is applied to dynamic storyboard generation in the film and television industry. Background Art

[0002] Current film and television pre-production has evolved into a three-stage system: scriptwriting, storyboarding, and virtual pre-production. Scriptwriting builds the story framework, storyboard design uses visual language to plan shot composition and action flow, and virtual pre-production uses scene simulation to validate creative ideas. Recent technological advancements have not only introduced script visualization tools and collaboration platforms, but also achieved breakthroughs by combining diffusion models with NeRF technology, enabling text-driven shot trajectory generation and style transfer, making significant progress in the field of dynamic storyboard generation.

[0003] However, the existing system still suffers from systemic flaws. Script and storyboard design rely heavily on the creator's professional experience. Storyboard production, in particular, requires simultaneous consideration of multiple dimensions, including composition, perspective, and logical cohesion, creating a barrier to professional expertise for new creators. Tools for each link are fragmented, with scriptwriting, storyboard design, and pre-production systems operating independently, creating a gap in collaboration. Existing AI technology faces bottlenecks in constructing complex logic and expressing detailed information. When faced with complex action scenes inputted from text, it still struggles to accurately match the character's movements set by the user.

[0004] Therefore, how to improve the accuracy of user character action matching in existing storyboard videos in scenarios based on multiple texts and highly complex actions has become an urgent problem to be solved. Summary of the Invention

[0005] This invention aims to address the technical problem of improving the accuracy of user character action matching in existing storyboard videos based on multiple texts and highly complex actions. This invention proposes a dynamic storyboard generation system and method based on a controllable diffusion model. This system utilizes ChatGPT, CLIP, Stable Diffusion, and Pika Labs models to transform text descriptions into images and video dynamic storyboards, from script creation to shot design. By optimizing the model and controlling image generation to match character actions and positions, the matching between text and storyboard actions is improved.

[0006] The purpose of the present invention is achieved through the following technical solutions:

[0007] The present invention discloses a method for generating dynamic storyboards based on a controllable diffusion model, the main steps of which are as follows:

[0008] Step 1: The user sets the keyword text for script generation and inputs the keyword text into ChatGPT to generate the script in JSON structured storage;

[0009] Step 2: Use ChatGPT to convert the script into shot language, including shot categories, camera movement methods, character positions and actions, and image descriptions.

[0010] Step 3: Use the CLIP model to perform comparative training on the image and text encoders;

[0011] Step 3.1: Construct the MPII Human Pose Descriptions dataset for CLIP model training; use the images in the dataset as the image input of the CLIP model; use the images in the dataset as the text input of the CLIP model;

[0012] Step 3.2: Construct the image encoder and text encoder of the CLIP model mapped in the same vector space;

[0013] Step 3.2.1: Construct the image encoder of the CLIP model to convert the input image into an image embedding vector, capture the deep features of the image and embed the image using the method shown in formula (1);

[0014] E I =Vit(I) (1)

[0015] Where I is the input image, E I is the embedding vector of the image, and Vit is the image encoding function.

[0016] Step 3.2.2: Construct the text encoder of the CLIP model to convert the text description into the corresponding text embedding vector, capture the semantic information of the text and perform text embedding as shown in formula (2);

[0017] E T = Transformer(T) (2)

[0018] Where T is the input text, E T is the embedding vector of the text, and Transformer is the text encoder.

[0019] Step 3.3: Use the CLIP model to perform comparative training on the image and text encoders.

[0020] Step 3.3.1: Fine-tune the image encoder and text encoder using the same vector space mapping during training using the method shown in Equation (3);

[0021] L total =Lcontrastive +λ·L regularization (3)

[0022] Among them, L regularization L is a regularization term used to prevent the model from overfitting, and λ is the regularization weight. contrastive The loss optimization is performed using the method shown in formula (4);

[0023]

[0024] in, and are the embedding vectors of the i-th image and text respectively, τ is the temperature parameter that controls the scaling of the similarity, and are the other vectors in all image and text pairs.

[0025] Step 3.3.2: Use the AutoAugment method to perform data augmentation on the image to improve the generalization ability of the CLIP model;

[0026] Step 3.3.3: Use the method shown in formula (5) to optimize the image-text alignment during training;

[0027]

[0028] Among them, θ t is the model parameter of the tth iteration, m t and v t is the gradient moment estimate of the Adam algorithm, η is the learning rate, and ∈ is a small constant to prevent division by zero errors.

[0029] Step 3.3.4: Obtain action reference images through contrast training as a condition for generating storyboard images;

[0030] Step 4: Obtain the skeleton key points of the character through the OpenPose model on the action reference image, and then convert the skeleton key points into the image character action;

[0031] Step 4.1: Use the OpenPose model to obtain the skeleton key points from the action reference image, and further form the coordinate set of the skeleton key points as shown in formula (6);

[0032] K={k1,k2,…k n},k i =(x i ,y i ) (6)

[0033] Among them, k i is the coordinate of the i-th bone point.

[0034] Step 4.2: Use the coordinate set of the skeleton key points to set the coordinate position of the character in the coordinate system using the method shown in formula (7) and formula (8);

[0035]

[0036] Among them, (x i ,y i ) is the coordinate of the skeleton key point in the original coordinate system, indicating the position of the character in the initial coordinate system. x Is the horizontal scaling factor, which represents the ratio of the new coordinate system width W to the original coordinate system width W0. y It is the scaling factor in the vertical direction, which represents the ratio of the height H of the new coordinate system to the height H0 of the original coordinate system. is the coordinate after scaling transformation, which means the result after scaling the original coordinate. Formula (7) is used for scaling. x ,t y ) is the translation vector, which means moving the skeleton point set to the correct position. Formula (8) is used for translation.

[0037] Step 4.3: Generate the image character action by using the coordinate position and the skeleton key point association;

[0038] Step 5: Use the Stable Diffusion model and ControlNet model to perform fine-grained control of the character's actions in the image and generate storyboard images;

[0039] Step 5.1: Use the ControlNet model to extract fine-grained features of the image character action using the method shown in formula (9) to obtain fine-grained action vectors;

[0040] z control =φ control (C control ) (9)

[0041] where Φ control Represents a conditional feature extraction network, which controls the input C control Converted to a fine-grained action vector z control . Ensure that the control information is compatible with the latent space of Stable Diffusion.

[0042] Step 5.2: Input the lens language into the text encoder of the Stable Diffusion model for text embedding to obtain the text embedding vector shown in formula (10);

[0043] z text =Encoder CLIP (C text ) (10)

[0044] Among them, the input lens language is C text . Encoder CLIP It is the text encoder of the CLIP (Contrastive Language-Image Pretraining) model, which converts the input text into a text embedding vector z text .

[0045] Step 5.3: Denoise and perform feature vector fusion on the randomly sampled image that conforms to the standard normal distribution;

[0046] Step 5.3.1: Perform feature vector fusion on the text embedding vector and the fine-grained action vector using the method shown in Equation (11);

[0047]

[0048] Among them, h l-1 It is the feature vector of the previous layer, which represents the feature information extracted by the network in the previous layer. l Is the feature vector of the current layer, which represents the feature representation after calculation of this layer. l It is a convolution operation on the feature vector of the previous layer, used to extract deeper features. is the fine-grained action vector z control The convolution operation is performed to extract its features and integrate them into the overall representation.

[0049] Step 5.3.2: De-noise the image’s latent vector using the method shown in Equation (12);

[0050]

[0051] Among them, z t is the potential vector of the current time step, which represents the latent space representation of the model at this time step. t-1 is the denoised latent vector, which represents a clearer representation after removing certain noise. θ is the output of the noise prediction network, which depends on the time step t, the text embedding z text and fine-grained action vector z control To estimate the current noise. t is the denoising coefficient corresponding to time step t, which is used to adjust the denoising process. t is the noise weight parameter, which controls the influence of noise. ∈ is the standard Gaussian noise, which provides randomness for the denoising process.

[0052] Step 5.4: Execute step 5.3 in an iterative loop until a clear storyboard image is obtained;

[0053] Step 5.5: Using VAE Decoder VAE Decode the latent vector to obtain the storyboard image;

[0054] x output =Decoder VAE (z0) (13)

[0055] Among them, the obtained split-mirror image is x output , the denoised latent vector obtained by step 5.4 is z0.

[0056] Step 6: Use the Pika Labs model to generate a dynamic video from the storyboard images.

[0057] The present invention discloses a dynamic storyboard generation system based on a controllable diffusion model, which includes a script generation module, a lens language generation module, an image generation module and a dynamic storyboard integration module.

[0058] The script generation module is used to generate a complete script based on the creative core and story settings input by the user. Users provide the system with basic information, including the main plot, character relationships, and scene settings, through an interactive interface. The script generation module uses a large language model to process this input and generate a script with coherent narrative and clear logic. The generation process supports multiple rounds of user interaction, allowing users to modify and improve the script to ensure that the output script meets the creator's expectations.

[0059] The lens language generation module is used to convert the narrative content in the text script into lens language; by analyzing the narrative structure of the script, the module refines each scene, including defining the appropriate shot size, lens movement method and the position of the character in the picture.

[0060] The image generation module converts the shot description into a specific storyboard. This module uses a fine-tuned CLIP model to parse the text description and retrieve the corresponding action reference image. The system utilizes a ControlNet network combined with a Stable Diffusion model to generate images and text scene descriptions that match the description based on matched reference footage.

[0061] The dynamic storyboard integration module is used to integrate static storyboard images into a dynamic storyboard video. This module uses a dynamic video generation tool to perform time sequence processing on the static images and simulates the camera movement effects and character action effects based on the motion information in the shot language description.

[0062] Beneficial effects:

[0063] 1. This invention addresses the reliance of screenplays and storyboard designs on the creators' professional experience in film and television production by introducing a controlled diffusion model and various AI technologies. By automatically generating scripts and converting them into shot language, it effectively improves the accessibility and operability of creative production.

[0064] 2. By optimizing the lens language generation and image generation modules, this invention accurately matches the character action requirements set by the user when constructing complex action scenes. This solves the problem of insufficient matching of existing AI technologies in complex action scenes.

[0065] 3. This invention utilizes a dynamic storyboard integration module to convert static storyboards into dynamic videos that meet creative requirements, further resolving the unintuitive nature of traditional static storyboard previews. By simulating camera positions and character movements, it ensures creative continuity across different production stages, avoiding gaps between storyboarding and subsequent work. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic flow chart of the dynamic storyboard generation system of the present invention;

[0067] Figure 2 This is a schematic diagram of the CLIP model structure. DETAILED DESCRIPTION

[0068] In order to better illustrate the purpose and advantages of the present invention, the following is a further description of the invention in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the scope of protection of the present invention.

[0069] Example

[0070] like Figure 1 As shown, the present embodiment is a method for generating dynamic storyboards based on a controllable diffusion model, and the specific implementation steps are as follows:

[0071] Step 1: The user sets the keyword text for script generation and inputs the keyword text into ChatGPT to generate the script in JSON structured storage;

[0072] In the embodiment, the user provides preliminary information about the script through an interactive interface, including the core of the story, such as the main conflict, character relationships, and background settings. After the system receives this content, it inputs it into the ChatGPT model for multiple rounds of interaction, and the model generates a preliminary script based on these inputs. The generated script includes detailed descriptions of each chapter, the lines and actions of the characters, and the script content is stored in JSON format. This ensures structured storage of the script, which facilitates subsequent processing and modification. Each character's lines, background settings, and plot advancement are clearly reflected in the JSON file, which is convenient for calling and processing of subsequent modules.

[0073] Step 2: Use ChatGPT to convert the script into shot language, including shot categories, camera movement methods, character positions and actions, and image descriptions.

[0074] In this embodiment, the system inputs the script content obtained in step 1 into the ChatGPT model, which then generates a shot language based on the plot and description information in the script. This includes determining the shot type for each scene, such as "medium shot" or "close-up," and the camera movement type for each scene, such as "push," "pull," or "pan." Furthermore, the system determines the coordinates of the characters in the scene based on the character action descriptions in the script. The resolution of the storyboard images is specified as 1980 pixels x 1080 pixels.

[0075] Step 3: Use the CLIP model to perform comparative training on the image and text encoders;

[0076] Step 3.1: Construct the MPII Human Pose Descriptions dataset for CLIP model training; use the images in the dataset as the image input of the CLIP model; use the images in the dataset as the text input of the CLIP model;

[0077] Step 3.2: Construct the image encoder and text encoder of the CLIP model mapped in the same vector space;

[0078] Step 3.2.1: Construct the image encoder of the CLIP model to convert the input image into an image embedding vector, capture the deep features of the image and embed the image using the method shown in formula (1);

[0079] E I =Vit(I) (1)

[0080] Where I is the input image, E I is the embedding vector of the image, and Vit is the image encoding function.

[0081] Step 3.2.2: Construct the text encoder of the CLIP model to convert the text description into the corresponding text embedding vector, capture the semantic information of the text and perform text embedding as shown in formula (2);

[0082] E T =Transformer(T) (2)

[0083] Where T is the input text, E T is the embedding vector of the text, and Transformer is the text encoder.

[0084] Step 3.3: Use the CLIP model to perform comparative training on the image and text encoders.

[0085] Step 3.3.1: Fine-tune the image encoder and text encoder using the same vector space mapping during training using the method shown in Equation (3);

[0086] L total =L contrasive +λ·L regularization (3)

[0087] Among them, L regularization L is a regularization term used to prevent the model from overfitting, and λ is the regularization weight. contrastive The loss optimization is performed using the method shown in formula (4);

[0088]

[0089] in, and are the embedding vectors of the i-th image and text respectively, τ is the temperature parameter that controls the scaling of the similarity, and are the other vectors in all image and text pairs.

[0090] Step 3.3.2: Use the AutoAugment method to perform data augmentation on the image to improve the generalization ability of the CLIP model;

[0091] Step 3.3.3: Use the method shown in formula (5) to optimize the image-text alignment during training;

[0092]

[0093] Among them, θ t is the model parameter of the tth iteration, m t and v t is the gradient moment estimate of the Adam algorithm, η is the learning rate, and ∈ is a small constant to prevent division by zero errors.

[0094] Step 3.3.4: Obtain action reference images through contrast training as a condition for generating storyboard images;

[0095] In the embodiment, Figure 2 As shown, the system uses the CLIP model (Contrastive Language-Image Pretraining) to perform contrast training on images and text. First, the system collects and constructs the MPII Human PoseDescriptions dataset. The images in the dataset serve as input to the CLIP model, and each image corresponds to a detailed text description. The system uses these image and text pairs for training, converting the image and text into their respective vector representations. During this process, the system adjusts the image encoder and text encoder in the CLIP model to ensure that the two can be mapped in the same vector space. Through this training step, the system can align the semantics between images and text in order to generate more accurate and demand-compliant action reference images.

[0096] Step 4: Obtain the skeleton key points of the character through the OpenPose model on the action reference image, and then convert the skeleton key points into the image character action;

[0097] Step 4.1: Use the OpenPose model to obtain the skeleton key points from the action reference image, and further form the coordinate set of the skeleton key points as shown in formula (6);

[0098] K={k1,k2,…k n},k i =(x i ,y i ) (6)

[0099] Among them, k i is the coordinate of the i-th bone point.

[0100] Step 4.2: Use the coordinate set of the skeleton key points to set the coordinate position of the character in the coordinate system using the method shown in formula (7) and formula (8);

[0101]

[0102] Among them, (x i ,y i ) is the coordinate of the skeleton key point in the original coordinate system, indicating the position of the character in the initial coordinate system. x Is the horizontal scaling factor, which represents the ratio of the new coordinate system width W to the original coordinate system width W0. y It is the scaling factor in the vertical direction, which represents the ratio of the height H of the new coordinate system to the height H0 of the original coordinate system. is the coordinate after scaling transformation, which means the result after scaling the original coordinate. Formula (7) is used for scaling. x ,t y ) is the translation vector, which means moving the skeleton point set to the correct position. Formula (8) is used for translation.

[0103] Step 4.3: Generate the image character action by using the coordinate position and the skeleton key point association;

[0104] In this embodiment, the system first retrieves action reference images that match the lens language description using a fine-tuned CLIP model. The system then processes the characters in these images using the OpenPose model to extract the skeletal key points of each character. The OpenPose model accurately detects the locations of key joints in the human skeleton, such as the head, shoulders, wrists, and knees. By locating the skeletal key points of the characters in the image, the system obtains the two-dimensional coordinate information of these joints and generates a dynamic description of the skeleton.

[0105] Step 5: Use the Stable Diffusion model and ControlNet model to perform fine-grained control of the character's actions in the image and generate storyboard images;

[0106] Step 5.1: Use the ControlNet model to extract fine-grained features of the image character action using the method shown in formula (9) to obtain fine-grained action vectors;

[0107] z control =φ control (C control ) (9)

[0108] where Φ control Represents a conditional feature extraction network, which controls the input C control Converted to a fine-grained action vector z control . Ensure that the control information is compatible with the latent space of Stable Diffusion.

[0109] Step 5.2: Input the lens language into the text encoder of the Stable Diffusion model for text embedding to obtain the text embedding vector shown in formula (10);

[0110] z text =Encoder CLIP (C text ) (10)

[0111] Among them, the input lens language is C text . Encoder CLIPIt is the text encoder of the CLIP (Contrastive Language-Image Pretraining) model, which converts the input text into a text embedding vector z text .

[0112] Step 5.3: Denoise and perform feature vector fusion on the randomly sampled image that conforms to the standard normal distribution;

[0113] Step 5.3.1: Perform feature vector fusion on the text embedding vector and the fine-grained action vector using the method shown in Equation (11);

[0114]

[0115] Among them, h l-1 It is the feature vector of the previous layer, which represents the feature information extracted by the network in the previous layer. l Is the feature vector of the current layer, which represents the feature representation after calculation of this layer. l It is a convolution operation on the feature vector of the previous layer, used to extract deeper features. is the fine-grained action vector z control The convolution operation is performed to extract its features and integrate them into the overall representation.

[0116] Step 5.3.2: De-noise the image’s latent vector using the method shown in Equation (12);

[0117]

[0118] Among them, z t is the potential vector of the current time step, which represents the latent space representation of the model at this time step. t-1 is the denoised latent vector, which represents a clearer representation after removing certain noise. θ is the output of the noise prediction network, which depends on the time step t, the text embedding z text and fine-grained action vector z control To estimate the current noise. t is the denoising coefficient corresponding to time step t, which is used to adjust the denoising process. t is the noise weight parameter, which controls the influence of noise. ∈ is the standard Gaussian noise, which provides randomness for the denoising process.

[0119] Step 5.4: Execute step 5.3 in an iterative loop until a clear storyboard image is obtained;

[0120] Step 5.5: Using VAE Decoder VAE Decode the latent vector to obtain the storyboard image;

[0121] x output =Decoder VAE (z0) (13)

[0122] Among them, the obtained split-mirror image is x output , the denoised latent vector obtained by step 5.4 is z0.

[0123] In this embodiment, the system uses a combination of the Stable Diffusion model and the ControlNet model to perform fine-grained control of the actions of characters in images. First, the coordinates of the character's skeleton key points extracted by the OpenPose model are used as control input, and the ControlNet model is used to generate action vectors.

[0124] The ControlNet model combines fine-grained motion control information with the Stable Diffusion model to ensure that the generated image accurately reflects the character's specific movements and positions. During this process, the system optimizes the motion descriptions in the shot language to ensure that the generated image is consistent with the original script description, ensuring that the image and motion performance meet the creative requirements.

[0125] Step 6: Use the Pika Labs model to generate a dynamic video from the storyboard images.

[0126] In the embodiment, the system uses the Pika Labs model to convert static storyboard images into dynamic videos. First, the system integrates multiple static storyboard images and performs timing processing according to the motion information in the lens language, such as "push lens", "pull lens" or "pan lens", to simulate the camera's shooting position motion effect. Since Pika Labs does not provide a direct API interface, the system indirectly realizes the image-to-video conversion through a custom-developed interface. This step can turn the static storyboard images into video clips with dynamic effects. The system will eventually splice these dynamic storyboard clips together to generate a complete dynamic storyboard video, simulating the actual shooting effect and achieving the desired effect.

[0127] This embodiment discloses a dynamic storyboard generation system based on a controllable diffusion model, which includes a script generation module, a shot language generation module, an image generation module, and a dynamic storyboard integration module.

[0128] The script generation module is used to generate a complete script based on the creative core and story settings input by the user. Users provide the system with basic information, including the main plot, character relationships, and scene settings, through an interactive interface. The script generation module uses a large language model to process this input and generate a script with coherent narrative and clear logic. The generation process supports multiple rounds of user interaction, allowing users to modify and improve the script to ensure that the output script meets the creator's expectations.

[0129] The lens language generation module is used to convert the narrative content in the text script into lens language; by analyzing the narrative structure of the script, the module refines each scene, including defining the appropriate shot size, lens movement method and the position of the character in the picture.

[0130] The image generation module converts the shot description into a specific storyboard. This module uses a fine-tuned CLIP model to parse the text description and retrieve the corresponding action reference image. The system utilizes a ControlNet network combined with a Stable Diffusion model to generate images and text scene descriptions that match the description based on matched reference footage.

[0131] The dynamic storyboard integration module is used to integrate static storyboard images into a dynamic storyboard video. This module uses a dynamic video generation tool to perform time sequence processing on the static images and simulates the camera movement effects and character action effects based on the motion information in the shot language description.

[0132] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating dynamic storyboards based on a controllable diffusion model, characterized by: The following steps are included: Step 1: The user sets the keyword text for script generation and inputs the keyword text into ChatGPT to generate the script in JSON structured storage; Step 2: Use ChatGPT to convert the script into shot language, including shot categories, camera movement methods, character positions and actions, and image descriptions. Step 3: Use the CLIP model to perform comparative training on the image and text encoders; Step 3.1: Construct the MPII Human Pose Descriptions dataset for CLIP model training; use the images in the dataset as the image input of the CLIP model; use the images in the dataset as the text input of the CLIP model; Step 3.2: Construct the image encoder and text encoder of the CLIP model mapped in the same vector space; Step 3.3: Use the CLIP model to perform comparative training on the image and text encoders. Step 4: Obtain the skeleton key points of the character through the OpenPose model on the action reference image, and then convert the skeleton key points into the image character action; Step 4.1: Use the OpenPose model to obtain the skeleton key points from the action reference image, and further form the coordinate set of the skeleton key points as shown in formula (6); K={k1,k2,…k n },k i =(x i ,y i ) (6) Among them, k i is the coordinate of the i-th bone point; Step 4.2: Use the coordinate set of the skeleton key points to set the coordinate position of the character in the coordinate system using the method shown in formula (7) and formula (8); Among them, (x i ,y i ) is the coordinate of the skeleton key point in the original coordinate system, indicating the position of the character in the initial coordinate system; α x Is the horizontal scaling factor, which represents the ratio of the new coordinate system width W to the original coordinate system width W0; α y It is the scaling factor in the vertical direction, which represents the ratio of the height H of the new coordinate system to the height H0 of the original coordinate system; is the coordinate after scaling transformation, which means the result after scaling the original coordinate; Formula (7) is used for scaling; t=(t x ,t y ) is the translation vector, which means moving the skeleton point set to the correct position; Formula (8) is used for translation; Step 4.3: Generate the image character action by using the coordinate position and the skeleton key point association; Step 5: Use the Stable Diffusion model and ControlNet model to perform fine-grained control of the character's actions in the image and generate storyboard images; Step 5.1: Use the ControlNet model to extract fine-grained features of the image character action using the method shown in formula (9) to obtain fine-grained action vectors; z control =Φ control (C control ) (9) where Φ control Represents a conditional feature extraction network, which controls the input C control Converted to a fine-grained action vector z control ; Ensure that the control information is compatible with the potential space of Stable Diffusion; Step 5.2: Input the lens language into the text encoder of the Stable Diffusion model for text embedding to obtain the text embedding vector shown in formula (10); z text =Encoder CLIP (C text ) (10) Among them, the input lens language is C text ;Encoder CLIP is the text encoder of the CLIP model, which converts the input text into a text embedding vector z text ; Step 5.3: Denoise and perform feature vector fusion on the randomly sampled image that conforms to the standard normal distribution; Step 5.4: Execute step 5.3 in an iterative loop until a clear storyboard image is obtained; Step 5.5: Using VAE Decoder VAE Decode the latent vector to obtain the storyboard image; x output =Decoder VAE (z0) (13) Among them, the obtained split-mirror image is x output , the denoised latent vector obtained in step 5.4 is z0; Step 6: Use the Pika Labs model to generate a dynamic video from the storyboard images.

2. The method for generating dynamic storyboards based on a controllable diffusion model according to claim 1, wherein: The implementation method of step 3.2 is: Step 3.2.1: Construct the image encoder of the CLIP model to convert the input image into an image embedding vector, capture the deep features of the image and embed the image using the method shown in formula (1); It is I =Vit(I) (1) Where I is the input image, E I is the embedding vector of the image, Vit is the image encoding function; Step 3.2.2: Construct the text encoder of the CLIP model to convert the text description into the corresponding text embedding vector, capture the semantic information of the text and perform text embedding as shown in formula (2); E T =Transformer(T) (2) Where T is the input text, E T is the embedding vector of the text, and Transformer is the text encoder.

3. The method for generating dynamic storyboards based on a controllable diffusion model according to claim 1, wherein: The implementation method of step 3.3 is: Step 3.3.1: Fine-tune the image encoder and text encoder using the same vector space mapping during training using the method shown in Equation (3); L total =L contrastive +λ·L regularization (3) Among them, L regularization is a regularization term used to prevent the model from overfitting, and λ is the regularization weight; L contrastive The loss optimization is performed using the method shown in formula (4); in, and are the embedding vectors of the i-th image and text respectively, τ is the temperature parameter that controls the scaling of the similarity, and are the other vectors in all image and text pairs; Step 3.3.2: Use the AutoAugment method to perform data augmentation on the image to improve the generalization ability of the CLIP model; Step 3.3.3: Use the method shown in formula (5) to optimize the image-text alignment during training; Among them, θ t is the model parameter of the tth iteration, m t and v t is the gradient moment estimate of the Adam algorithm, η is the learning rate, and ∈ is a small constant to prevent division by zero errors; Step 3.3.4: Obtain action reference images through contrast training as a condition for generating storyboard images.

4. The method for generating dynamic storyboards based on a controllable diffusion model according to claim 1, wherein: Step 5.3 is implemented as follows: Step 5.3.1: Perform feature vector fusion on the text embedding vector and the fine-grained action vector using the method shown in Equation (11); Among them, h l-1 is the feature vector of the previous layer, which represents the feature information extracted by the network in the previous layer; h l Is the feature vector of the current layer, which represents the feature representation after calculation of this layer; Conv l It is a convolution operation on the feature vector of the previous layer, used to extract deeper features; is the fine-grained action vector z control The convolution operation is performed to extract its features and integrate them into the overall representation; Step 5.3.2: De-noise the image’s latent vector using the method shown in Equation (12); Among them, z t is the potential vector of the current time step, which represents the latent space representation of the model at this time step; z t-1 is the denoised latent vector, which represents a clearer representation after removing certain noise; ∈ θ is the output of the noise prediction network, which depends on the time step t, the text embedding z text and fine-grained action vector z control To estimate the current noise; α t is the denoising coefficient corresponding to time step t, which is used to adjust the denoising process; σ t is the noise weight parameter, which controls the influence of noise; ∈ is the standard Gaussian noise, which provides randomness for the denoising process.

5. A dynamic storyboard generation system based on a controllable diffusion model that implements the method of claim 1, characterized in that: It includes script generation module, lens language generation module, image generation module and dynamic storyboard integration module; The script generation module is used to generate a complete text script based on the creative core and story settings input by the user. The user provides the system with basic information including the main plot, character relationships and scene settings through an interactive interface. The script generation module uses a large language model to process the input content and generate a script text with coherent narrative and clear logic. The generation process supports multiple rounds of user interaction, allowing users to modify and improve the script to ensure that the output script meets the creator's expectations; The lens language generation module is used to convert the narrative content in the text script into lens language. By analyzing the narrative structure of the script, the module refines each scene, including defining the appropriate shot size, lens movement method, and the position of the character in the picture. The image generation module is used to convert the shot language description into a specific storyboard. This module uses the fine-tuned CLIP model to parse the text description and retrieve the corresponding action reference image. The system uses the ControlNet network combined with the StableDiffusion model to generate images and text scene descriptions that match the description based on the matched reference materials. The dynamic storyboard integration module is used to integrate static storyboard images into dynamic storyboard videos; this module uses a dynamic video generation tool to perform time sequence processing on the static images, and simulates the camera movement effects and character action effects based on the motion information in the lens language description.

Images