A method and system for automatic shot breakdown and keyframe generation
By using artificial intelligence to analyze storyboards and adaptively generate keyframe images, the problems of camera motion trajectory recognition and material management in existing technologies have been solved. This enables automated management and efficient production of in-vehicle visual materials, improving the smoothness of in-vehicle animation and driving safety.
Patent Information
- Application Number
- CN202611279299.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-21
- Publication Date
- 2026-11-10
AI Technical Summary
Existing AI image generation tools cannot recognize and analyze the camera movement trajectory in storyboards, nor can they adaptively determine the number of keyframes, resulting in redundant static shots or missing frames in dynamic shots. They cannot guarantee the consistency of multiple frames and lack a material management system, making it difficult to meet the batch production needs of highly reliable visual content such as in-vehicle interactive animation and autonomous driving simulation visualization.
By using artificial intelligence to analyze the semantic information of shots in the storyboard, the number of keyframes is adaptively determined, and a global consistency constraint mechanism is used to generate keyframe images, thereby achieving automated management and batch generation of storyboard keyframes.
It enables automated management of in-vehicle visual materials, reduces manual image editing workload, improves driving safety, adapts to the smoothness of in-vehicle animation, supports industrial production of multiple vehicle models and multiple cockpit modes, and broadens the application scope of generative AI in the automotive field.
Smart Images

Figure CN123075237A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent cockpits and human-computer interaction, and in particular to a method and system for automatic shot breakdown and keyframe generation. Background Technology
[0002] With the rapid development of generative artificial intelligence technology, AI-based image generation tools have been widely used in film and television, creative content, and automotive visual materials production scenarios. The current mainstream content production process generally adopts the model of first generating storyboards by artificial intelligence, and then generating storyboards one by one based on the storyboards. Although this model can produce single high-quality images, it has many inherent defects in parsing and processing professional storyboards, making it difficult to meet the batch production needs of industrial-grade, highly continuous visual materials such as automotive interactive animation, autonomous driving simulation visualization, and cockpit digital human motion effects.
[0003] First, existing conventional image generation systems only support independent prompts for a single image. Their semantic understanding capabilities are limited to static image descriptions. They cannot recognize or parse professional camera movement language such as push-in, pull-out, tracking, circling, and panning in storyboards. They cannot automatically recognize camera movement trajectories and camera position change logic, and cannot adapt dynamic image generation logic based on camera movement characteristics. They can only output isolated static images, lacking the progressive relationship of the image brought about by camera movement.
[0004] Secondly, existing technologies cannot adaptively determine the number of keyframes required for a single shot based on the dynamic information of the shot. The number of keyframes depends entirely on manual setting based on human experience. Low-dynamic static shots are prone to redundant keyframes, resulting in wasted material storage and rendering computing power. High-dynamic shots with high-speed movement and multi-stage changes are prone to losing core visual nodes such as motion transitions and climaxes due to insufficient frame count, resulting in a break in the dynamic narrative logic. This defect can also lead to safety hazards such as incomplete driver information reception and failure of driving prompt display in vehicle-mounted dynamic warning animations and autonomous driving dangerous situation visualization scenarios.
[0005] Furthermore, traditional image generation tools lack a consistency constraint mechanism for multi-frame linkage. When generating multiple keyframes continuously in the same shot, it is easy to encounter problems such as character subject, vehicle appearance, scene environment, lighting and color tone, and shot logic drift and distortion. It is impossible to guarantee the consistency of the main image, scene atmosphere, and lighting style in continuous images, which greatly increases the workload of manual retouching and alignment calibration in post-production. It is difficult to adapt to the stringent requirements of the continuity of images for vehicle-mounted digital humans and vehicle dynamic display animations.
[0006] Finally, the existing image generation process lacks a complete project-based automated management system. After keyframe generation, there are no standardized automatic naming, classification, archiving, or metadata retention functions. All material organization and version recording work relies on manual completion. When mass-producing automotive visual materials for multiple vehicle models, multiple cockpit modes, and multiple simulation conditions, problems such as chaotic material files, difficulty in version traceability, and poor reusability of materials across projects are prone to occur, making it impossible to support the industrialized and standardized visual material pipeline operations of automakers.
[0007] In summary, existing AI image generation solutions focus on the quality of single image generation, without systematically designing the architecture for professional shot semantics, dynamic layering logic, continuity of shots, and engineering material management in storyboards. This limits their application in high-reliability, high-volume, and highly continuous visual content production scenarios such as in-vehicle intelligent cockpits and autonomous driving simulation visualization. The industry urgently needs an intelligent storyboard keyframe generation solution that can automatically parse shot language, adaptively split keyframes, ensure the consistency of subjects across multiple frames, and achieve automated material management. Summary of the Invention
[0008] In view of this, the purpose of this invention is to provide a method and system for automatic storyboard decomposition and keyframe generation, which aims to automatically determine the number of keyframes based on shot type, dynamic level, subject movement changes, and shot duration by parsing the shot semantic information in the storyboard script. It also decomposes dynamic shots into multiple shot stages, generates corresponding keyframe prompts for each stage, and calls images to generate keyframe images. This achieves automatic generation, naming, saving, and management of storyboard keyframes.
[0009] This invention provides the following solution: According to one aspect of the present invention, an automatic shot breakdown and keyframe generation method is provided, comprising the following steps: S1. Obtain a standardized storyboard, which includes camera movement, subject action, scene information, lighting information, and style requirements. S2. The storyboard is subjected to feature extraction using an artificial intelligence semantic parsing model to obtain the camera movement features and subject dynamic features. S3. The dynamic level of the shot is classified according to the camera movement characteristics and the dynamic characteristics of the subject. The total number of key frames of the corresponding shot is adaptively determined, and the continuous dynamic shot is decomposed into multiple sequentially connected shot stages. S4. Generate differentiated keyframe images and prompts for each shot stage; S5. Overlay global consistency constraints on all keyframe cue words for the same shot to obtain the final generated cue words for each keyframe. S6. Automatically call the image generation system to generate corresponding keyframe images in batches based on the final generation prompts for each keyframe.
[0010] Furthermore, methods for obtaining standardized storyboards include: Directly input the structured storyboard script; Input a plot summary and style description in natural language, and the large language model will automatically parse and generate a standardized storyboard script that conforms to preset field specifications.
[0011] Furthermore, the output of feature extraction also includes basic lens attributes and visual style features; The basic attributes of a lens include lens number, shot duration, shot type, and initial camera position; Visual style features include scene environment, lighting direction, color tone, emotional atmosphere, and art style.
[0012] Furthermore, based on the combination of camera movement and subject movement, the dynamic range of the shot is divided into four levels from low to high: Low dynamic range: no camera movement and no subject movement; Medium-motion: Only camera movement exists, or only the subject moves; High dynamic range: Simultaneous camera movement and subject movement, with a moderate range of motion; Extremely high dynamic range: Simultaneous high-speed camera movement and large-scale subject movement.
[0013] Furthermore, the mapping rule between the number of keyframes and the shot stage is as follows: A low dynamic range shot corresponds to one keyframe, which is divided into the initial stage; The medium-motion shot corresponds to 2 keyframes, divided into two stages: the beginning and the end; The high dynamic range shot corresponds to 3 keyframes, which are divided into three stages: start, development, and end. The extremely high dynamic range shot corresponds to 4 keyframes, divided into four stages: beginning, development, climax, and ending.
[0014] Furthermore, there are preset matching rules for typical shots and dynamic range levels: Fixed lens matching low dynamic range; Slow-motion zoom shots and fixed shots are used to match the dynamic level of the subject's movement. The tracking camera is matched with a high dynamic range; High-speed action shots, surround shots, and long shots are matched with extremely high dynamic range.
[0015] Furthermore, the image generation prompts for each keyframe include a description of the positive scene, camera language, shot composition, lighting settings, subject action status, art style, color tone, dynamic effects, and consistency constraint indicators. Keyframe cue words in adjacent stages retain the basic constraints of the subject and scene, while also marking the changes in camera movement trajectory and subject action in this stage relative to the previous stage.
[0016] Furthermore, global consistency constraints employ global prompt word constraints and a reference graph anchoring mechanism; Global cue word constraints include: consistent main appearance, consistent character clothing, consistent core product form, consistent scene environment, consistent lighting direction, consistent color style, and consistent camera logic.
[0017] Furthermore, the reference image anchoring mechanism is as follows: the stage frame with the most complete semantics and the most representative image in the same shot is selected as the benchmark reference frame. When generating subsequent keyframes, the benchmark reference frame is used as the image reference input to anchor the subject image and scene style features.
[0018] According to two aspects of the present invention, an automatic shot breakdown and keyframe generation system is provided, comprising: The module includes a storyboard acquisition module, a feature extraction module, a shot breakdown module, a prompt word generation module, a final prompt word generation module, and a keyframe image generation module. The storyboard acquisition module is used to acquire standardized storyboards, which include camera movement methods, main body actions, scene information, lighting information and style requirements. The feature extraction module is used to extract features from the storyboard script through an artificial intelligence semantic parsing model to obtain the camera movement features and subject dynamic features. The lens decomposition module is used to classify the dynamic level of the lens according to the camera movement characteristics and the dynamic characteristics of the subject, adaptively determine the total number of keyframes of the corresponding lens, and decompose the continuous dynamic lens into multiple sequentially connected lens stages. The prompt word generation module is used to generate different keyframe images and prompt words for each shot stage; The final prompt word generation module is used to overlay global consistency constraints on all keyframe prompt words of the same shot to obtain the final generated prompt words for each keyframe. The keyframe image generation module is used to automatically call the image generation system and generate corresponding keyframe images in batches based on the final generation prompts for each keyframe.
[0019] Compared with the prior art, the present invention has the following advantages: This invention utilizes an AI-powered lens semantic analysis module to autonomously identify various professional camera techniques in storyboards, such as push-in shots, pull-out shots, follow shots, and surround shots, without requiring manual analysis of camera movement logic. For scenarios such as in-vehicle cockpit animations, ADAS risk visualizations, and vehicle promotional videos, structured analysis can be completed simply by inputting a natural language script, eliminating the need for designers to manually analyze camera positions and movement changes, and significantly shortening the pre-production cycle of in-vehicle visual materials.
[0020] This invention determines the number of keyframes based on lens type, dynamic range, and subject movement amplitude: low-dynamic lenses generate only a small number of keyframes, reducing the rendering load on the vehicle's infotainment system and saving onboard storage and bandwidth; high-dynamic, high-risk warning lenses are automatically split into multi-stage keyframes to fully recreate the entire motion process. On the one hand, this avoids computational redundancy or image gaps caused by a fixed frame rate, ensuring smooth and uninterrupted animations for the vehicle's instrument panel, central control unit, and HUD; on the other hand, it fully displays the vehicle's operating conditions and the evolution logic of dangerous scenarios, ensuring that the driver receives complete driving warning information and improving driving safety.
[0021] This invention employs a dual control mechanism combining global consistency constraints and reference image anchoring. It uniformly constrains core visual elements such as subject appearance, vehicle shape, character clothing, scene environment, lighting and color, and shot logic across all keyframes within the same shot. This addresses the root causes of issues like subject drift, stylistic inconsistencies, and vehicle shape distortion in traditional solutions across multiple consecutive frames. When applied to in-vehicle digital human interaction animations, cockpit welcome effects, and autonomous driving simulation sequences, it eliminates the need for manual frame-by-frame calibration and correction, significantly reducing post-production retouching manpower and lowering the production costs of automotive visual materials for automakers.
[0022] This invention enables automated batch generation and standardized project management of keyframe images. It can automatically connect to various cloud and local image generation systems to produce images in batches, and automatically name and archive them according to project, shot, and frame number rules, while simultaneously retaining a complete set of metadata such as shot information, prompts, and generation parameters. Compared to existing technologies that lack automatic material management capabilities and rely on manual file organization, this invention can build a standardized in-vehicle visual material library, enabling traceable materials, cross-model and cross-project reuse, and adapting to the industrial production needs of automakers for synchronous iteration of multi-configuration and multi-version cockpit visual materials. It standardizes the digital content production pipeline in the automotive industry and simultaneously supports various business scenarios such as intelligent cockpit interaction, autonomous driving simulation dataset production, and storyboard creation for vehicle marketing short films, thus broadening the application scope of generative AI in the automotive field. Attached Figure Description
[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0024] Figure 1 This is a flowchart of a method for automatic shot breakdown and keyframe generation provided by one or more embodiments of the present invention.
[0025] Figure 2 This is an architecture diagram of an automatic shot breakdown and keyframe generation system provided by one or more embodiments of the present invention.
[0026] Figure 3 This is an architecture diagram of an automatic shot breakdown and keyframe generation system provided by one or more embodiments of the present invention.
[0027] Figure 4 This is a block diagram of an electronic device for an automatic shot breakdown and keyframe generation method provided by one or more embodiments of the present invention. Detailed Implementation
[0028] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0029] Figure 1 This is a flowchart of a method for automatic shot breakdown and keyframe generation provided by one or more embodiments of the present invention.
[0030] like Figure 1 As shown, it includes the following steps: S1. Obtain a standardized storyboard, which includes camera movement, subject action, scene information, lighting information, and style requirements. Specifically, standardized storyboards can be obtained through the following methods: Directly input the structured storyboard script; Input a plot summary and style description in natural language, and the large language model will automatically parse and generate a standardized storyboard script that conforms to preset field specifications.
[0031] S2. The storyboard is subjected to feature extraction using an artificial intelligence semantic parsing model to obtain the camera movement features and subject dynamic features. Specifically, the output of feature extraction also includes basic lens attributes and visual style features; The basic attributes of a lens include lens number, lens duration, shot type, and initial camera position; Visual style features include scene environment, lighting direction, color tone, emotional atmosphere, and art style.
[0032] S3. The dynamic level of the shot is classified according to the camera movement characteristics and the dynamic characteristics of the subject. The total number of key frames of the corresponding shot is adaptively determined, and the continuous dynamic shot is decomposed into multiple sequentially connected shot stages. Specifically, based on the combination of camera movement and subject movement, the dynamic range of the shot is divided into four levels from low to high: Low dynamic range: no camera movement and no subject movement; Medium-motion: Only camera movement exists, or only the subject moves; High dynamic range: Simultaneous camera movement and subject movement, with a moderate range of motion; Extremely high dynamic range: Simultaneous high-speed camera movement and large-scale subject movement.
[0033] S4. Generate differentiated keyframe images and prompts for each shot stage; Specifically, the mapping rule between the number of keyframes and the shot stage is as follows: A low dynamic range shot corresponds to one keyframe, which is divided into the initial stage; The medium-motion shot corresponds to 2 keyframes, divided into two stages: the beginning and the end; The high dynamic range shot corresponds to 3 keyframes, which are divided into three stages: start, development, and end. The extremely high dynamic range shot corresponds to 4 keyframes, divided into four stages: beginning, development, climax, and ending.
[0034] S5. Overlay global consistency constraints on all keyframe cue words for the same shot to obtain the final generated cue words for each keyframe. S6. Automatically call the image generation system to generate corresponding keyframe images in batches based on the final generation prompts for each keyframe; S7. The generated keyframe images are automatically named according to preset rules and archived by project dimension.
[0035] Furthermore, there are preset matching rules for typical shots and dynamic range levels: Fixed lens matching low dynamic range; Slow-motion zoom shots and fixed shots are used to match the dynamic level of the subject's movement. The tracking camera is matched with a high dynamic range; High-speed action shots, surround shots, and long shots are matched with extremely high dynamic range.
[0036] Furthermore, the image generation prompts for each keyframe include a description of the positive scene, camera language, shot composition, lighting settings, subject action status, art style, color tone, dynamic effects, and consistency constraint indicators. Keyframe cue words in adjacent stages retain the basic constraints of the subject and scene, while also marking the changes in camera movement trajectory and subject action in this stage relative to the previous stage.
[0037] Furthermore, global consistency constraints employ global prompt word constraints and a reference graph anchoring mechanism; Global cue word constraints include: consistent main appearance, consistent character clothing, consistent core product form, consistent scene environment, consistent lighting direction, consistent color style, and consistent camera logic.
[0038] Furthermore, the reference image anchoring mechanism is as follows: the stage frame with the most complete semantics and the most representative image in the same shot is selected as the benchmark reference frame. When generating subsequent keyframes, the benchmark reference frame is used as the image reference input to anchor the subject image and scene style features.
[0039] Figure 2 This is an architecture diagram of an automatic shot breakdown and keyframe generation system provided by one or more embodiments of the present invention.
[0040] like Figure 2 As shown, it includes: The module includes a storyboard acquisition module, a feature extraction module, a shot breakdown module, a prompt word generation module, a final prompt word generation module, a keyframe image generation module, and an archive management module. The storyboard acquisition module is used to acquire standardized storyboards, which include camera movement methods, main body actions, scene information, lighting information and style requirements. The feature extraction module is used to extract features from the storyboard script through an artificial intelligence semantic parsing model to obtain the camera movement features and subject dynamic features. The lens decomposition module is used to classify the dynamic level of the lens according to the camera movement characteristics and the dynamic characteristics of the subject, adaptively determine the total number of keyframes of the corresponding lens, and decompose the continuous dynamic lens into multiple sequentially connected lens stages. The prompt word generation module is used to generate different keyframe images and prompt words for each shot stage; The final prompt word generation module is used to overlay global consistency constraints on all keyframe prompt words of the same shot to obtain the final generated prompt words for each keyframe. The keyframe image generation module is used to automatically call the image generation system and generate corresponding keyframe images in batches based on the final generation prompts for each keyframe. The archive management module is used to automatically name the generated keyframe images according to preset rules and manage their archives by project dimension.
[0041] It is worth noting that although only some basic functional modules are disclosed in this embodiment, it does not mean that the composition of this system is limited to the above-mentioned basic functional modules. On the contrary, what this embodiment intends to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technology to form an infinite number of embodiments or technical solutions. That is to say, this system is open rather than closed. The fact that this embodiment only discloses a few basic functional modules does not mean that the scope of protection of the claims of this invention is limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above device is described separately according to its functions as various units and modules. Of course, in implementing this invention, the functions of each unit and module can be implemented in one or more software and / or hardware.
[0042] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0043] Figure 3 This is an architecture diagram of an automatic shot breakdown and keyframe generation system provided by one or more embodiments of the present invention.
[0044] like Figure 3 As shown, it fully covers six core stages: standardized input of storyboard scripts → dynamic decomposition of shot semantics → generation of differentiated keyframe prompts → batch rendering of AI images → intelligent control of inter-frame consistency → automatic naming, classification and archiving. The process is closed-loop and logically coherent, and it is suitable for AI storyboard creation of all types of videos, including short videos, short films, and promotional videos.
[0045] As the foundational input for the entire process, it strictly follows the professional film and television storyboard dimensions to complete the standardized input of all parameters for a single shot, eliminating issues such as missing parameters and vague descriptions, and providing accurate basis for subsequent AI decomposition and generation.
[0046] The core required fields for entry are: shot number, scene number, shot duration, shot type, camera movement, main subject's action, scene information, lighting information, emotional atmosphere, and style requirements.
[0047] Input guidelines: For dynamic long shots, the camera movement speed, range of motion, and dynamic changes in the image must be clearly marked; for static shots, the scene, lighting, and subject status parameters must be fixed to ensure that all parameters can be recognized, analyzed, and reused by AI. Simultaneously, the project's overall basic information must be entered, including the project name, unique project identifier, overall style, and global consistency constraints.
[0048] Based on the recorded shot duration, dynamic type, camera movement range, and degree of action change, AI automatically determines the shot attributes, performs phased breakdown of dynamic long shots, and can reduce the number of keyframes in short shots as needed, with a unified core breakdown logic.
[0049] 1. Shot Judgment: For long shots with smooth dynamics of 8 seconds or more and continuous camera movements, they are uniformly broken down into four standard stages: beginning, development, climax, and ending; 2. Time segmentation: Distribute the time nodes of each stage evenly according to the total duration, and generate a unique keyframe sequence number for each stage; 3. Feature decomposition: For each stage, the dynamic features of the shot and the core logic of the image changes are extracted independently to ensure that the four stages are progressive and the dynamics of the image are coherent, without any gaps or abrupt changes. 4. Output: Generates a structured list of keyframes containing stage name, frame number, time node, dynamic features, and screen change logic.
[0050] In one embodiment, artificial intelligence is used to break down dynamic shots into multiple stages. The number of keyframes is determined based on shot type, dynamic level, duration, and amplitude of action changes. A continuous shot is automatically broken down into stages such as beginning, development, climax, and end. Corresponding image prompts are generated for each node (while maintaining continuity, there should also be differentiation), as shown in the rule table in Table 1.
[0051] Table 1
[0052] A dynamic long take can be broken down into: the beginning stage, the development stage, the climax stage, and the ending stage, as shown in Table 2; Table 2
[0053] Abandoning the full-frame repetitive prompt word mode, AI automatically generates standardized image prompt words that are both differentiated and continuous based on the stage characteristics of each key frame. Each frame contains nine fixed dimensions to adapt to the rendering logic of the AI drawing model.
[0054] 1. Fixed Dimensions: All keyframes uniformly follow nine dimensions: "positive cue words + camera language + shot size + composition + lighting + subject action + style + color + consistency constraints"; 2. Differentiated Generation: Based on the characteristics of the initial setup, development, climax, and conclusion of the scene, fine-tune the main action, lighting effects, composition, and color texture to achieve dynamic changes in the image; 3. Continuity Guarantee: All keyframes are forcibly bound to global consistency constraints of the project, fixing the main character's appearance, clothing, scene, lighting direction, color tone, and camera logic, with only dynamic details being adjusted differently; 4. Global negative word matching: Unified configuration of universal negative warning words to avoid problems such as image distortion, jumps, blurring, deformity, and watermarks.
[0055] The system allows for the batch input of structured keyframe prompts into a multimodal image generation system, supporting cloud APIs, locally generated models, and multimodal drawing systems. Standardized parameter calls enable batch automatic image generation.
[0056] 1. Basic parameter configuration: Set cinematic aspect ratio, resolution, and sharpness parameters uniformly, lock the overall style parameters, and turn off non-target image styles; 2. Reference Frame Linkage Configuration: Enable inter-frame reference mechanism, with the next frame using the previous frame as a reference sample, and set fixed reference weights to preserve the basic structure of the image to the greatest extent. 3. Batch call logic: Strictly follow the time sequence of "FRAME1→FRAME2→FRAME3→FRAME4" to call frame by frame, with independent prompts for each frame overlaid with global constraint parameters; 4. Image Output: Batch generate high-resolution images of corresponding keyframes, preserving the original image parameters and style features.
[0057] To address issues such as subject distortion, scene abrupt changes, color banding, and light shifts that are prone to occur in AI-generated images, a dual-layer consistency control mechanism is implemented to lock global parameters throughout the entire process.
[0058] 1. Global parameter hard locking: Seven core dimensions are fixed and remain unchanged, including the main appearance, character clothing, prop form, scene environment, light direction, color style, and camera logic; 2. Technical implementation methods: First, reference image relay constraints are used to achieve the inheritance of image structure between frames; second, global prompt words are forcibly bound, and basic constraint parameters cannot be deleted in any keyframe; third, model style parameters are globally locked, with no random changes throughout the process. 3. Detailed dynamic fine-tuning: Only dynamic details such as the amplitude of movement, light and shadow, and fog density are allowed to change, ensuring the vividness of the picture while eliminating visual breaks.
[0059] Once the images are generated, the system automatically names and categorizes the files according to project rules, achieving standardized management and facilitating later editing, iteration, and reuse.
[0060] 1. Automatic naming rules: Follow the fixed formula "Project ID_Shot ID_Keyframe ID.png" to generate a unique filename for each frame; 2. Automatic classification and archiving: The system automatically creates directories by project, shot number, and keyframe level, and assigns the corresponding keyframe images to the specified folders; 3. Output: The final output is a complete set of structured, batch-accessible keyframe footage that can be directly used for video editing.
[0061] The relevant processes enable manual one-time input and AI-automated processing across the entire chain. From professional storyboard script breakdown, intelligent prompt word generation, and AI image rendering to consistency maintenance and automatic archiving, the entire process is standardized, automated, and batch reusable. It balances film-level professionalism with the efficiency of AI generation, and is perfectly adapted to various video AI storyboard creation scenarios.
[0062] In one embodiment, the implementation steps of this method are fully described using the dynamic long shot numbered shot-011 in the ancient style animation short film project kuafu as a specific implementation case, combined with the automatic generation process of storyboard-keyframe. This implementation method sequentially executes the steps of storyboard script generation, artificial intelligence shot semantic analysis, shot type identification, shot stage decomposition, keyframe number determination, keyframe prompt information generation, calling the image generation system, generating keyframe images, automatic naming and saving, and configuring a consistency control module to ensure the continuity of multi-frame images.
[0063] S01, Storyboard generation; First, a complete storyboard at the director's level is generated. The storyboard includes all parameters such as shot number, scene number, shot duration, shot size, camera movement, main subject's action, scene information, lighting information, emotional atmosphere, and style requirements. In this example, the storyboard parameters for shot-011 are as follows: Shot number: shot_011; Scene number: scene_06, the scene of chasing the sun in the wasteland; Shot duration: 6 seconds; Shot type: medium wide shot; Camera movement: a dynamic long shot with a slow forward push-in; Main action: Kuafu strides forward, swinging his arms widely, his gaze fixed on the sun high in the sky, his breathing rising and falling, and his hair fluttering in the wind; Scene information: a vast, barren Gobi desert, the ground covered with cracked yellow soil, a blazing red sun suspended on the distant horizon, and withered wild grass scattered on the ground; Lighting information: midday overhead light, strong warm golden direct sunlight, casting a long, dark silhouette of Kuafu on the ground; Emotional atmosphere: anxious, persistent, and tragically heroic; Style requirements: realistic, thick-painted Chinese style with a cinematic feel, a widescreen epic atmosphere, and an ancient Chinese animation style.
[0064] S02, AI-powered semantic analysis of camera footage; The AI module was invoked to read all fields of the complete storyboard script for shot-011 and perform semantic analysis of the shot. The analysis revealed that the shot is a forward-moving dynamic long shot with a total duration of 6 seconds. The main character, Kuafu, has a continuous and progressive running action. The shot sequence begins with Kuafu taking his first step, and as the camera zooms in on the main character until Kuafu sprints towards the sun, the character's movements become increasingly larger and the emotional intensity of the shot gradually rises. Based on the dynamic range of the shot, its duration, and the magnitude of the changes in the main character's movements, it was predicted that the shot needs to be broken down into multiple time-series action nodes.
[0065] S03, Lens type recognition; The AI analysis module identifies the current shot type, and the result is a forward-pushing dynamic narrative long shot; the dynamic level of the shot is determined to be high, with a large range of changes in the subject's movements within the shot.
[0066] S04, Lens stage disassembly; For dynamic long shots, a four-part time sequence is decomposed according to the rule of initiation phase - development phase - climax phase - conclusion phase; This embodiment decomposes the shot into four phases: Initial stage: As soon as the camera starts rolling, Kuafu takes his first step and enters the starting preparation state; Development stage: Kuafu runs steadily, leans forward, and begins to accelerate during the running process; Climax: Kuafu enters full sprint mode, his limbs stride out in large strides, and the emotional intensity of the scene reaches its peak. End phase: After the running action ends, Kuafu briefly stops and looks up at the blazing sun in the sky.
[0067] S05. Determine the number of keyframes; Each shot stage after the shot is disassembled corresponds to one independent keyframe. Therefore, shot-011 has a total of 4 keyframes, which are marked as frame-1 (start stage keyframe), frame-2 (development stage keyframe), frame-3 (climax stage keyframe), and frame-4 (end stage keyframe).
[0068] S06, Keyframe prompt information generation; The consistency control module generates global prompts, which are bound to the generation parameters of all keyframes to constrain the continuity of multiple frames. The specific global consistency constraints are: maintaining the appearance of the Kuafu character, his brown, rough-hewn ancient-style clothing, and his robust physique; maintaining a consistent environment of dry, cracked yellow earth; fixing the direction of the warm golden overhead light at midday; maintaining a unified overall Chinese-style, thick-painted epic color scheme; and using forward-pushing medium-wide shots throughout to ensure a coherent narrative logic. For four different shot stages, independent image-based positive prompts, camera language, composition, lighting, main character actions, style, and color parameters are generated. Details of the prompts for each keyframe are as follows: Frame-1 Initial Stage: The positive prompt is that the imposing, ancient-style Kuafu has just taken his first running step, his body slightly leaning forward, his hair gently swaying, his feet on the cracked yellow earth wasteland, and the bright red sun hanging high in the distance; the camera language is a push-in shot in the opening frame, a medium-wide shot; the composition uses a horizontal widescreen third-order composition, with Kuafu placed on the left side of the screen; the lighting uses warm golden hard top lighting from noon; the main action is the starting preparation, taking the first step; the style is a realistic, thick-painted Chinese style with an epic cinematic feel; the color scheme uses an earthy yellow wasteland background paired with orange-red sunlight; Frame-2 Development Stage: The positive prompt is Kuafu starting to accelerate, his legs stride forward, his arms swing back and forth, his body leans forward more, and the wind and sand blow his long hair backward; the camera language is a push-in shot in a mid-frame and a medium-wide shot; the composition uses a horizontal widescreen composition, with Kuafu moving closer to the center of the frame; the lighting uses warm golden hard top light from midday; the main action is a steady acceleration run; the style is a Chinese-style realistic thick paint, with an epic cinematic feel; the color scheme uses an earthy yellow wasteland background paired with orange-red sunlight. Frame 3 Climax: The positive cues are Kuafu sprinting with all his might, his limbs spread wide, muscles taut, head fixed on the sun high in the sky, sandstorms violently blowing his long hair and coarse linen clothes; the camera language is a push-in shot of the climax frame closest to the subject, and a medium wide shot; the composition uses a horizontal widescreen centered composition, with Kuafu at the visual center of the image; the lighting uses warm gold hard top lighting from noon; the main action is a full-speed sprint; the style is a Chinese-style realistic thick paint, with an epic cinematic feel; the color scheme uses an earthy yellow wasteland background, paired with intense and fiery orange-red strong light; Frame-4 End Phase: The positive cues are: Kuafu slows down after running, stands firmly with legs apart, looks up at the blazing sun, and his chest heaves violently; the camera language is a push-in close-up and a medium-wide shot; the composition uses a horizontal widescreen rule of thirds; the lighting uses warm golden hard top lighting from midday; the main action is stopping and looking up at the sun; the style is a realistic, thick-painted Chinese style with an epic cinematic feel; the color scheme uses an earthy yellow wasteland background paired with orange-red sunlight. The continuity of the consistency control module is achieved by: loading global cues for hard constraints during each image generation, and using a chain-like progressive reference image calling method, using the keyframe images generated in the previous stage as reference images for subsequent keyframe generation, sequentially locking the character's appearance, scene environment, and lighting parameters to avoid temporal discontinuities in the image; and fixing image style parameters throughout the keyframe generation process to ensure a consistent overall image style.
[0069] S07, Invoke the image generation system This embodiment uses a local multimodal image generation system as the image generation carrier; it sequentially loads four sets of differentiated keyframe cue information and global consistency constraint parameters; and sets the output image size to 1920. 1080p, 16:9 widescreen ratio; perform chained reference image import, frame-2 reads the finished image of frame-1 as the reference image, frame-3 reads the finished image of frame-2 as the reference image, frame-4 reads the finished image of frame-3 as the reference image; lock the style parameters of the Chinese style thick paint, enable the character consistency and scene consistency controls, and batch submit the four sets of keyframe image generation tasks.
[0070] S08. Generate keyframe images After receiving the prompts, reference images, canvas size, and style constraint parameters, the image generation system outputs four keyframe images with differentiated temporal actions and consistent subject-scene-lighting. These correspond to the starting preparation keyframe, the acceleration running keyframe, the full-speed sprint keyframe, and the gazing at the sun keyframe. In all keyframes, there are no changes to Kuafu's appearance, clothing, wasteland environment, or top lighting tone.
[0071] S09, Automatic Naming / Saving The system reads the current project name, shot number, and keyframe number and performs an automatic naming operation, with the naming format set to "project name_shot_shot number_frame_keyframe number.png". In this embodiment, four keyframe files are output with the names kuafu_shot_011_frame_1.png, kuafu_shot_011_frame_2.png, kuafu_shot_011_frame_3.png, and kuafu_shot_011_frame_4.png. Subsequently, all keyframe images are archived and stored in the shot material storage folder corresponding to project kuafu and scene_06, completing the automated generation process of the entire set of dynamic shot keyframes.
[0072] This embodiment is only a preferred implementation of the present invention. For other shot types such as close-up shots, fixed-position static shots, and short-duration shots, those skilled in the art can adapt the number of shot disassembly stages, the number of keyframes, and the content of prompts to achieve automatic generation of storyboards to sequential keyframes.
[0073] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.
[0074] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.
[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for automatic shot breakdown and keyframe generation, characterized in that, Includes the following steps: S1. Obtain a standardized storyboard, which includes camera movement, subject action, scene information, lighting information, and style requirements. S2. Extract features from the storyboard to obtain the camera movement features and subject dynamic features; S3. Classify the dynamic level of the shot based on the camera movement characteristics and the dynamic characteristics of the subject, determine the total number of key frames for the corresponding shot, and decompose the continuous dynamic shot into multiple sequentially connected shot stages. S4. Generate differentiated keyframe images and prompts for each shot stage; S5. Overlay global consistency constraints on all keyframe cue words for the same shot to obtain the final generated cue words for each keyframe. S6. Automatically call the image generation system to generate corresponding keyframe images in batches based on the final generation prompts for each keyframe.
2. The method for automatic shot breakdown and keyframe generation according to claim 1, characterized in that, The standardized storyboard script can be obtained in the following ways: Directly input the structured storyboard script; Input a plot summary and style description in natural language, and the large language model will automatically parse and generate a standardized storyboard script that conforms to preset field specifications.
3. The method for automatic shot breakdown and keyframe generation according to claim 1, characterized in that, The feature extraction also includes: Obtain the basic attributes and visual style characteristics of the lens; The basic attributes of the lens include lens number, lens duration, shot type, and initial camera position; The visual style features include scene environment, light and shadow direction, color tone, emotional atmosphere, and art style.
4. The method for automatic shot breakdown and keyframe generation according to claim 1, characterized in that, The classification of lens dynamics includes: Based on the combination of camera movement and subject movement, the dynamic range of the shot is divided into four levels from low to high: Low dynamic range: no camera movement and no subject movement; Medium-motion: Only camera movement exists, or only the subject moves; High dynamic range: Simultaneous camera movement and subject movement, with a moderate range of motion; Extremely high dynamic range: Simultaneous high-speed camera movement and large-scale subject movement.
5. The method for automatic shot breakdown and keyframe generation according to claim 4, characterized in that, The process of breaking down a continuous dynamic shot into multiple sequentially connected shots includes: Mapping rules between the number of keyframes and shot stages; Among them, low dynamic range shots correspond to 1 keyframe and are divided into the initial stage; The medium-motion shot corresponds to 2 keyframes, divided into two stages: the beginning and the end; The high dynamic range shot corresponds to 3 keyframes, which are divided into three stages: start, development, and end. The extremely high dynamic range shot corresponds to 4 keyframes, divided into four stages: beginning, development, climax, and ending.
6. The method for automatic shot breakdown and keyframe generation according to claim 1, characterized in that, The step of classifying the dynamic level of a shot based on camera movement characteristics and subject dynamic characteristics, determining the total number of keyframes for the corresponding shot, and breaking down a continuous dynamic shot into multiple sequentially connected shot stages also includes: Preset matching rules for typical shots and dynamic range levels: Fixed lens matching low dynamic range; Slow-motion zoom shots and fixed shots are used to match the dynamic level of the subject's movement. The tracking camera is matched with a high dynamic range; High-speed action shots, surround shots, and long takes are matched with extremely high dynamic range.
7. The method for automatic shot breakdown and keyframe generation according to claim 1, characterized in that, The generation of differentiated keyframe images and prompts for each shot stage includes: Each keyframe's image generation prompts include a description of the positive shot, camera language, shot composition, lighting settings, subject's action status, art style, color tone, dynamic effects, and consistency constraint indicators. Keyframe cue words in adjacent stages retain the basic constraints of the subject and scene, while also marking the changes in camera movement trajectory and subject action in this stage relative to the previous stage.
8. The method for automatic shot breakdown and keyframe generation according to claim 1, characterized in that, include: The global consistency constraint adopts a global prompt word constraint and reference graph anchoring mechanism; The global cue word constraints include: consistent main appearance, consistent character clothing, consistent core product form, consistent scene environment, consistent lighting direction, consistent color style, and consistent camera logic.
9. The method for automatic shot breakdown and keyframe generation according to claim 8, characterized in that, include: The reference image anchoring mechanism is as follows: the stage frame with the most complete semantics and the most representative image in the same shot is selected as the benchmark reference frame. When generating subsequent keyframes, the benchmark reference frame is used as the image reference input to anchor the subject image and scene style features.
10. A system for automatic shot breakdown and keyframe generation, characterized in that, include: The module includes a storyboard acquisition module, a feature extraction module, a shot breakdown module, a prompt word generation module, a final prompt word generation module, and a keyframe image generation module. The storyboard acquisition module is used to acquire standardized storyboards, which include camera movement methods, main body actions, scene information, lighting information and style requirements. The feature extraction module is used to extract features from the storyboard script through an artificial intelligence semantic parsing model to obtain the camera movement features and subject dynamic features. The lens decomposition module is used to classify the dynamic level of the lens according to the camera movement characteristics and the dynamic characteristics of the subject, adaptively determine the total number of keyframes of the corresponding lens, and decompose the continuous dynamic lens into multiple sequentially connected lens stages. The prompt word generation module is used to generate different keyframe images and prompt words for each shot stage; The final prompt word generation module is used to overlay global consistency constraints on all keyframe prompt words of the same shot to obtain the final generated prompt words for each keyframe. The keyframe image generation module is used to automatically call the image generation system and generate corresponding keyframe images in batches based on the final generation prompts for each keyframe.
