A video data split-screen method and system for smart video generation

Abstract

The application discloses a video data lensing method and system for intelligent video generation, and first builds a multi-modal film training material library, obtains three layers of semantic tags of narrative characters, dialogue voice and shot scheduling through hierarchical analysis of script texts; generates a face track in a single shot through bidirectional offline tracking, constructs a four-layer cross-shot gate based on character semantics to complete track fusion and generate a global character identity label; automatically allocates voice to corresponding characters based on three layers of constraints and generates lip reconstruction parameters; integrates four types of multi-modal data to construct four-modal training samples, performs two-stage fine-tuning on a large language model, and introduces a multi-modal joint loss of faces, voice and lips in the second stage; analyzes the preposed semantic constraints after a user inputs a creative script, inputs a special lensing model to generate a structured lensing, secondarily checks a global face identity, generates lip parameters in batches, and outputs an integrated multi-modal lensing data package. The application greatly reduces the AI video post-production manual correction cost.

Description

Technical Field

[0001] This invention relates to the field of computer vision processing technology, and more specifically to a video data storyboard method and system for intelligent video generation. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, intelligent video automatic generation has become a research hotspot in the field of content creation. Storyboard design, as a core element of video production, directly determines the efficiency and quality of AI video generation through its level of automation and intelligence. However, current mainstream technical solutions still have significant bottlenecks, mainly in the following two aspects:

[0003] I. Pure text-driven AI storyboard generation technology suffers from a lack of multimodal information:

[0004] Existing technologies (such as CN118660115A) primarily rely on Large Language Models (LLMs) to perform semantic understanding of text scripts, thereby generating video storyboards. The core limitation of these methods lies in their "pure text" nature, completely ignoring the constraints of visual and audio information.

[0005] 1. Lack of consistency in character identity: The model generates storyboards based solely on text descriptions, failing to establish a unified global identity (GID) for the same character across multiple storyboard clips. This results in the same character potentially being identified as different people in different shots during subsequent video rendering, leading to "character face-changing" or "ID drift" phenomena, severely disrupting narrative coherence.

[0006] 2. Lack of audio-visual synchronization mechanism: This type of method lacks the ability to separate speakers and bind voice to roles. In multi-person dialogue or cross-camera conversations, voice segments cannot be automatically assigned to the correct roles, which can easily lead to serious misalignment between lips and dialogue content, resulting in extremely high costs for manual correction in post-production.

[0007] 3. Limited Model Training Dimensions: Model fine-tuning only utilizes a textual semantic loss function, failing to incorporate multimodal supervision signals such as facial trajectories and lip movements. Consequently, the generated storyboards merely satisfy narrative logic and cannot be directly used to guide lip-synchronized video generation, resulting in a severe data disconnect with downstream rendering processes.

[0008] II. There are gaps in the post-processing technology for simultaneous face and lip processing:

[0009] Another type of technology (such as CN122244928A) focuses on post-processing of the generated video footage to achieve cross-camera face ID unification and synchronized lip restoration. This type of solution is a post-processing method and has inherent limitations:

[0010] 1. Inability to plan ahead: This technology can only process completed videos and cannot intervene in the storyboard design stage. The storyboard planning cannot constrain the logic of characters' appearances and the spatial and temporal relationships of shots in advance, resulting in identity conflicts being discovered only after the final cut. The difficulty and cost of correction are far higher than that of early planning.

[0011] 2. Limited matching accuracy: Cross-camera face matching relies solely on visual features (such as facial appearance similarity) and lacks guidance from script text information. This makes it easy for different characters with similar visual features to be incorrectly merged in multi-person scenes, and it is impossible to use the character settings in the script to eliminate such errors in advance.

[0012] 3. Fragmented Technology Chain: Storyboard generation, face tracking, and lip reconstruction exist as independent modules with incompatible data formats and disconnected processing flows. Storyboard scripts cannot directly carry key information such as face ID, voice assignment, and lip parameters, preventing the AI ​​video generation process from forming an efficient closed loop.

[0013] In summary, the current field of intelligent video generation generally faces the following core challenges:

[0014] 1. Data silos: Multimodal data such as storyboards, faces, voice, and lips are isolated from each other and cannot work together.

[0015] 2. Disjointed process: The process of initial storyboard design and post-production audio-visual synchronization restoration is disconnected, lacking a unified planning and constraint mechanism.

[0016] 3. High labor costs: In multi-scene, multi-character scenarios, issues such as character identity shifting, voice mismatch, and lip-syncing rely heavily on manual proofreading and correction, which severely restricts the efficiency of AI video industrial production. Summary of the Invention

[0017] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a video data storyboard method and system for intelligent video generation that can integrate multimodal information in the storyboard planning stage, achieve unified character identity, accurate voice binding, and pre-generation of lip parameters, so as to open up the entire AI video generation chain and improve the level of automation and the quality of the final film.

[0018] Regarding video data storyboarding methods, in order to achieve the above objectives, the present invention adopts the following solution: a video data storyboarding method for intelligent video generation, characterized by comprising the following steps:

[0019] S10. Collect a large number of annotated film and television sample videos, build a multimodal training material library, store the original complete text corresponding to each sample in the training material library, manually annotate the storyboard sequence, detect key point data of face frame by frame, separate the completed single-person voice segments, and annotate the lip mask information of the storyboard.

[0020] S20. For each sample video, perform layered semantic analysis of the text and output three-layer structured semantic tags. The three-layer structured semantic tags are narrative character semantic tags, dialogue voice semantic tags, and camera movement semantic tags.

[0021] S30. Based on the original video, perform shot segmentation operation to obtain the original shot fragments, perform face detection and key point localization operation frame by frame, and complete bidirectional offline face association operation within a single shot to generate face trajectory within the shot.

[0022] S40, combined with the narrative character semantic tags output by S20, constructs a semantically enhanced cross-camera gating system, performs quality control operations on all facial trajectories, extracts trajectory-level representation information, completes fusion calculation after multi-layer gating filtering to generate a global identity identifier, establishes a mapping correspondence between facial trajectories and global identity identifiers, and generates a global facial trajectory table.

[0023] S50. Based on the semantic tags of dialogue speech, speaker separation operation is completed. Under the constraints of three types of rules, namely visibility constraint, consistency constraint and conflict resolution constraint, speech segments are bound to the corresponding storyboard and the corresponding global identity identifier. At the same time, based on the face trajectory of the global identity identifier, lip reconstruction conditions are generated.

[0024] S60. Bind the three-layer text semantic tags, storyboard spatiotemporal metadata, global face trajectory table, and global voice identity identifier to the mapping relationship, and integrate the lip reconstruction parameters to construct a four-modal matching sample of text, storyboard, face, and voice, and store it in the multimodal training dataset.

[0025] S70. Based on the multimodal training dataset, the first stage of fine-tuning is performed on the basic large language model. The first stage of fine-tuning only uses the semantic matching loss of the text storyboard, so that the model has the ability to generate basic storyboard scripts based on the original text.

[0026] S80, integrates global face identifier consistency loss, voice role binding loss, and lip timing alignment loss to build a multimodal total loss function, and performs the second-stage multimodal fine-tuning operation on the large language model that has completed the first stage of fine-tuning to obtain a large model dedicated to semantic-aware storyboard generation.

[0027] S90. Receive the user's original creative text input and perform a process that is completely consistent with S20 to conduct hierarchical semantic analysis of the input text to obtain the pre-semantic constraint tags corresponding to the storyboard to be generated.

[0028] S100: Input the original creative script and the pre-existing semantic constraint labels into a dedicated large model for generating semantically perceptive storyboards, and generate a structured storyboard sequence carrying complete spatiotemporal metadata, global character placement identifiers, and dialogue speech intervals.

[0029] S110. Call the global face identifier alignment process, rely on the spatiotemporal metadata of the storyboard, re-verify and optimize the semantic constraint labels of the pre-narrative characters, optimize the face trajectory fusion correspondence within the storyboard, and update the global identity identifier mapping table.

[0030] S120. Based on the updated global identity mapping table, complete the internal speech segment allocation operation of the storyboard and generate lip reconstruction parameters in batches.

[0031] S130 outputs an integrated multimodal structured storyboard data package, which includes a text hierarchical parsing report, an AI-generated structured storyboard script, a global face trajectory table, a global identifier three-element binding mapping for voice storyboards, and independent lip reconstruction condition parameters for each storyboard.

[0032] As another improvement to the video data storyboard method for intelligent video generation of the present invention, the text hierarchical semantic parsing in step S20 specifically includes the following sub-steps:

[0033] S201. Narrative Semantic Layer Analysis: Extract all characters appearing in the text, the time interval of each character's appearance, actions, and scene information, generate narrative character semantic tags, and record the unique text identifier of each character and the start and end range of their appearance.

[0034] S202, Dialogue Speech Layer Parsing: Segment the characters' dialogue within the text, distinguish the dialogue segments corresponding to different speakers, estimate the video frame duration range matched by each dialogue segment, and generate dialogue speech semantic tags;

[0035] S203, Shot Scheduling Layer Analysis: Identify keywords describing shots within the text, extract shot type, camera movement, hard cuts, soft cuts, transitions, and composition information, and generate shot scheduling semantic tags.

[0036] As another improvement to the video data segmentation method for intelligent video generation in this invention, the semantically enhanced cross-shot gating described in S40 includes four layers of verification gating, namely, text role semantic gating, temporal gating, spatial displacement gating, and intersection-union gating. If any layer of gating fails verification, the corresponding face trajectory candidate merging pair is prohibited from performing fusion operation.

[0037] If the candidate merging operation corresponds to two independent characters with no identity association in the semantic tags of the text narrative, the merging operation is directly determined to be invalid.

[0038] If there is a temporal overlap between two face trajectory frame intervals, the timing gating operation rule prohibits merging operations.

[0039] The spatial displacement gating operation rule only applies to soft or hard cut sequences that connect before and after the previous and subsequent sequences. It calculates the Euclidean distance between the normalized center points of the faces at the beginning and end of the two trajectories. If the calculation result exceeds the preset threshold, the merging operation is prohibited.

[0040] The intersection-union (IUU) gating operation rule only applies to adjacent shots during soft cut transitions. It compares the IUU values ​​of face bounding boxes in adjacent frames, and if the value is lower than a preset threshold, the merging operation is prohibited.

[0041] As another improvement to the video data segmentation method for intelligent video generation in this invention, the single-shot bidirectional offline face association described in S30 includes the following operations: Kalman filtering motion prediction is used to generate candidate face boxes; a cost matrix is ​​constructed containing four factors: intersection-over-union ratio, center point distance, scale penalty, and appearance similarity; and inter-frame matching is performed using the Hungarian algorithm. Forward and reverse temporal sequence associations are performed separately, the consistency of the bidirectional association results is compared, and fusion verification is used to eliminate the identifier drift problem caused by unidirectional tracking.

[0042] As another improvement to the video data segmentation method for intelligent video generation in this invention, the face trajectory quality control in S40 includes the following operations: Dynamically configuring an effective frame count threshold based on the video's frame rate per second, and eliminating short trajectories with fewer than the threshold number of consecutive effective face frames. For each trajectory's face samples, a quality score is calculated based on five dimensions: sharpness, face bounding box area, keypoint validity, pose deviation, and occlusion degree. Multiple high-quality face samples ranking high within the trajectory are selected, and the optimal single-frame feature or sample weighted mean vector is chosen as the trajectory-level representation information.

[0043] As another improvement to the video data segmentation method for intelligent video generation of the present invention, the expression for the total loss function of the second-stage multimodal fine-tuning in S80 is:

[0044] Mtotal=αMsem+βMface+γMvoice+δMlip;

[0045] Where Msem is the semantic matching loss for the storyboard; Mface is the GID consistency loss for characters across shots in the storyboard, constraining the identity consistency of the same character across different shots; Mvoice is the voice character binding loss; Mlip is the alignment loss between voice timing and the lip-out interval in the storyboard; α, β, and γ are weight adjustment coefficients. As another improvement to the video data storyboard method for intelligent video generation in this invention, the voice segment allocation constraint in S50 includes three types of constraint rules:

[0046] Visibility constraint: The allocation operation can only be performed if the proportion of visible frames of the target's global identifier within the frame interval corresponding to the speech reaches more than 80%, and the average face quality score meets the standard.

[0047] Consistency constraint: All speech segments corresponding to the same speaker are uniformly bound to a single global identity identifier and are not allowed to be assigned across multiple characters;

[0048] Conflict resolution constraints: When multiple global identifiers simultaneously meet the allocation conditions, a unique target character is selected based on the priority of three indicators: face trajectory coverage in the storyboard, average face quality, and total on-screen time in the entire video.

[0049] As another improvement to the video data storyboard method for intelligent video generation of the present invention, the lip reconstruction condition in S50 includes three types of parameters:

[0050] Audio features: Mel spectrum or deep speech time-frequency features extracted from spoken dialogue;

[0051] Lip region mask: A binary mask is generated by extracting the lip contour based on the 68 key points of the face and expanding it outward by 10% to 20%.

[0052] Identity constraint features: These correspond to the global identifier of the optimal face frame feature vector or the high-quality sample mean feature vector, and are used to suppress the problem of identity drift during the reconstruction process.

[0053] In terms of the system, to achieve the above objectives, the present invention adopts the following solution: a smart video generation storyboard system, characterized in that it is used to execute the method described in any one of the above embodiments, and the system includes the following modules:

[0054] The multimodal material storage module is used to store the film and television training material library, the multimodal training dataset, and the final output integrated structured storyboard data package;

[0055] The copywriting layered semantic parsing module connects with the multimodal material storage module to perform three-layer semantic parsing operations on the original copywriting, including narrative, dialogue, and camera movement, and outputs structured semantic tags.

[0056] The in-camera face trajectory generation module is used to perform face detection frame by frame on the video segment, perform bidirectional offline tracking calculations, and generate the original face trajectory for a single shot.

[0057] The semantically enhanced global face identifier fusion module has a built-in four-layer cross-camera gating verification unit, face quality control unit, trajectory representation extraction unit, and disjoint set fusion unit. It generates a unified global identity identifier and a global face trajectory table based on text semantic tags.

[0058] The voice role binding and lip parameter generation module is used for speaker separation calculation, voice segment allocation calculation, and batch generation of lip reconstruction condition parameters for each role;

[0059] The four-modal sample construction module integrates text semantic tags, scene data, face trajectory table, and voice binding correspondence into four-modal matching training samples and stores them in the database;

[0060] The two-stage large model fine-tuning module sequentially performs the first stage of text semantic fine-tuning and the second stage of multimodal audiovisual supervised fine-tuning, outputting a semantically aware large model for storyboard generation.

[0061] The intelligent storyboard generation module receives user input of creative text and pre-defined semantic constraint tags, and calls a dedicated large model to generate a structured storyboard sequence carrying metadata.

[0062] The storyboard face identification secondary verification module, based on the spatiotemporal information of the storyboard, re-verifies the global identity identification mapping relationship and updates the trajectory table by re-verifying the semantic tags of the text characters;

[0063] The integrated structured output module packages the text analysis report, storyboard script, face trajectory table, voice binding mapping, lip reconstruction parameters, and outputs a complete storyboard data package.

[0064] As another improvement to the intelligent video generation storyboard system of this invention, the four-layer cross-camera gating verification unit of the semantic enhancement face global identification fusion module integrates text role semantic gating, temporal gating, spatial displacement gating, and intersection-union gating. It prioritizes the execution of text role semantic gating filtering operations to pre-screen out invalid merging pairs that are visually similar but whose texts belong to different people.

[0065] In summary, the advantages of this invention over the prior art are:

[0066] I. This invention pioneers a semantic gating system for script characters as the first layer of filtering. It uses script text character tags to pre-screen visually similar but independent face merging pairs. Combined with three layers of visual constraints (temporal, spatial, and IoU), and four layers of gating for verification, it solves the core pain point of character identity drift under multi-scene editing. It eliminates one-way tracking ID switching through bidirectional offline face association and removes invalid short trajectories through trajectory quality control, further improving the global GID uniformity accuracy.

[0067] Second, this invention relies on dialogue semantic tags and three-layer voice allocation constraints to automatically assign dialogue to the corresponding on-screen characters in scenarios with multiple people in the same shot and cross-camera switching, without the need for manual dragging and correction; at the same time, it generates standardized voice, scene, and GID three-element mapping data, which can be directly read by downstream rendering models, completely solving the problem of lip synchronization disorder.

[0068] Third, this invention adopts a two-stage fine-tuning architecture. The first stage learns the text narrative logic, and the second stage introduces face consistency, voice binding, and lip-temporal multimodal loss supervision. When the model generates storyboards, it automatically takes into account the continuity of character appearances and dialogue matching relationships. The output storyboards come with complete audiovisual metadata and do not require secondary modification in the later stage.

[0069] Fourth, this invention constructs a complete closed loop of material library construction → four-modal training → AI-generated storyboard → secondary face verification → voice allocation → lip parameter output. One multimodal data package is compatible with the entire business chain of storyboard preview, video rendering, digital lip generation, and post-production editing, saving the deployment, data conversion, and manual correction costs of three independent systems. The efficiency of batch generation of long videos and short dramas is improved by more than 60%.

[0070] 5. The output data package contains complete intermediate diagnostic data: text layering parsing results, face removal trajectory logs, gating rejection merging records, and voice allocation conflict handling records, which facilitates fault diagnosis, manual fine-tuning, and model iteration optimization, and adapts to the operation and maintenance needs of large-scale AI video production platforms. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of the complete process of the overall method of the present invention.

[0072] Figure 2 This is a diagram showing the module architecture connection of the intelligent video generation storyboard system of the present invention. Detailed Implementation

[0073] The above-mentioned and other technical features and advantages of the present invention will be described in more detail below with reference to the accompanying drawings.

[0074] The technical solution of the present invention will be clearly and completely described below with reference to complete embodiments. Those skilled in the art can reproduce the entire technical process based on these embodiments.

[0075] Detailed Implementation Methods (3 Specific Examples for Live Streaming E-commerce / Short Video Promotion Scenarios)

[0076] The following three embodiments are all geared towards live streaming segments, e-commerce short videos, and product promotion short videos, respectively corresponding to three mainstream commercial scenarios: "single-host routine live streaming segment storyboard generation," "dual-host interactive e-commerce short video production," and "multi-person mixed-edit product promotion short videos." They fully cover all the technical features of the claims and address the pain points of live streaming e-commerce businesses, such as host ID breakage across shots, misaligned lip-sync dialogue, confusion of characters in multi-person mixed-edit videos, high labor costs for batch slicing, and audio-visual asynchrony in product explanations.

[0077] Example 1: Method for Automatic Generation of Storyboards from Single-Anchor Live Broadcast Segments

[0078] S10. Build a multimodal training material library specifically for live-streaming e-commerce:

[0079] We collected 15,000 annotated live stream clips, and each clip stored the following: complete live stream script, manually edited product-selling sequence, 68 facial key points frame by frame, separate audio segments of the single anchor, and lip annotation mask for the anchor; additional product explanation interval labels were added to the annotation fields to distinguish different product introduction time periods.

[0080] S20. Three-layer semantic analysis of sales copywriting:

[0081] S201, Narrative Semantic Layer: Generates only a unique anchor text identifier, marking the anchor's on-screen appearance range throughout the entire live broadcast and the time periods of each product explanation screen;

[0082] S202, Dialogue Voice Layer: Decompose the voice-over scripts for each product, estimate the video frame range that each product-selling voice segment matches, and generate dialogue semantic tags;

[0083] S203, Camera Steering Layer: Identifies live stream camera keywords such as "hard cut to product close-up," "medium shot of people," and "overhead shot of product," and marks all hard cut transitions and camera shot types.

[0084] S30, Single-lens two-way offline face association:

[0085] The live stream footage is segmented into shots, and face detection is performed frame by frame. Kalman filtering is used to predict candidate boxes for the anchor's face, with the following cost matrix weights: IoU 0.4, center point distance 0.3, scale penalty 0.2, and appearance similarity 0.1. Bidirectional tracking is performed in both forward and reverse directions to filter out unidirectional tracking ID drift caused by raising hands to cover or looking down at products during the live stream, generating the complete face trajectory of the anchor within a single shot.

[0086] S40, semantically enhanced four-layer cross-camera gating + trajectory quality control:

[0087] Quality control: The effective frame threshold for 25fps video is 7 frames, and short local trajectories of the anchor that flash by are removed; the face quality scoring dimensions are simplified to three items: clarity, degree of occlusion, and face area. The top 5 high-quality faces are selected for each trajectory, and the best single frame is selected as the trajectory-level representation.

[0088] Four-layer gating adapted for single-person live streaming: Semantic gating of text characters has only one anchor ID and no cross-character merging interception; temporal gating intercepts time overlap trajectories; spatial displacement gating and IoU gating only take effect when switching between product close-up shots, filtering invalid merging pairs with excessively large differences in face position after the shot jump; all compliant trajectories are fused through disjoint-set data structure, assigned a unique global GID, and a global face trajectory table is generated, with no anchor frame filling [0,0,0,0] zero coordinates.

[0089] S50, three constraints of voice assignment, and parameters for reconstructing lips for product promotion:

[0090] S51. Voice Assignment: Visibility constraint requires that the anchor's face be visible for ≥80% before assigning the corresponding product for voice broadcast; consistency constraint binds all voice explanations of the same product to the anchor's unique GID; conflict resolution requirement in single-person scenarios.

[0091] S52. Lip Reconstruction: Extract the Mel spectrum of the product-selling voice as audio features; extract the lip contour based on 68 key points and expand it by 12% to generate a binary mask; use the anchor's optimal facial features as identity constraints to avoid facial deformation during product close-ups.

[0092] S6. Construct training samples for the four modalities of live-streaming e-commerce:

[0093] By integrating three layers of semantic tags for product sales, spatiotemporal metadata of product shots, global facial trajectory of the anchor, voice-GID binding relationship, and lip reconstruction parameters, a million-level four-modal training dataset for product sales is generated.

[0094] S70 and S80 dual-stage model fine-tuning

[0095] The basic 7B model is trained using only text loss in the first stage; the second stage adjusts the loss weights to adapt to the e-commerce scenario, prioritizing lip-sync alignment.

[0096] M total =0.35M sem +0.25M face +0.2M voice +0.2M lip。

[0097] After training, a large semantic-aware storyboard model for live streaming segments is obtained.

[0098] S90–S130, Live Script Input Fully Automatic Output of Storyboard Data Package

[0099] The system automatically performs layered semantic parsing to generate pre-constraint tags by inputting the complete script for the entire live stream; it also generates structured storyboards with the anchor's GID, product explanation frame intervals, and shot types by inputting a dedicated model; a secondary verification module unifies the global ID of all live stream anchors and generates lip reconstruction parameters for each product explanation segment in batches; finally, it outputs an integrated data package: a product script analysis report, product segmentation storyboard scripts, anchor's global face trajectory, voice binding mapping, and independent lip mask parameters for each storyboard. This allows for the direct batch rendering of multiple product promotional short videos without the need for manual correction of anchor identity and lip movements.

[0100] For 4 hours of live stream footage, traditional manual editing requires 4 hours to proofread the anchor ID and correct lip-syncing errors; this solution automatically generates a storyboard data package in just 12 minutes, completely eliminating the problem of anchor ID breakage across shots, achieving 100% audio-visual synchronization accuracy for product explanations, and reducing manual editing costs by 90%.

[0101] Example 2: Generation of Storyboards for Interactive E-commerce Short Videos by Two Hosts

[0102] S10, Building a dual-person product labeling material library:

[0103] We collected 8,000 sets of short videos featuring two people interacting and selling products. We stored the independent scripts, storyboard sequences, 68 facial key points for each person, separate voice recordings, and lip masks for each person. We also labeled and distinguished the two narrative roles: Host 1 and Host 2.

[0104] S20, Three-layer semantic parsing:

[0105] S201, Narrative Semantic Layer: Generate independent text identifiers for anchor A and anchor B respectively, and record their on-screen appearances, solo explanations, and frames where the two are in the same frame;

[0106] S202, Dialogue Voice Layer: Distinguish between the sales scripts of the two anchors and estimate the duration of each voice segment matched with the video;

[0107] S203, Shot Scheduling Layer: Marks medium shot of two people, close-up of one person, and hard cut transition of product.

[0108] S30 bidirectional offline face tracking:

[0109] 30fps video, Kalman filter adapted to the interaction of two people walking back and forth; cost matrix increases appearance similarity weight to 0.25, distinguishing two people with similar facial features; positive + negative bidirectional correlation eliminates tracking drift caused by a single person looking down to pick up a product, and generates two independent facial trajectories for anchor A and anchor B in the shot.

[0110] S40, four-layer semantic enhancement gating core innovation implemented

[0111] Face quality control: 30fps effective frame threshold 9 frames, five-dimensional complete weighted scoring, top 10 face samples for each trajectory, and use the feature mean as the trajectory representation;

[0112] The four-layer gating system verifies data sequentially according to priority:

[0113] ① Semantic gating of copywriting characters (core innovation): The two trajectories correspond to two independent narrative characters, anchor A and anchor B, respectively. They are directly intercepted and merged, completely solving the mis-merging caused by the similarity of the two characters' facial features;

[0114] ② Timing gating: Filters out invalid merging of two overlapping trajectories at the same time;

[0115] ③ Spatial displacement gating: Single-person close-up switching camera verifies the distance threshold of the face normalization center point to 0.35;

[0116] ④IoU Gating: Verify face intersection and union ratio using soft-slice transition frames for two people in the same frame;

[0117] The compliant trajectory is merged and the data set is integrated, and two independent global face trajectory tables, GID1 (anchor A) and GID2 (anchor B), are assigned.

[0118] S50, speech conflict resolution + dual-person lip parameter generation:

[0119] Voice allocation employs a complete three-layer constraint: visibility ≥80%, and voices of the same speaker are bound to a single GID; when two people are on screen together and explaining at the same time, conflict resolution rules are activated, and the corresponding spoken voices are allocated according to the priority of face coverage > face quality > on-screen time; independent lip masks and identity constraint features are generated for anchor A and anchor B respectively, without interference between them.

[0120] S60, Two-person live-streaming four-modal sample entry:

[0121] Training samples are constructed by integrating the semantics of two characters, the spatiotemporal context of the shot, the dual GID trajectories, the dual voice mapping, and two sets of lip parameters.

[0122] S70-S80, multimodal fine-tuning weights for adapting to two-person interaction scenarios:

[0123] Increase the weight of face consistency loss to prevent confusion between two individuals:

[0124] M total =0.3M sem +0.35M face +0.2M voice +0.15M lip。

[0125] S90–S130, Input the script for a two-person live-streaming sales event to generate storyboards:

[0126] The system analyzes the semantic constraints of the two hosts' on-camera appearances by inputting the script for a live-streaming sales event. The large model generates a structured storyboard with GID1 / GID2 placeholders and their respective speaking segments. The secondary verification module unifies the global identifiers of the two hosts across shots and generates two sets of independent lip reconstruction parameters in batches. The output data package can render the host's solo product recommendation segment and the two-person interactive segment separately, without any mismatch of voice characters.

[0127] Traditional solutions rely solely on visual matching, resulting in a 41% false merging rate of stranger faces in dual-person product promotion short videos. This solution, however, utilizes pre-gating of text and character semantics to reduce the false merging rate to 1.2%, eliminating the need for manual differentiation between the two hosts and increasing the efficiency of batch production of dual-person product promotion short videos by 70%.

[0128] Example 3: Generation of Storyboards for Multi-User Mashup Product Promotion Short Videos

[0129] S10, Multiplayer Mashup Training Material Library:

[0130] We collected 6,000 short videos featuring influencers creating montages and promoting products. Each video was labeled with textual characters of 3-5 independent influencers, cross-camera montage sequences, key facial features of each influencer, separated audio, and lip masking.

[0131] S20, Three-layer hierarchical semantic parsing:

[0132] S201, Narrative Semantic Layer: Assign a unique text ID to each influencer, marking all their on-screen moments and product explanation segments in the montage video;

[0133] S202, Dialogue Voice Layer: Break down the product-selling voiceover for each influencer and divide the video frame duration range corresponding to each voice segment;

[0134] S203 Shot Scheduling Layer: Identifies shot keywords such as hard cuts across materials, product close-ups, and solo close-ups of celebrities, and records all transition types and shot composition;

[0135] S30, lightweight bidirectional face tracking:

[0136] The 24fps footage uses lightweight tracking logic, performs full forward temporal correlation, and only adds local reverse verification for occluded segments to reduce computational power; Kalman filtering simplifies the motion model, the cost matrix retains three factors: IoU, center point distance, and appearance similarity, and removes scale penalty terms; frame matching is completed through the Hungarian algorithm, and bidirectional consistent frames are fused to eliminate tracking ID drift caused by cross-film jumps, outputting independent face trajectories for each scene.

[0137] S40, four-layer gating adapted for multi-person video editing scenarios:

[0138] S401, Face Trajectory Quality Control: 24fps dynamic effective frame threshold of 5 frames, filtering out fleeting local face short trajectories; face quality retains only the two-dimensional weighted score of clarity and occlusion degree; the top 3 high-quality faces are selected for each trajectory, and the best single-frame feature is selected as the trajectory-level representation, eliminating the need for mean vector operation and reducing computational overhead.

[0139] S402, a four-layer gating system that performs priority-based verification, relies on semantic gating of copywriting roles to intercept cross-influencer error merging:

[0140] ① Semantic gating for copywriting roles: If two tracks belong to different influencer text IDs, merging is directly prohibited;

[0141] ② Timing gating: If trajectory frame intervals overlap, the merged pairs are eliminated;

[0142] ③ Spatial displacement gating: Only enabled for shots connecting different materials, with a normalized center point distance threshold of 0.32;

[0143] ④IoU Gating: Only enabled for soft transition shots; adjacent frames with an IoU below 0.2 are prohibited from being merged.

[0144] All compliant trajectories are merged separately through a disjoint-set data structure, and each influencer is assigned an independent global GID. Multiple independent global face trajectory tables are generated, and frames without faces are uniformly filled with zero coordinates [0,0,0,0].

[0145] S50, a multi-person voice conflict resolution mechanism:

[0146] S501, three-layer constraints for voice allocation are fully enabled: visibility constraint requires that the target GID face is visible in ≥80% of the frames before voice can be allocated; consistency constraint requires that all sales voices of the same influencer are only bound to their own GID; conflict resolution selects only the influencer who explains the content based on the priority of "face coverage > average face quality > total on-screen time".

[0147] S502, Lightweight configuration for lip reconstruction: Lightweight Mel spectrum is used for audio features; the lip contour is generated by expanding outward by 10% based on 68 key points to create a binary mask; the identity constraint adopts the optimal benchmark facial feature vector of each expert, eliminating the need for real-time batch calculation of sample mean.

[0148] S60, lightweight four-modal training samples:

[0149] It integrates three layers of influencer semantic tags, cross-material storyboard spatiotemporal metadata, multiple sets of GID global trajectory tables, voice-influencer binding mapping, and independent lip parameters for each influencer, simplifies redundant multi-role fields, compresses sample storage volume, and stores it in a lightweight multimodal training dataset.

[0150] S70-S80, Lightweight LoRA Fine-Tuning:

[0151] A lightweight, large-scale 3B model was selected, and LoRA fine-tuning was used to reduce computational power consumption.

[0152] S70, First-stage fine-tuning: Using only copywriting, storyboarding, and semantic loss M sem After 30 rounds of training, the basic ability to generate product-related storyboards was achieved.

[0153] S8 Phase 2 multimodal joint fine-tuning adapts to the lip-sync priority requirements of short videos in the information stream. Total loss formula:

[0154] M total =0.4M sem +0.25M face +0.15M voice +0.25Mlip。

[0155] The model incorporates new multi-person GID consistency loss, speech matching loss, and lip timing alignment loss for synchronous supervision. After 45 training rounds, it outputs a large semantic-aware storyboard model specifically for multi-person montage editing.

[0156] S90. Input the mixed-cut product description and generate pre-semantic constraint tags:

[0157] The operation input integrates the complete scripts of multi-influencer live-streaming sales, and the system reuses the full set of layered analysis logic of S20: extract the narrative character ID of each influencer, their respective voice-over dialogue range, and cross-material shot scheduling information to generate a complete set of three-layer semantic constraint tags as the prior conditions for storyboard generation.

[0158] S100. Generate a structured storyboard sequence carrying up to [number] person metadata:

[0159] The original montage script and the three layers of semantic constraint tags are input into the lightweight dedicated storyboard model, and the model automatically outputs a standardized storyboard sequence. Each storyboard entry carries: the start and end frames of the storyboard, the shot size / transition, the corresponding influencer's global GID, the product's voice-over frame range matched by the storyboard, and the product image marker.

[0160] S110, optimization of secondary verification of global identifier for top performers:

[0161] The process involves calling the global face identifier alignment process, reading all spatiotemporal metadata of the storyboard and the pre-narrative semantic tags of each influencer, and re-performing the four-layer gating review of the face trajectory of the storyboard when jumping between different materials; correcting the problem of incorrect fusion of influencer trajectory caused by cross-material splicing, and updating the GID mapping relationship and global face trajectory table corresponding to all influencers to ensure that the identity of each influencer is unique and not confused throughout the film.

[0162] S120, Batch allocation of voice recordings, generation of independent lip reconstruction parameters for each expert:

[0163] Based on the updated multi-GID mapping table, all voice segments within the storyboard are traversed, and each segment of product promotion is automatically bound to the corresponding influencer's global identifier according to the three-layer allocation constraints. For each set of influencer GIDs, exclusive audio time-frequency features, lip binary masks, and identity constraint features are generated in batches and bound and stored one by one according to the storyboard number.

[0164] S130, outputs lightweight integrated multimodal storyboard data package:

[0165] The system packages five types of standardized files to output lightweight data packages:

[0166] ① Copywriting Layered Analysis Report: Records the semantic extraction results of each influencer's role, voice-over, and camera shots;

[0167] ② AI-structured mixed editing and storyboard script: labeling the influencer GID and product explanation time period for each shot;

[0168] ③ Multi-person global face trajectory table: Each GID has an independent frame face coordinate matrix, with frames without people set to zero;

[0169] ④ Voice-Storyboard-Expert GID Tripartite Binding Mapping Table;

[0170] ⑤ Each scene has its own independent set of lip reconstruction condition parameters, which are archived and stored by the experts.

[0171] The data package is in lightweight JSON format and can be directly integrated with e-commerce information flow short video rendering tools to batch render multiple product promotion short videos with one click.

[0172] Example 4

[0173] A smart video storyboard generation system, comprising the following modules:

[0174] The multimodal material storage module is used to store the film and television training material library, the multimodal training dataset, and the final output integrated structured storyboard data package;

[0175] The copywriting layered semantic parsing module connects with the multimodal material storage module to perform three-layer semantic parsing operations on the original copywriting, including narrative, dialogue, and camera movement, and outputs structured semantic tags.

[0176] The in-camera face trajectory generation module is used to perform face detection frame by frame on the video segment, perform bidirectional offline tracking calculations, and generate the original face trajectory for a single shot.

[0177] The semantically enhanced global face identifier fusion module has a built-in four-layer cross-camera gating verification unit, face quality control unit, trajectory representation extraction unit, and disjoint set fusion unit. It generates a unified global identity identifier and a global face trajectory table based on text semantic tags.

[0178] The voice role binding and lip parameter generation module is used for speaker separation calculation, voice segment allocation calculation, and batch generation of lip reconstruction condition parameters for each role;

[0179] The four-modal sample construction module integrates text semantic tags, scene data, face trajectory table, and voice binding correspondence into four-modal matching training samples and stores them in the database;

[0180] The two-stage large model fine-tuning module sequentially performs the first stage of text semantic fine-tuning and the second stage of multimodal audiovisual supervised fine-tuning, outputting a semantically aware large model for storyboard generation.

[0181] The intelligent storyboard generation module receives user input of creative text and pre-defined semantic constraint tags, and calls a dedicated large model to generate a structured storyboard sequence carrying metadata.

[0182] The storyboard face identification secondary verification module, based on the spatiotemporal information of the storyboard, re-verifies the global identity identification mapping relationship and updates the trajectory table by re-verifying the semantic tags of the text characters;

[0183] The integrated structured output module packages the text analysis report, storyboard script, face trajectory table, voice binding mapping, lip reconstruction parameters, and outputs a complete storyboard data package.

[0184] The semantic enhancement face global identifier fusion module of this invention integrates text role semantic gating, temporal gating, spatial displacement gating, and intersection-union gating in its four-layer cross-camera gating verification unit. It prioritizes the execution of text role semantic gating filtering operations to pre-screen out invalid merging pairs that are visually similar but whose text belongs to different people.

[0185] The software execution flow of each module in this invention system is as follows:

[0186] Module 1: Multimodal Material Storage Module

[0187] 1. Two types of data are entered into the database:

[0188] (1) Training materials: 6,000 short videos of live-streaming sales with multiple human editing and annotations. Each video contains the original sales script, manual storyboard, 68 facial key points, separate single-person voice, and lip mask annotation.

[0189] (2) User-generated materials to be processed: original videos of the live broadcast replays of the three brand influencers and the integrated complete sales copy;

[0190] 2. Storage space partitioning: training material library partitioning, multimodal training dataset partitioning, and finished storyboard data package partitioning, with automatic indexing and support for quick retrieval by influencer ID and product keywords;

[0191] 3. External read / write interface, synchronously connecting to the copywriting parsing module, the four-modal sample construction module, and the integrated output module for data read / write requests.

[0192] Module 2: Layered Semantic Analysis Module for Copywriting

[0193] A data reading link is established with the storage module, and three-layer semantic parsing (weight 2 logic) is performed on the training material text and the user-input mashup script respectively:

[0194] 1. Narrative semantic layer: Assign unique text character IDs to the three influencers and extract their on-screen frames and product explanation segments throughout the video;

[0195] 2. Dialogue Voice Layer: Each influencer's unique sales-related voiceover is broken down and matched with the corresponding video length range;

[0196] Shot scheduling layer: Identifies keywords such as product close-ups, hard cuts across materials, and soft fade-in cuts, and records all shot spatiotemporal metadata;

[0197] 3. Output three-layer structured semantic labels and push them to the four-modal sample construction module (training phase) and the intelligent storyboard generation module (user reasoning phase).

[0198] Module 3: In-lens Face Trajectory Generation Module

[0199] The system reads video footage from the storage module, performs face detection and 68-point key point localization frame by frame, and uses Kalman filtering and Hungarian algorithm for forward and local reverse offline tracking. It constructs a cost matrix that includes IoU, center point distance, and appearance similarity. It filters tracking drift caused by occlusion and image truncation, outputs the original face trajectory for each independent shot, and pushes the trajectory data to the semantically enhanced face global identifier fusion module.

[0200] Module 4: Semantic Enhancement Face Global Identifier Fusion Module (Weight 10 Core Implementation)

[0201] It has four built-in sub-units: a four-layer cross-lens gating verification unit, a face quality control unit, a trajectory representation extraction unit, and a disjoint-set fusion unit.

[0202] 1. Face quality control unit: video frame rate 24fps, dynamic effective frame threshold 5 frames, remove fleeting short face trajectories; calculate face quality score by weighting clarity and occlusion degree, select the top 3 high-quality faces for each trajectory, and generate trajectory feature vector from the best single frame;

[0203] 2. Four-layer cross-camera gating verification unit (weight 10 limits execution order): Prioritize the execution of text role semantic gating, and then sequentially verify temporal gating, spatial displacement gating, and intersection-union gating;

[0204] 2.1. Semantic gating of copywriting characters: If two facial trajectories are respectively bound to the narrative semantic IDs of two different influencers, the merging is directly determined to be invalid, and incorrect matching pairs of different influencers with similar facial features are filtered out in advance;

[0205] 2.2. Timing Gating: Trajectory frames with overlapping times are prohibited from being merged;

[0206] 2.3. Spatial displacement gating: Only applies to shots that connect across different footage; interception occurs when the distance between the normalized center point of the face exceeds 0.32.

[0207] 2.4. Intersection over Union (IoU) Gating: Only soft-cut transition shots are checked; adjacent frames with an IoU below 0.2 are prohibited from merging.

[0208] 3. Disjoint-set data fusion unit: Cluster and fuse all face trajectories that pass through the four-layer gating, assign independent global GIDs to the three influencers, generate a global face trajectory table, and fill the [0,0,0,0] coordinates with zeros for no face frames;

[0209] 4. Output global GID mapping relationship and global face trajectory table, and push them to the voice role binding and lip parameter generation module and the four-modal sample construction module.

[0210] Module 5: Voice Role Binding and Lip Parameter Generation Module

[0211] 1. Speaker separation: Conv-TasNet is used to split the independent audio segments of 3 influencers promoting products;

[0212] 2. Voice allocation operation: Implements three-layer constraints: visibility, consistency, and conflict resolution; when multiple people are in the same frame, voice is allocated to the corresponding GID based on the priority of face coverage, face quality, and on-screen time.

[0213] 3. Batch generation of lip reconstruction parameters: Generate Mel spectrum audio features, lip expansion 10% binary mask, and optimal face identity constraint features for each influencer;

[0214] 4. Voice-GID binding mapping, synchronous push of the complete set of lip parameters to the four-modal sample construction module, and secondary verification module for storyboard face identification.

[0215] Module 6: Four-modal sample construction module

[0216] It receives three-layer semantic tags of the text, spatiotemporal metadata of the storyboard, global face trajectory table, voice-GID mapping, and lip reconstruction parameters, and integrates them into a single four-modal training sample; it automatically removes abnormal and dirty data, writes them in batches into the multimodal training dataset, stores them in the multimodal material storage module, and supplies them to the two-stage large model fine-tuning module.

[0217] Module 7: Two-Stage Large Model Fine-Tuning Module

[0218] Based on the storage module, read the four-modal training dataset and fine-tune the 3B lightweight e-commerce-specific large model in two steps:

[0219] 1. First-stage text semantic fine-tuning: Training the model for 30 rounds using only the copywriting-storyboard semantic loss to enable it to generate basic product-selling storyboards;

[0220] 2. Second-stage multimodal audiovisual supervision and fine-tuning, total loss:

[0221] M total =αM sem +βM face +γM voice +δM lip

[0222] The system simultaneously constrains character GID consistency, voice matching, and lip timing alignment, and trains for 45 rounds. The finely tuned semantic-aware storyboard model is then embedded into the inference server for use by the intelligent storyboard generation module.

[0223] Module 8: Intelligent Storyboard Generation Module

[0224] The system receives multi-person montage e-commerce copy uploaded from the operations side, reads the three-layer semantic constraint tags output by the copy layer semantic analysis module, calls the finely tuned dedicated large model, and automatically generates a structured storyboard sequence. Each storyboard carries the start and end frames of the shot, the scene transition, the influencer's position GID, and the product's voiceover interval. The system outputs the original storyboard sequence and pushes it to the storyboard face identification secondary verification module.

[0225] Module 9: Secondary Verification Module for Storyboard Face Identifiers

[0226] Read the AI-generated structured storyboard, storyboard spatiotemporal metadata, and text narrative character semantic tags, and call the four-layer gating logic of the semantically enhanced face global identifier fusion module again to review the fusion relationship of all face trajectories under cross-film mixed editing shots; correct the GID mismatch of influencers caused by cross-shot and cross-film splicing, update the global identity identifier mapping table and the global face trajectory table, and push the updated data to the integrated structured output module.

[0227] Module 10: Integrated Structured Output Module

[0228] Integrate all intermediate standardized data, package a lightweight, integrated storyboard data package, and write it to the finished product partition of the multimodal material storage module. The data package includes:

[0229] 1. Complete report on layered semantic analysis of copywriting;

[0230] 2. AI automatically generates multi-person montage structured storyboards;

[0231] 3. Independent global face trajectory table for each user;

[0232] 4. Voice-Scene-Expert GID Tripartite Binding Mapping Table;

[0233] 5. The complete set of parameters for lip reconstruction corresponding to each scene;

[0234] The data package supports one-click export and direct connection to short video rendering tools to batch generate short videos with consistent lip-syncing and user identity for product promotion.

[0235] The foregoing has shown and described the basic principles and main features of the present invention, as well as its advantages. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A video data storyboard method for intelligent video generation, characterized in that, Includes the following steps: S10. Collect a large number of annotated film and television sample videos, build a multimodal training material library, store the original complete text corresponding to each sample in the training material library, manually annotate the storyboard sequence, detect key point data of face frame by frame, separate the completed single-person voice segments, and annotate the lip mask information of the storyboard. S20. For each sample video, perform layered semantic analysis of the text and output three-layer structured semantic tags. The three-layer structured semantic tags are narrative character semantic tags, dialogue voice semantic tags, and camera movement semantic tags. S30. Based on the original video, perform shot segmentation operation to obtain the original shot fragments, perform face detection and key point localization operation frame by frame, and complete bidirectional offline face association operation within a single shot to generate face trajectory within the shot. S40, combined with the narrative character semantic tags output by S20, constructs a semantically enhanced cross-camera gating system, performs quality control operations on all facial trajectories, extracts trajectory-level representation information, completes fusion calculation after multi-layer gating filtering to generate a global identity identifier, establishes a mapping correspondence between facial trajectories and global identity identifiers, and generates a global facial trajectory table. S50: Based on the semantic tags of dialogue speech, speaker separation operation is completed. Under the constraints of three types of rules, namely visibility constraint, consistency constraint and conflict resolution constraint, speech segments are bound to the corresponding storyboard and the corresponding global identity identifier. At the same time, based on the face trajectory of the global identity identifier, lip reconstruction conditions are generated. S60. Bind the three-layer text semantic tags, storyboard spatiotemporal metadata, global face trajectory table, and global voice identity identifier to the mapping relationship, and integrate the lip reconstruction parameters to construct a four-modal matching sample of text, storyboard, face, and voice, and store it in the multimodal training dataset. S70. Based on the multimodal training dataset, the first stage of fine-tuning is performed on the basic large language model. The first stage of fine-tuning only uses the semantic matching loss of the text storyboard, so that the model has the ability to generate basic storyboard scripts based on the original text. S80, integrates global face identifier consistency loss, voice role binding loss, and lip timing alignment loss to build a multimodal total loss function, and performs the second-stage multimodal fine-tuning operation on the large language model that has completed the first stage of fine-tuning to obtain a large model dedicated to semantic-aware storyboard generation. S90. Receive the user's original creative text input and perform a process that is completely consistent with S20 to conduct hierarchical semantic analysis of the input text to obtain the pre-semantic constraint tags corresponding to the storyboard to be generated. S100: Input the original creative text and the pre-existing semantic constraint labels into a dedicated large model for generating semantically perceptive storyboards, and generate a structured storyboard sequence carrying complete spatiotemporal metadata, global character placement identifiers, and dialogue speech intervals. S110. Call the global face identifier alignment process, rely on the spatiotemporal metadata of the storyboard, re-verify and optimize the semantic constraint labels of the pre-narrative characters, optimize the face trajectory fusion correspondence within the storyboard, and update the global identity identifier mapping table. S120. Based on the updated global identity mapping table, complete the internal speech segment allocation operation of the storyboard and generate lip reconstruction parameters in batches. S130 outputs an integrated multimodal structured storyboard data package, which includes a text hierarchical parsing report, an AI-generated structured storyboard script, a global face trajectory table, a global identifier three-element binding mapping for voice storyboards, and independent lip reconstruction condition parameters for each storyboard.

2. The video data storyboard method for intelligent video generation according to claim 1, characterized in that, The text hierarchical semantic parsing described in step S20 specifically includes the following sub-steps: S201. Narrative Semantic Layer Analysis: Extract all characters appearing in the text, the time interval of each character's appearance, actions, and scene information, generate narrative character semantic tags, and record the unique text identifier of each character and the start and end range of their appearance. S202, Dialogue Speech Layer Parsing: Segment the characters' dialogue within the text, distinguish the dialogue segments corresponding to different speakers, estimate the video frame duration range matched by each dialogue segment, and generate dialogue speech semantic tags; S203, Shot Scheduling Layer Analysis: Identify keywords describing shots within the text, extract shot type, camera movement, hard cuts, soft cuts, transitions, and composition information, and generate shot scheduling semantic tags.

3. The video data storyboard method for intelligent video generation according to claim 1, characterized in that, The semantically enhanced cross-camera gating described in S40 includes four layers of verification gating: text role semantic gating, temporal gating, spatial displacement gating, and intersection-union gating. If any layer of gating fails verification, the corresponding face trajectory candidate merging pair is prohibited from performing fusion operation. If the candidate merging operation corresponds to two independent characters with no identity association in the semantic tags of the text narrative, the merging operation is directly determined to be invalid. If there is a temporal overlap between two face trajectory frame intervals, the timing gating operation rule prohibits merging operations. The spatial displacement gating operation rule only applies to soft or hard cut sequences that connect before and after the previous and subsequent sequences. It calculates the Euclidean distance between the normalized center points of the faces at the beginning and end of the two trajectories. If the calculation result exceeds the preset threshold, the merging operation is prohibited. The intersection-union (IUU) gating operation rule only applies to adjacent shots during soft cut transitions. It compares the IUU values ​​of face bounding boxes in adjacent frames, and if the value is lower than a preset threshold, the merging operation is prohibited.

4. The video data storyboard method for intelligent video generation according to claim 1, characterized in that, The single-lens bidirectional offline face association described in S30 includes the following operations: generating candidate face boxes using Kalman filter motion prediction, constructing a cost matrix containing four factors: intersection-over-union ratio, center point distance, scale penalty, and appearance similarity, performing inter-frame matching operations using the Hungarian algorithm, performing forward temporal sequence association and reverse temporal sequence association respectively, comparing the consistency of bidirectional association results, and fusion verification to eliminate the label drift problem caused by unidirectional tracking.

5. The video data storyboard method for intelligent video generation according to claim 1, characterized in that, The face trajectory quality control described in S40 includes the following operations: dynamically configuring an effective frame threshold based on the video frame rate per second, eliminating short trajectories with fewer than the threshold number of consecutive effective face frames, calculating a quality score for face samples within each trajectory based on five dimensions: clarity, face box area, key point validity, pose deviation, and occlusion degree, selecting the top-ranked high-quality face samples within the trajectory, and choosing the optimal single-frame feature or sample weighted mean vector as the trajectory-level representation information.

6. The video data storyboard method for intelligent video generation according to claim 1, characterized in that, The expression for the total loss function of the second-stage multimodal fine-tuning in S80 is as follows: Mtotal=αMsem+βMface+γMvoice+δMlip Msem is the semantic matching loss for the text and scene; Mface is the GID consistency loss for characters across scenes, which constrains the identity of the same character across scenes; Mvoice is the voice character binding loss; Mlip is the alignment loss between voice timing and the mouth-out-of-camera interval of the scene; α, β, and γ are weight adjustment coefficients.

7. The video data storyboard method for intelligent video generation according to claim 1, characterized in that, The speech segment allocation constraints described in S50 include three types of constraint rules: Visibility constraint: The allocation operation can only be performed if the proportion of visible frames of the target's global identifier within the frame interval corresponding to the speech reaches more than 80%, and the average face quality score meets the standard. Consistency constraint: All speech segments corresponding to the same speaker are uniformly bound to a single global identity identifier and are not allowed to be assigned across multiple characters; Conflict resolution constraints: When multiple global identifiers simultaneously meet the allocation conditions, a unique target character is selected based on the priority of three indicators: face trajectory coverage in the storyboard, average face quality, and total on-screen time in the entire video.

8. The video data storyboard method for intelligent video generation according to claim 1, characterized in that, The lip reconstruction conditions described in S50 include three types of parameters: Audio features: Mel spectrum or deep speech time-frequency features extracted from spoken dialogue; Lip region mask: A binary mask is generated by extracting the lip contour based on the 68 key points of the face and expanding it outward by 10% to 20%. Identity constraint features: These correspond to the global identifier of the optimal face frame feature vector or the high-quality sample mean feature vector, and are used to suppress the problem of identity drift during the reconstruction process.

9. A smart video generation storyboard system, characterized in that, For performing the method according to any one of claims 1 to 8, the system comprises the following modules: The multimodal material storage module is used to store the film and television training material library, the multimodal training dataset, and the final output integrated structured storyboard data package; The copywriting layered semantic parsing module connects with the multimodal material storage module to perform three-layer semantic parsing operations on the original copywriting, including narrative, dialogue, and camera movement, and outputs structured semantic tags. The in-camera face trajectory generation module is used to perform face detection frame by frame on the video segment, perform bidirectional offline tracking calculations, and generate the original face trajectory for a single shot. The semantically enhanced global face identifier fusion module has a built-in four-layer cross-camera gating verification unit, face quality control unit, trajectory representation extraction unit, and disjoint set fusion unit. It generates a unified global identity identifier and a global face trajectory table based on text semantic tags. The voice role binding and lip parameter generation module is used for speaker separation calculation, voice segment allocation calculation, and batch generation of lip reconstruction condition parameters for each role; The four-modal sample construction module integrates text semantic tags, scene data, face trajectory table, and voice binding correspondence into four-modal matching training samples and stores them in the database; The two-stage large model fine-tuning module sequentially performs the first stage of text semantic fine-tuning and the second stage of multimodal audiovisual supervised fine-tuning, outputting a semantically aware large model for storyboard generation. The intelligent storyboard generation module receives user input of creative text and pre-defined semantic constraint tags, and calls a dedicated large model to generate a structured storyboard sequence carrying metadata. The storyboard face identification secondary verification module, based on the spatiotemporal information of the storyboard, re-verifies the global identity identification mapping relationship and updates the trajectory table by re-verifying the semantic tags of the text characters; The integrated structured output module packages the text analysis report, storyboard script, face trajectory table, voice binding mapping, lip reconstruction parameters, and outputs a complete storyboard data package.

10. The intelligent video generation and storyboard system according to claim 9, characterized in that, The semantically enhanced face global identifier fusion module's four-layer cross-camera gating verification unit integrates text role semantic gating, temporal gating, spatial displacement gating, and intersection-union gating. It prioritizes the execution of text role semantic gating filtering operations to pre-screen out invalid merging pairs that are visually similar but whose text belongs to different people.

Images