An automated video generation system and method with AIAgentic multimodal cooperative control

Abstract

This application discloses an automated video generation system and method based on AI Agentic multimodal collaborative control. The system includes: a requirement acquisition module for acquiring multimodal generation requirement information; a hierarchical intent parsing and dynamic graph construction module for generating an intent graph with priorities and constraints; a multi-scale spatiotemporal capsule scene generation module for generating multi-scale spatiotemporal capsule modeling information; a physical-semantic joint constraint verification module for acquiring verified modeling information; a spatiotemporal consistency optimization and content fusion module for acquiring a spatiotemporally consistent optimized video frame sequence; and a dynamic bitrate video synthesis and output module for generating video files. This application overcomes the rigidity and separation problems of traditional video generation technologies through a dynamic adaptive architecture and global joint optimization, achieving intelligent control throughout the entire process from multimodal input to high-quality video output.

Description

Technical Field

[0001] This application relates to the field of video generation technology, specifically to an automated video generation system and method based on AIAgentic multimodal collaborative control. Background Technology

[0002] Existing automated video generation methods mostly adopt a traditional pipeline architecture, consisting of four stages: multimodal input parsing, static intent mapping, step-by-step scene generation, and post-physical verification. Multimodal processing relies on fixed rules (such as keyword matching and template libraries), resulting in rigid intent parsing that cannot understand emerging combinations (such as the implicit contradiction of "a dragon breathing fire + a clear sky"). Intent mapping decomposes sub-intents through hard-coded rules, resulting in a static and fixed graph lacking dynamic adjustment capabilities. Scene generation involves step-by-step stitching together macro, meso, and micro levels, with action matching relying on a pre-set library and details dependent on texture mapping, easily leading to spatiotemporal inconsistencies. Physical verification and semantic checking are processed sequentially; only errors require manual parameter adjustment, making it impossible to jointly optimize physical laws and user intent. Its core shortcomings are weak dynamic adaptability, simplistic multimodal fusion, and separation of physical and semantic processing, making it difficult to balance image quality and feasibility. Summary of the Invention

[0003] The purpose of this invention is to provide an automated video generation system with AIAgentic multimodal collaborative control to at least solve one of the above-mentioned technical problems.

[0004] One aspect of the present invention provides an AI Agentic multimodal collaborative control automated video generation system, the AI ​​Agentic multimodal collaborative control automated video generation system comprising:

[0005] A requirement acquisition module, which is used to acquire multimodal generation requirement information provided by the user;

[0006] A hierarchical intent parsing and dynamic graph construction module is used to parse the multimodal generation requirement information, thereby generating an intent graph with priority and constraints.

[0007] A multi-scale spatiotemporal capsule scene generation module is used to generate multi-scale spatiotemporal capsule modeling information based on the dynamically evolving intent graph.

[0008] The physical-semantic joint constraint verification module is used to verify the multi-scale spatiotemporal capsule modeling information, thereby obtaining the verified modeling information.

[0009] The spatiotemporal consistency optimization and content fusion module is used to perform spatiotemporal consistency optimization and content fusion on the verified modeling information, thereby obtaining a spatiotemporally consistent optimized video frame sequence;

[0010] A dynamic bitrate video synthesis and output module is used to generate video files based on a video frame sequence optimized for spatiotemporal consistency.

[0011] Optionally, the demand acquisition module includes:

[0012] A text information acquisition module, which is used to acquire text information provided by the user;

[0013] A visual information acquisition module, wherein the visual information acquisition module is used to acquire visual information provided by the user;

[0014] An audio information acquisition module, wherein the audio information acquisition module is used to acquire audio information provided by the user;

[0015] The text processing module is used to extract text features from the text information and expand the text features to obtain text modal features, which include text semantic vectors, expanded keyword lists and corresponding semantic weights.

[0016] A visual processing module is used to extract features from visual information to obtain visual modal features, which include object-level visual feature vectors and visual dynamic trajectory feature vectors.

[0017] An audio processing module is used to extract the audio information to obtain audio modal features, which include acoustic feature vectors and emotion tags.

[0018] The dynamic modal weight allocation module is used to allocate weights to text modal features, visual modal features and audio modal features respectively, so as to obtain a multimodal feature vector after dynamic weight allocation.

[0019] A cross-modal residual learning and alignment module is used to perform residual adjustment on the multimodal feature vectors after dynamic weight allocation, thereby obtaining residual-adjusted multimodal feature vectors, which serve as the required information.

[0020] Optionally, the hierarchical intent parsing and dynamic graph construction module includes:

[0021] An intent classification module is used to generate a core intent set based on the multimodal generation requirement information, the core intent set including at least one core intent information;

[0022] A confidence scoring module is used to score the confidence of each core intent information.

[0023] The sub-intent recursive decomposition module is used to decompose each core intent information separately to obtain executable sub-intents. A core intent information includes at least one sub-intent.

[0024] A hierarchical intent graph construction module is used to generate and construct a hierarchical intent graph based on core intent information and sub-intents; wherein, the nodes of the constructed hierarchical intent graph include core intent information and sub-intents, and the edges represent dependencies.

[0025] A dynamic intent graph evolution and adjustment module is used to generate a dynamically adjusted intent graph based on a hierarchical intent graph.

[0026] The intent priority sorting and constraint generation module is used to adjust the priority and constraints of each node in the dynamically adjusted intent graph, thereby generating an intent graph with priorities and constraints.

[0027] Optionally, the multi-scale spatiotemporal capsule scene generation module includes:

[0028] A macro-scale scene layout generation module is used to generate macro-scale scene capsules based on the intent graph with priority and constraints.

[0029] A mesoscale character motion and special effects generation module, which is used to generate mesoscale motion and special effects capsules based on the macroscale scene capsule and the requirement information;

[0030] The microscale detail texture and lighting generation module is used to generate microscale detail and lighting capsules based on macroscale scene capsules, mesoscale action and special effects capsules, and requirement information.

[0031] The spatiotemporal capsule routing and cross-scale connection module is used to merge macroscopic scene capsules, mesoscopic action and special effects capsules, and microscopic detail and light and shadow capsules into a spatiotemporal consistency capsule set.

[0032] The multi-scale content fusion and preliminary rendering module is used to perform capsule fusion and preliminary rendering on the spatiotemporal consistency capsule set to obtain a preliminary rendered video frame sequence, which serves as multi-scale spatiotemporal capsule modeling information.

[0033] Optionally, the physical-semantic joint constraint verification module includes:

[0034] A joint constraint field construction module is used to generate a joint constraint field model based on the multi-scale spatiotemporal capsule modeling information and an intent graph with priority and constraint conditions.

[0035] The simulated annealing optimization module is used to optimize the joint constraint field model using a simulated annealing algorithm, thereby obtaining the optimized generation parameters.

[0036] A real-time physics engine verification module is used to verify the optimized generation parameters, thereby generating physics violation markers and correction suggestions.

[0037] A real-time semantic discriminator verification module is used to perform semantic scoring and evaluation on the optimized generated parameters, thereby obtaining semantic violation tags and correction suggestions.

[0038] A parameter adjustment module is provided, which is used to adjust the optimized generation parameters according to the physical violation markers and correction suggestions and the semantic violation markers and correction suggestions, so as to obtain the adjusted generation parameters.

[0039] A re-rendering module is used to adjust the multi-scale spatiotemporal capsule modeling information according to the adjusted generation parameters, thereby obtaining verified modeling information.

[0040] Optionally, the spatiotemporal consistency optimization and content fusion module includes:

[0041] A spatiotemporal consistency detection and marking module is used to perform spatiotemporal consistency detection on the verified modeling information to obtain spatiotemporal inconsistency markers, which include the frame number to be optimized and the region coordinates.

[0042] An optical flow prediction and motion trajectory smoothing module is used to process the spatiotemporal inconsistency marker to form a smoothed motion vector field, which includes corrected inter-frame motion parameters.

[0043] A timing discriminator verification and coherence optimization module is used to generate a sequence of video frames that pass timing coherence verification based on a smoothed motion vector field.

[0044] Multi-scale content fusion and final rendering are used to generate a spatiotemporally consistent video frame sequence based on the video frame sequence that has passed the temporal coherence verification, macro scene capsules, meso action and special effects capsules, and micro detail and light and shadow capsules.

[0045] Optionally, the dynamic bitrate video synthesis and output module includes:

[0046] The content importance analysis and region segmentation module is used to identify the video frame sequence after spatiotemporal consistency optimization, thereby obtaining a content importance map, which includes the importance score of each region in each frame;

[0047] A dynamic bitrate allocation strategy formulation module is used to dynamically allocate bitrate based on the content importance graph, thereby generating a dynamic bitrate allocation table, which includes the bitrate values ​​of each region in each frame.

[0048] Video encoding and dynamic bitrate compression are used to encode and compress the spatiotemporally consistent optimized video frame sequence according to the dynamic bitrate allocation table, thereby obtaining a dynamically bitrate compressed video file.

[0049] Optionally, the hierarchical intent parsing and dynamic graph construction module further includes:

[0050] The intent conflict detection and self-repair module detects and repairs conflicts based on the dynamically adjusted intent graph, thereby obtaining a repaired intent graph; the intent priority sorting and constraint generation module processes the repaired intent graph.

[0051] Optionally, the intent contradiction detection and self-repair module includes:

[0052] An intent conflict detection module is used to detect intent conflicts using the ICOF formula, thereby obtaining conflict markers.

[0053] An intent conflict repair module is used to repair intent conflicts based on the conflict markers, thereby obtaining a repaired intent graph.

[0054] This application also provides an automated video generation method based on AI Agentic multimodal collaborative control, characterized in that the automated video generation method based on AI Agentic multimodal collaborative control includes:

[0055] Obtain multimodal generation requirements information provided by the user;

[0056] The multimodal generation requirement information is parsed to generate an intent graph with priorities and constraints.

[0057] Multi-scale spatiotemporal capsule modeling information is generated based on the dynamically evolving intent map;

[0058] The modeling information of the multi-scale spatiotemporal capsule is validated to obtain validated modeling information;

[0059] The validated modeling information is subjected to spatiotemporal consistency optimization and content fusion to obtain a spatiotemporally consistent optimized video frame sequence;

[0060] Video files are generated based on the video frame sequence optimized for spatiotemporal consistency.

[0061] The AIAgentic multimodal collaborative control automated video generation method of this application overcomes the rigidity and separation problems of traditional video generation technologies through dynamic adaptive architecture and global joint optimization, realizing intelligent control of the entire process from multimodal input to high-quality video output. Its core advantages lie in the flexibility of intent parsing, physical-semantic collaborative verification, strong guarantee of spatiotemporal consistency, and accuracy of output adaptation, significantly improving the accuracy, coherence, and user experience of the generated video. Attached Figure Description

[0062] Figure 1 This is a system schematic diagram of an AIAgentic multimodal collaborative control automated video generation system according to an embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0064] like Figure 1 The AIAgentic multimodal collaborative control automated video generation system shown includes a demand acquisition module, a hierarchical intent parsing and dynamic atlas construction module, a multi-scale spatiotemporal capsule scene generation module, a physical-semantic joint constraint verification module, a spatiotemporal consistency optimization and content fusion module, and a dynamic bitrate video synthesis and output module.

[0065] The requirement elicitation module is used to obtain multimodal generation requirement information provided by users;

[0066] The hierarchical intent parsing and dynamic graph construction module is used to parse the multimodal generation requirement information, thereby generating an intent graph with priority and constraints.

[0067] The multi-scale spatiotemporal capsule scene generation module is used to generate multi-scale spatiotemporal capsule modeling information based on the dynamically evolving intent map.

[0068] The physical-semantic joint constraint verification module is used to verify the multi-scale spatiotemporal capsule modeling information, thereby obtaining verified modeling information;

[0069] The spatiotemporal consistency optimization and content fusion module is used to perform spatiotemporal consistency optimization and content fusion on the validated modeling information, thereby obtaining a spatiotemporal consistency optimized video frame sequence;

[0070] The dynamic bitrate video synthesis and output module is used to generate video files based on the video frame sequence optimized for spatiotemporal consistency.

[0071] The AIAgentic multimodal collaborative control automated video generation method of this application overcomes the rigidity and separation problems of traditional video generation technologies through dynamic adaptive architecture and global joint optimization, realizing intelligent control of the entire process from multimodal input to high-quality video output. Its core advantages lie in the flexibility of intent parsing, physical-semantic collaborative verification, strong guarantee of spatiotemporal consistency, and accuracy of output adaptation, significantly improving the accuracy, coherence, and user experience of the generated video.

[0072] In this embodiment, the requirement acquisition module includes:

[0073] A text information acquisition module, which is used to acquire text information provided by the user;

[0074] A visual information acquisition module, wherein the visual information acquisition module is used to acquire visual information provided by the user;

[0075] An audio information acquisition module, wherein the audio information acquisition module is used to acquire audio information provided by the user;

[0076] The text processing module is used to extract text features from the text information and expand the text features to obtain text modal features, which include text semantic vectors, expanded keyword lists and corresponding semantic weights.

[0077] A visual processing module is used to extract features from visual information to obtain visual modal features, which include object-level visual feature vectors and visual dynamic trajectory feature vectors.

[0078] An audio processing module is used to extract the audio information to obtain audio modal features, which include acoustic feature vectors and emotion tags.

[0079] The dynamic modal weight allocation module is used to allocate weights to text modal features, visual modal features and audio modal features respectively, so as to obtain a multimodal feature vector after dynamic weight allocation.

[0080] A cross-modal residual learning and alignment module is used to perform residual adjustment on the multimodal feature vectors after dynamic weight allocation, thereby obtaining residual-adjusted multimodal feature vectors, which serve as the required information.

[0081] For example, the text information acquisition module can perform the following operations: The user types in: "I want a video of a red dragon fighting in a snow mountain, with lightning and roar sounds", corrects "roar" to "dragon roar", and finally obtains the cleaned text: "I want a video of a red dragon fighting in a snow mountain, with lightning and dragon roar sounds".

[0082] For example, the visual information acquisition module can perform the following operations: the user uploads a picture with a "snow mountain background" and a 10-second short video of a "dragon flying". The module first scales the picture to 512×512, extracts one key frame for every two frames of the video (a total of 5 frames), and removes the watermark from the corner of the video, finally obtaining 6 visual reference materials.

[0083] For example, the audio information acquisition module can perform the following operations: the user uploads a mixed audio containing "thunder" and "wind". The module first performs noise reduction processing, and then uses silence detection to divide the audio into two independent segments: "thunder (3 seconds)" and "wind (5 seconds)", which are used as inputs for subsequent audio feature extraction.

[0084] For example, the text processing module can perform the following operations: Process the text "Red dragon battles in the snowy mountains" as follows:

[0085] Semantic vector: A 768-dimensional vector encoded with CLIP-Text, containing the combined semantics of "red + dragon + snow mountain + battle";

[0086] Extended keywords: [red dragon, giant dragon, snow mountain, battle, fight, snow accumulation, cliff, flame];

[0087] Semantic weights: "dragon" (0.9), "battle" (0.8), "red" (0.7), "snow mountain" (0.6), and other related words (0.3-0.5).

[0088] For example, the vision processing module can perform the following operations:

[0089] Object-level visual feature vectors:

[0090] Use the YOLOv8 object detection model to identify core objects (such as "dragon", "snow mountain", "sky") in visual materials;

[0091] For each object, a 2048-dimensional feature vector (containing shape, color, and texture information) is extracted using the ResNet50 CNN model.

[0092] Visual dynamic trajectory feature vector (for video / sequence frames):

[0093] The FlowNet optical flow model is used to calculate motion vectors between adjacent frames to capture the motion trajectory of objects.

[0094] The trajectory is temporally encoded (e.g., LSTM) to generate a 128-dimensional dynamic feature vector (containing velocity, direction, and acceleration information).

[0095] For example, the audio processing module can perform the following operations:

[0096] Acoustic eigenvectors:

[0097] Acoustic features such as MFCC (Mehr frequency cepstral coefficients, 39-dimensional), spectral entropy (1-dimensional), and zero-crossing rate (1-dimensional) are extracted.

[0098] The features are fused into a 256-dimensional vector using a CNN model (containing pitch, loudness, and rhythm information).

[0099] Emotion tag generation:

[0100] Use a pre-trained audio emotion classification model (such as CNN-LSTM) to classify audio and output emotion labels (such as "majestic", "tense", "calm"); for audio without clear emotion (such as pure wind sound), label it as "neutral".

[0101] For example, the dynamic modal weight allocation module can perform the following operations:

[0102] The weights are dynamically calculated using an attention mechanism (such as multi-head attention in Transformer) to ensure that the sum of the weights is 1. Specifically, the weights are calculated based on the following:

[0103] Modal quality: text clarity (unambiguous → high weight), visual resolution (high definition → high weight), audio signal-to-noise ratio (no noise → high weight);

[0104] User input intensity: If the user provides detailed text + blurry image → text weight is high; if the user provides clear image + brief text → visual weight is high.

[0105] Cross-modal consistency: If the text “red dragon” matches the color of the dragon in the visual image, the weight of both is increased; if they contradict each other, the weight of the conflicting modality is decreased.

[0106] In this embodiment, the cross-modal residual learning and alignment module can perform the following operations:

[0107] Using CLIP's contrastive learning mechanism, the distance between text, visual, and audio features in the semantic space is calculated (the greater the distance, the greater the difference).

[0108] The residual network (ResNet) is used to correct modal features with large differences (e.g., if the visual features deviate greatly from the text "red", the color dimension of the visual features is adjusted).

[0109] By calculating cross-modal similarity (such as cosine similarity), we ensure that the similarity of each modal feature after adjustment is ≥0.8 (the threshold can be dynamically adjusted).

[0110] In this embodiment, the hierarchical intent parsing and dynamic graph construction module includes:

[0111] An intent classification module is used to generate a core intent set based on the multimodal generation requirement information, the core intent set including at least one core intent information;

[0112] A confidence scoring module is used to score the confidence of each core intent information.

[0113] The sub-intent recursive decomposition module is used to decompose each core intent information separately to obtain executable sub-intents. A core intent information includes at least one sub-intent.

[0114] A hierarchical intent graph construction module is used to generate and construct a hierarchical intent graph based on core intent information and sub-intents; wherein, the nodes of the constructed hierarchical intent graph include core intent information and sub-intents, and the edges represent dependencies.

[0115] A dynamic intent graph evolution and adjustment module is used to generate a dynamically adjusted intent graph based on a hierarchical intent graph.

[0116] The intent priority sorting and constraint generation module is used to adjust the priority and constraints of each node in the dynamically adjusted intent graph, thereby generating an intent graph with priorities and constraints.

[0117] In this embodiment, the intent classification module can perform the following operations:

[0118] The cross-modal aligned feature vectors (from the demand acquisition module) are input into the "CLIP+BiLSTM" hybrid model and classified using a pre-trained intent category library (such as "scene", "subject", "action", "effects" and "style").

[0119] In this embodiment, the confidence scoring module can evaluate the intent based on modal consistency, feature clarity, and model output probability. Specifically, modal consistency means that if the text, visual, and audio all mention the same intent (e.g., "red dragon" appears in both text and visual), the score is improved. Feature clarity means that intents with unambiguous text and high visual resolution receive higher scores. Model output probability means that the prediction probability of the classification model for the intent (e.g., the classification probability of CLIP) directly affects the score.

[0120] For example, the confidence score is calculated using a multi-dimensional weighted approach, comprehensively considering the cross-modal consistency of intent, the quality of associated materials, and the determinism of the model classification. The specific rules are as follows:

[0121] Weighting: Cross-modal feature alignment score accounts for 40% of the weight, material quality score accounts for 30% of the weight, and intent classification probability accounts for 30% of the weight (this weighting ratio has been verified through a large number of experiments and can balance multimodal consistency and material reliability, avoiding single-dimensional bias).

[0122] Score Calculation: First, convert the raw scores of 0-1 for each dimension into standard scores of 0-100 (e.g., alignment score 0.9 → 90, material quality score 0.8 → 80, classification probability 0.85 → 85). Then, calculate the weighted sum according to the weights, and the final result is rounded to one decimal place.

[0123] The formula is as follows: Confidence score = (Cross-modal alignment standard score × 40%) + (Material quality standard score × 30%) + (Classification probability standard score × 30%).

[0124] In this embodiment, the sub-intent recursive decomposition module can perform the following operations:

[0125] Step 1: Matching core intent with decomposed knowledge base:

[0126] For each high-confidence core intent in the confidence scoring module, the corresponding standard decomposition rules are matched in the domain intent decomposition knowledge base based on its category and description text. For example, for the core intent "Dragon Battle" (action type), the decomposition rule "Battle Action → Attack Action / Defense Action / Movement Action" is matched in the knowledge base, determining that the initial decomposition direction of this core intent is three types of action branches.

[0127] If a core intent does not have a completely matching rule in the knowledge base (such as the emerging style intent "cyberpunk snow mountain"), then the similar intent decomposition rule is called for adaptation and marked as "decomposition direction to be verified", which will be supplemented and improved in the future with manual review.

[0128] Step 2: First-level sub-idea generation:

[0129] Based on the matched decomposition rules, first-level sub-intents of the core intent are generated. Taking "Dragon Battle" as an example, according to the decomposition direction of "attack action / defense action / movement action", three first-level sub-intents are generated: "Dragon Attack", "Dragon Defense", and "Dragon Battlefield Movement". Each first-level sub-intent must clearly describe the core action it refers to. For example, "Dragon Attack" must specify that it is an active attack behavior against an enemy target. At the same time, each first-level sub-intent is assigned a unique identifier and associated with the corresponding core intent ID.

[0130] Step 3: Second-level sub-idea decomposition:

[0131] Each primary sub-intention is further broken down into more refined secondary sub-intentions. Taking "dragon attack" as an example, based on the subdivision rules of "attack actions" in the domain knowledge base, it is broken down into three secondary sub-intentions: "flame jet attack," "claw swipe attack," and "tail sweep attack." During the decomposition process, it is necessary to determine whether the secondary sub-intentions are close to atomic operations. If some parameters can be preliminarily extracted from a certain secondary sub-intention (such as "flame jet attack," which can preliminarily determine the spray range), then the direction of parameter extraction is marked to prepare for the next step of atomic-level decomposition.

[0132] Step 4: Atomic-level sub-intention verification and determination:

[0133] For each secondary sub-intention, check the atomic operation library to determine if it can be directly mapped to an atomic operation. If the secondary sub-intention "Flame Jet Attack" can find the "Effect Generation - Flame Jet" operation in the atomic operation library, and parameters such as jet duration (e.g., 2-3 seconds by default based on "Battle Scene") and flame color (e.g., orange-red by default based on "Dragon Red Appearance") can be extracted from the intention description, then the secondary sub-intention is converted into an atomic-level sub-intention and marked as a leaf node. If the secondary sub-intention "Claw Swing Attack" cannot directly extract complete parameters (e.g., the swing amplitude is not specified), then it is further broken down into three atomic-level sub-intentions: "Claw Raises", "Claw Swings Forward", and "Claw Retracts". Each sub-intention corresponds to the "Limb Action - Claw Raises", "Limb Action - Claw Swings", and "Limb Action - Claw Retracts" operations in the atomic operation library, and the execution duration of each action is specified (e.g., 0.5 seconds for raising, 1 second for swinging, and 0.5 seconds for retracting).

[0134] Step 5: Structured Integration of Decomposition Trees

[0135] The core intent, first-level sub-intents, second-level sub-intents, and atomic-level sub-intents are integrated into a decomposition tree according to their hierarchical relationship. The execution dependencies of each node are supplemented (e.g., "claw waving" depends on "claw lifting") to ensure that the decomposition tree structure is clear and logically coherent. At the same time, the decomposition tree is checked for completeness to see if there are any cases where the core intent has not been decomposed or the sub-intents are not associated with atomic operations. If so, the previous step is returned to re-decompose and adjust.

[0136] In this embodiment, the hierarchical map construction module can perform the following operations:

[0137] Step 1: Node Information Extraction and Initialization

[0138] From the hierarchical sub-intent decomposition tree, the unique identifier, description text, hierarchy, and associated core intent ID of each intent are extracted one by one. Then, the scores in the intent confidence list are matched and integrated into the basic information of the graph nodes. The initial state of each node is set to "pending activation" and it is categorized according to its hierarchy (e.g., core intents are categorized into top-level nodes, and atomic operations are categorized into bottom-level nodes) to ensure that no node information is missing (e.g., nodes without associated core intent IDs need to be supplemented by backtracking the decomposition tree).

[0139] Step 2: Dependency Analysis and Classification

[0140] First, extract explicit dependencies from the "execution dependencies" of the decomposition tree (e.g., "claw waving depends on claw lifting"), then combine this with the domain intent association knowledge base to mine implicit dependencies (e.g., "all combat-related sub-intents depend on the core intent 'dragon combat'", "the special effects sub-intent in the snow mountain scene depends on the 'snow mountain environment' scene intent"). For each set of dependencies identified, classify them according to "premise / temporal / spatial" type. For example, "dragon existence → flame spray" is classified as a premise dependency, "claw lifting → claw waving" as a temporal dependency, and "dragon stomping → snow splashing" as a spatial dependency.

[0141] Step 3: Calculate the dependency strength weight:

[0142] For each group of categorized dependencies, calculate the dependency strength weight: First, take the average of the confidence scores of the two related intentions (e.g., A has a confidence score of 90, B has a confidence score of 85, the average is 87.5, converted to 0.875). Then, adjust according to the association coefficient of this type of dependency in the domain knowledge base (add 0.05 if it is a strong association, subtract 0.05 if it is a weak association, and leave it unchanged if there is no explicit association). Finally, constrain the result to between 0 and 1 (e.g., if the calculated result is 1.02, take 1.0; if it is 0.03, take 0.03). For example, "Dragon exists → Flame jet", the average is 0.875, and it is a strong association in the knowledge base, so add 0.05, and the final weight is 0.925; "Snow splashes → Rock collapse", the average is 0.75, there is no explicit association, so the weight is 0.75.

[0143] Step 4: Graph Structure Construction and Visualization Mapping

[0144] The graph is constructed according to the layout rule of "hierarchy from top to bottom, dependencies from left to right": the top-level core intent node is centered, the second-level first-level sub-intent nodes are distributed around the core node, the third-level second-level sub-intent nodes correspond to the nodes below their respective first-level sub-intent nodes, and the fourth-level atomic operation nodes correspond to the nodes below their respective second-level sub-intent nodes. The defined dependencies are connected to the corresponding nodes using "edges," with the edge thickness adjusted according to the dependency strength weight (the higher the weight, the thicker the edge). The dependency type and description are labeled next to the edges (e.g., "Prerequisite dependency: Flamejet requires the existence of a dragon"). Simultaneously, colors are added to the nodes to distinguish their types (red for core intents, blue for first-level sub-intents, and green for atomic operations) to ensure the graph is intuitively identifiable.

[0145] Step 5: Verification and Supplementation of Spectral Information:

[0146] Check the completeness of the graph: Confirm that all intents extracted from the decomposition tree have been converted into nodes without omission; all identified dependencies have been converted into edges without missing any. Check the logical rationality: Ensure there are no circular dependencies (e.g., A depends on B, B depends on A), and no abnormal dependencies across levels (e.g., atomic operation nodes depend on core intent nodes; confirm whether this conforms to knowledge base rules). If any issues are found, go back to step two to re-examine the dependencies. Finally, add a unique identifier to each node and edge to complete the structured integration of the graph.

[0147] In this embodiment, the dynamic graph evolution and adjustment module can perform the following operations:

[0148] Step 1: Analysis and Classification of Feedback Information

[0149] For each piece of real-time feedback, extract the "related intent ID, feedback type, and adjustment suggestion," and archive them according to "user feedback / module verification feedback / resource feedback." For example, user feedback "Add night scene" (related ID: none, type: add, suggestion: add the core intent of "night scene"), physics engine feedback "dragon mass exceeds the limit" (related ID: INT-subject-001, type: conflict, suggestion: reduce dragon mass parameters), and resource feedback "particle effect computing power is insufficient" (related ID: INT-effect-001, type: resource limitation, suggestion: simplify effect). Simultaneously, verify the completeness of the feedback information. If a related ID or adjustment suggestion is missing, return to the previous step to supplement it (e.g., if the "semantic conflict" feedback does not specify a related intent, prompt the relevant module to supplement it).

[0150] Step 2: Generating matching evolution rules and preliminary adjustment schemes:

[0151] For each type of feedback, a corresponding adjustment strategy is matched in the graph evolution rule base to generate a preliminary solution. For example, for feedback on a user adding an intent, the "Add Node Rule" is matched to determine the node level ("Night Scene" is the core intent, placed at the top level), the initial confidence level (the user clearly states their need, set at 90 points), and the initial state (to be activated); for feedback on physical conflicts, the "Conflict Handling Rule" is matched to reduce the confidence level of the associated node "Dragon" (from 90 points to 80 points) and mark the node state as "Conflict to be Repaired"; for feedback on resource constraints, the "Resource Compromise Rule" is matched to change the sub-intent of "Particle Effects" to "Simplify Flame Effects", adjust the node description, and reduce the confidence level (from 85 points to 70 points).

[0152] Step 3: Feasibility verification and conflict resolution of the solution:

[0153] The initial adjustment plan is verified from two aspects: "logical feasibility" and "resource feasibility." Logical feasibility requires checking whether the added dependencies conform to hierarchical rules (e.g., top-level nodes cannot depend on bottom-level nodes) and whether deleting nodes affects the execution of core intents (e.g., deleting "Dragon Exists" would cause all combat sub-intents to fail, thus rejecting the deletion plan). Resource feasibility requires considering current system resources (e.g., computing power, storage) to determine whether the adjusted intent is executable (e.g., whether the computing power requirement for "Simplifying Flame Effects" is within the graphics card's capacity). If a conflict is found (e.g., the addition of a "Night Scene" contradicts the existing "Snow Mountain Daytime" scene), conflict negotiation is initiated, prioritizing the retention of the user-required "Night Scene" and deleting the "Snow Mountain Daytime" node and related edges.

[0154] Step 4: Actual adjustment of graph nodes and edges:

[0155] Based on the verified solution, perform the specific adjustment operations:

[0156] Node adjustments: Adding nodes (inserting nodes into the graph hierarchically, such as adding a top-level core node to the "Night Scene"), deleting nodes (removing nodes and all references corresponding to associated IDs), changing status (such as "Conflict pending" → "Resolved"), and modifying confidence (updating scores in node information).

[0157] Edge adjustments include: adding edges (connecting new nodes to related nodes, such as the prerequisite dependency of "night scene" → "moon glowing", with a weight of 0.9), deleting edges (removing all edges related to the deleted node), and recalculating weights (e.g., if the confidence of "dragon" drops to 80 points, the weight of the related edge "dragon → flame jet" is recalculated from 0.925: average (80+85) / 2=82.5→0.825, strong associations add 0.05→0.875). During the adjustment process, operation logs (including time, operation content, and feedback ID) are recorded in real time to ensure traceability.

[0158] Step 5: Verification and Confirmation of the Adjusted Spectrum

[0159] Check the completeness of the adjusted graph: Ensure all necessary information (ID, type, confidence, etc.) has been added to new nodes; that deleted nodes and edges have been completely removed; and that weight adjustments are all within the range of 0-1. Check logical consistency: ensure there are no new circular dependencies (e.g., "night scene → moon glows" vs. "moon glows → night scene"), and that node states match feedback (e.g., node states have been updated after conflict resolution). Finally, associate and archive the adjusted graph with the feedback information to generate a "Graph Evolution Report" (including a comparison before and after adjustment and explanations of key changes), and submit it to the next module.

[0160] In this embodiment, the intent priority ranking and constraint generation module can perform the following operations:

[0161] Step 1: Input data preparation and preprocessing:

[0162] Extract the ID, type, description, confidence level, and state of all nodes from the dynamically adjusted intent graph, as well as the dependency type and weight of edges; integrate user needs and preferences (e.g., "core is fighting dragons") with system resource configuration (e.g., 16GB of graphics card computing power and 32GB of memory); retrieve the weights of each dimension (40% for core needs, 30% for confidence level, etc.) from the priority evaluation index library to ensure that all input data is complete; if a node lacks confidence level (e.g., a newly added node is not labeled), it is defaulted to 80 points for explicit user needs and 60 points for non-explicit needs to avoid evaluation bias.

[0163] Step 2: Calculation and sorting of intent nodes priority:

[0164] Multi-dimensional score calculation: Calculate the four-dimensional score for each node one by one. Taking "Dragon Battle" (core intent, confidence score of 90, user clearly understands core needs, 10 nodes with a dependency weight of 0.9 or higher, and resource consumption of 40%) as an example:

[0165] Coreness of demand: Users clearly identify the core demand, scoring 100 points (40% weight → 40 points);

[0166] Confidence level: 90 points (30% weight → 27 points);

[0167] The score increases by 20 points (20% weight → (90+20)×20%=22 points) for 10 strongly dependent nodes;

[0168] Reasonableness of resource consumption: Consuming 40% of computing power yields 60 points (10% weight → 6 points);

[0169] Total score: 40 + 27 + 22 + 6 = 95 points;

[0170] Priority level mapping: 95 points corresponds to priority level 10, marked "Priority 10, sorting reason: core user needs + high confidence + strong dependency on multiple nodes, reasonable resource consumption";

[0171] Priority adjustment: For nodes with similar scores (such as "Night Scene" 92 points and "Simplified Flame Effect" 88 points), since "Simplified Flame Effect" depends on "Night Scene", the priority of "Night Scene" is set to level 10 and "Simplified Flame Effect" is set to level 9 to ensure that the dependency relationship matches the priority.

[0172] Global sorting: Sort by priority level from high to low (level 10 → level 9 → ... → level 1), and sort by score in descending order for the same level, generating a "priority sorting table" and labeling the ID, level, score and reason for each node.

[0173] Step 3: Generating multiple types of constraints:

[0174] Timing constraint generation:

[0175] Extract temporal dependency edges from the graph (e.g., "night scene → moon glows", "dragon exists → claws raised");

[0176] Based on a video length of 15 seconds and a frame rate of 30fps, specific time differences were set: "Night scene" (priority level 10, activation time 0 seconds) → "Moon glowing" (priority level 9, activation time 0-2 seconds, corresponding to 0-60 frames); "Dragon present" (priority level 10, activation time 3 seconds) → "Claws raised" (priority level 8, activation time 4 seconds, corresponding to 91-120 frames).

[0177] Assign an ID to each temporal constraint (e.g., "Temporal Constraint - Night Scene - Moon Glowing - 001"), and label the associated node ID, time range, and consequences of violation (e.g., "If the moon glows for more than 2 seconds, the scene lighting mechanism will be automatically triggered").

[0178] Resource constraint generation:

[0179] Statistics on resource consumption at each node (e.g., "Dragon Action Calculation" 40% computing power, "Simplified Flame Effect" 30% computing power, "Moon Glowing" 5% computing power);

[0180] Based on the system's computing power limit of 100%, the following constraints are set: "Dragon Action Calculation" computing power usage ≤45%, "Simplified Flame Effect" ≤35%, "Moon Glowing" ≤8%, with 12% dynamic space reserved;

[0181] Mark resource monitoring thresholds (e.g., if "Simplified Flame Effect" occupies more than 35%, automatically reduce the number of particles) and generate a "Resource Constraint Table";

[0182] Style constraint generation:

[0183] Extract style parameters (main color dark purple + silver white, saturation 75%, light intensity 200 lux) from the "Nighttime Snow Mountain Fantasy Scene" node;

[0184] Transformed into constraints: "All scene elements have a primary color RGB 30-80, saturation 60%-80%, and light intensity 150-250 lux; the use of bright colors above RGB 200 is prohibited, and realistic style textures are prohibited."

[0185] Associate all visual nodes (such as "snow mountain", "dragon", "moon") and label the consequences of violations (such as "automatically reduce saturation to below 80% when bright color is detected").

[0186] Step 4: Priority and Constraint Graph Integration:

[0187] The levels, scores, and reasons from the "Priority Ranking Table" are added to the corresponding nodes in the dynamic intent graph, and a "Priority Information" field is added to each node. The constraints in the "Temporal / Resource / Style Constraint Table" are associated with the graph nodes, and a "Constraint ID" field is added to the edge set (e.g., temporal dependent edges are associated with corresponding temporal constraint IDs), forming an "intent graph with priorities and constraints." At the same time, a "Priority and Constraint Description Document" is added to the graph to explain the ranking logic and the basis for constraint formulation (e.g., "'Night Scene' priority level 10 is due to the premise of multiple nodes," "Computational power constraints refer to a 16GB graphics card configuration"), to facilitate understanding and execution by subsequent modules.

[0188] Step 5: Verification and confirmation of priorities and constraints:

[0189] Priority verification:

[0190] Check whether the priority of core nodes (such as "Dragon Battle" and "Night Scene") is level 10, whether the priority of auxiliary nodes (such as "Moon Glowing") is level 9 or above, and whether the priority of edge nodes (such as "Distant Birds") is level 5 or below, in accordance with the hierarchical logic of "core → auxiliary → edge".

[0191] Verify the consistency between dependencies and priorities (e.g., prerequisite nodes all have higher priorities than dependent nodes), and check for any anomalies such as "dependency nodes have higher priorities than prerequisite nodes".

[0192] Constraint verification:

[0193] Are the timing constraints within the time range reasonable (e.g., the activation of "moon glowing" within 0-2 seconds conforms to the scene introduction logic, and there is no error of "time range exceeding video duration")?

[0194] Is the total resource constraint ≤ the system upper limit (45% + 35% + 8% + ... = 88% ≤ 100%)? Is the reserved space sufficient?

[0195] Does the style constraint cover all visual nodes? Are the parameters quantifiable (e.g., "RGB 30-80" is clear, without vague expressions such as "aesthetically pleasing style")?

[0196] Final confirmation: Generate a "Priority and Constraint Verification Report", record the verification results (such as "Priority sorting is logical and constraints are executable"), and submit the intent graph with priorities and constraints to the multi-scale spatiotemporal capsule scene generation module.

[0197] In this embodiment, the multi-scale spatiotemporal capsule scene generation module includes:

[0198] A macro-scale scene layout generation module is used to generate macro-scale scene capsules based on the intent graph with priority and constraints.

[0199] A mesoscale character motion and special effects generation module, which is used to generate mesoscale motion and special effects capsules based on the macroscale scene capsule and the requirement information;

[0200] The microscale detail texture and lighting generation module is used to generate microscale detail and lighting capsules based on macroscale scene capsules, mesoscale action and special effects capsules, and requirement information.

[0201] The spatiotemporal capsule routing and cross-scale connection module is used to merge macroscopic scene capsules, mesoscopic action and special effects capsules, and microscopic detail and light and shadow capsules into a spatiotemporal consistency capsule set.

[0202] The multi-scale content fusion and preliminary rendering module is used to perform capsule fusion and preliminary rendering on the spatiotemporal consistency capsule set to obtain a preliminary rendered video frame sequence, which serves as multi-scale spatiotemporal capsule modeling information.

[0203] In this embodiment, the macro-scale scene layout generation module can perform the following operations:

[0204] Step 1: Extracting Scene-Related Intents and Constraints:

[0205] From the intent graph with priorities and constraints, intents with a "scene / subject type" and a "priority ≥ 8" are selected (high priority ensures no core elements are missed). The description text, constraints, and spatial dependencies of each intent are extracted. For example, "Night Snow Mountain Scene" (scene type, priority 10, constraints: "includes main peak, valley, sky, and moon, fantasy dark tone") and "Red Dragon" (subject type, priority 9, constraints: "active near the main peak of the snow mountain, large in size") are extracted. The spatial dependency of "Moon Glowing" (effect type, dependent on the "Night Snow Mountain" sky area) is also recorded. If a constraint is ambiguous (e.g., "large in size" does not specify dimensions), a default value is matched from the scene's basic parameter library (e.g., "the default size of the large subject is 1 / 3-1 / 2 of the height of the scene's core elements").

[0206] Step 2: Spatial Layout Construction

[0207] Determine the overall spatial scope: Based on the default parameters of the core intent of the scene, adjust the constraints accordingly. For example, the default size of the "snow mountain scene" is 1000m×800m×500m (length×width×height). Since the constraint requires "including valleys", the width is extended to 900m to ensure that there is enough space in the valley area. At the same time, define a three-dimensional coordinate system (e.g., with the bottom left corner of the scene as the origin (0,0,0), the x-axis as length, the y-axis as width, and the z-axis as height) to facilitate the subsequent annotation of position coordinates.

[0208] Divide the core area: Divide it into "core functional area → interaction area → background area". For example, the coordinate range of (400-700, 300-600, 200-500) of "night snow mountain" is defined as the main peak combat core area (main action core area) and (700-900, 300-600, 100-300) is defined as the valley interaction area (main body and scene interaction area). The remaining area is the background transition area (such as the outline of the distant snow mountain and the edge of the sky). Each area is marked with its functional positioning and spatial constraints (such as "background elements are not allowed to cover the main body" in the core area and "space for snow splashing needs to be reserved" in the interaction area).

[0209] Step 3: Building the Timeline Layout

[0210] Based on the video length and frame rate requirements, divide the scene into stages and define the state of each stage. For example, a video with a length of 15 seconds and a frame rate of 30fps (450 frames in total):

[0211] Opening phase (0-3 seconds, 0-90 frames): Scene introduction, defined as "pulling from a distant view to a panoramic view of the main peak, the moon slowly rises from the right side of the sky, the outline of the snow mountain gradually appears with the moonlight, no main elements";

[0212] Combat Phase (3-12 seconds, 91-360 frames): Core Interaction, defined as "The main dragon flies into the core area from the left side of the main peak, and the scene responds dynamically to the dragon's movements (such as the flapping of wings creating airflow and the mountain shaking slightly), while the moon maintains stable illumination";

[0213] Final phase (12-15 seconds, 361-450 frames): transition freeze frame, the state is defined as "the dragon is standing on the top of the main peak, its wings are slowly folded, the moon halo is enhanced, the background of the scene is gradually darkened, and finally the frame freezes on the panoramic view of the dragon and the main peak".

[0214] Each stage is labeled with keyframe nodes (e.g., 90 frames "the moon rises to the center of the sky", 360 frames "the dragon reaches the top of the main peak") to ensure that the timeline is synchronized with the generation of subsequent actions.

[0215] Step 4: Layout and style adaptation of core elements:

[0216] Determine the position and size of elements: Mark the coordinates and dimensions according to the layout rules of the core elements. For example, the center point of the snow mountain's main peak is (550, 450, 350), with a height of 300m; the dragon's initial position is (300, 450, 250) (to the left of the main peak), with a body length of 80m (about 1 / 3 of the main peak's height); the moon's position is (550, 100, 450) (in the sky area, aligned with the center of the main peak), with a diameter of 50m; all coordinates must ensure that the elements do not exceed the scene space and that the size proportions are coordinated.

[0217] Style Adaptation Adjustment: Based on the style constraints in the intent map (such as "fantasy dark tone"), stylize the core elements. Add light purple ice crystal texture to the surface of the snow mountain, and cover the top of the main peak with silver-white snow. Add a light blue halo to the edge of the moon, with the halo range being twice the diameter of the moon. Add dark gold reflection to the dragon scales to echo the scene's color tone. At the same time, determine the color parameters of the entire scene (primary color RGB(30,40,80), saturation 75%, brightness 30%) to ensure style consistency.

[0218] Step 5: Establishing Global Constraint Standards and Integrating Capsules

[0219] Establish global constraints: Integrate spatial constraints, style constraints, and lighting constraints. For example, spatial constraints: "The dragon's activity range is limited to the coordinates (300-800, 300-600, 200-400)"; style constraints: "All scene elements must not use bright colors above RGB (200, 200, 200), and special effect colors must be derived from the primary color"; lighting constraints: "The moon is the only primary light source, with an illumination intensity of 200 lux, and the shadow direction of ground objects is (-1, -1, 0) (corresponding to the upper left to lower right direction), and the shadow length is twice the object height."

[0220] Macro-scene capsule integration: The spatial layout, timeline layout, core element layout, and global constraint standards are integrated into a structured "macro-scene capsule". Each capsule is labeled with a unique identifier (e.g., "macro-capsule-night snow mountain-001"), generation time, and associated intent graph ID. It is then submitted to the meso-scale character action and special effects generation module. At the same time, the capsule generation log (including the basis for parameter adjustment and the source of constraints) is retained for easy traceability later.

[0221] In this embodiment, the mesoscale character motion and special effects generation module operates as follows:

[0222] Step 1: Input Data Parsing and Requirement Mapping

[0223] Extract spatial range (e.g., core area (400-700), timeline phase (3-12 second combat phase), initial character position (300, 450, 250), and style constraints (fantasy dark tone) from macro scene capsules; sort out character action requirements (e.g., "dragon combat should include wing flapping, claw waving, and flame spray") and special effects requirements (e.g., "snow splashing and airflow effects are required during combat") from user multimodal requirements; retrieve the default action template for "dragon" (e.g., wing flapping frequency 2 times / second) and the "fantasy dark tone" special effects template (e.g., flame color RGB (160, 70, 40)) from the action and special effects base library, initially associate the requirements with the templates, and clarify the generation direction (e.g., "wing flapping" refers to template parameters, and "flame spray" adjusts the color according to user requirements).

[0224] Step 2: Generating Character Motion Sequences

[0225] Taking the red dragon as an example, actions are generated step-by-step according to the timeline stages:

[0226] Pre-battle animation (3-4 seconds, 91-120 frames): Corresponding to the macro scene "Dragon flies into the core area", you can refer to the "Dragon Flight" template in the basic animation library (which can be preset in advance). Combine the initial position (300, 450, 250) and the core area entrance (400, 450, 250) to design the "wing flapping flight" animation: flapping frequency 2 times / second, each flapping angle from -35° to +35°, flight speed 10m / s (it takes 1 second to get from the initial position to the core area entrance). The animation parameters are labeled "Action ID: ACT-Dragon-001, time 91-120 frames, wing flapping angle -35°~+35°, flight speed 10m / s";

[0227] Combat actions (4-10 seconds, 121-300 frames): Corresponding to "combat attack", the action sequence is designed according to user needs such as "waving claws and spraying flames". For example, 4-4.5 seconds (121-135 frames): "Raise claws" action, the claws are raised from under the body to a 60° position in front, the raising speed is 0.5m / s, transitioning from the "flapping wings" action, the transition time is 0.2 seconds;

[0228] 4.5-5 seconds (136-150 frames): "Claw swing attack" action, the claw swings from a 60° position to a 30° position below, the swing speed is 2m / s, accompanied by the "Dragon Roar" action (mouth opening and closing range of 50cm);

[0229] 5-7 seconds (151-210 frames): "Flame Jet" action, mouth remains open, jet lasts for 2 seconds, action parameters are related to the trigger conditions of subsequent special effects;

[0230] Each action is marked with physical constraints (e.g., the claw swing speed should not exceed 2.5m / s to avoid exceeding the character's muscle load);

[0231] Step 3: Generating Scene Effects Sequences

[0232] Based on the character's actions and the constraints of the overall scene, generate corresponding special effects, such as:

[0233] "Airflow Effect": The trigger condition is the "wing flapping" action, and the time interval is synchronized with the wing flapping (3-12 seconds). Refer to the "Fantasy Airflow" template in the effect base library and adjust the style parameters: the airflow color is light purple (RGB(80,80,150), matching dark tones), the particle density is positively correlated with the flapping intensity (density of 300 particles / frame when the flapping frequency is 2 times / second, and density of 100 particles / frame when the flapping frequency is 0.5 times / second), and the spatial range is a 5-8m area around the wings (not exceeding the character size). The effect ID is marked "EFF-Airflow-001", and the associated action ID is "ACT-Dragon-001".

[0234] Step 4: Adjusting the spatiotemporal adaptation of actions and special effects, for example:

[0235] Spatial Adaptation: Check whether the spatiotemporal position of each action and effect is within the macro scene range. For example, the "Flame Jet" is 30m long, with a starting point (350, 450, 270) and an ending point (380, 450, 270), both of which are within the core area (400-700, 300-600, 200-500), thus meeting the spatial constraints. If the "Snow Splash" range exceeds the interaction area, adjust the splash range to 8m to ensure it does not exceed the macro scene partition.

[0236] Timing adaptation: Ensure that the time intervals of actions and effects are synchronized with the macro timeline. For example, the "dragon flies into the core area" action lasts 3-4 seconds, corresponding to the start of the macro combat phase. The "pause" action lasts 12 seconds (360 frames) and is completely synchronized with the macro keyframes. If the "flame jet" effect lasts for 2 seconds (5-7 seconds), it does not exceed the combat phase (3-12 seconds), and the timing is reasonable.

[0237] Dynamic adaptation: Adjust the rhythm of actions and effects to ensure smoothness and naturalness. For example, there is no pause between "raising the claw" (4-4.5 seconds) and "waving the claw" (4.5-5 seconds), with a transition time of 0.2 seconds, and the action is smoothly connected. The "flame spray" effect is triggered 0.1 seconds after the action starts, simulating the natural delay of "opening the mouth and breathing fire" to avoid the effect and action being out of sync.

[0238] Step 5: Integration of Mid-Level Action and Special Effects Capsules:

[0239] Integrate character action sequences with scene effect sequences, label each capsule with a unique identifier (e.g., "Meso-capsule - Dragon Combat - 001"), and clearly define the relationship between each action and effect (e.g., "ACT - Dragon - 003 (Flame Jet Action)" is associated with "EFF - Flame - 001 (Flame Jet Effect)"). Also, add "adaptation notes" (e.g., "Action fanning frequency references the dragon's size, effect color matches the macroscopic dark tone"). After integration, submit to the micro-scale detail texture and lighting generation module, while retaining the generation log (including the basis for action parameter adjustments and effect template modification records) for future tracking and adjustments.

[0240] In this embodiment, the microscale detail texture and lighting generation module operates as follows:

[0241] Step 1: Input data parsing and detailed requirement mapping:

[0242] Extract global illumination (from the upper left to the lower right of the moon, intensity 200 lux), style constraints (fantasy dark tone), and element sizes (dragon length 80m, snow mountain peak 300m) from the macro scene capsule; extract character action parameters and effect parameters from the meso action and effects capsule; retrieve matching templates (dragon scale metal texture template, snow mountain ice crystal translucent texture template, etc.) from the micro detail material library; extract detail requirements (e.g., "dark gold reflection on scales") from user preferences, link and integrate this information, and clarify the detail generation direction of each element (e.g., dragon scales need to have dark gold reflection, ice crystals need to have light purple patterns).

[0243] Step 2: Detailed Texture Generation and Dynamic Adaptation

[0244] Textures are generated in the order of "character elements → scene elements".

[0245] Step 3: Dynamic Light and Shadow Generation and Timing Synchronization:

[0246] Generates in the order of "global lighting → local lighting → dynamic adjustment" to ensure synchronization with the macro and meso levels.

[0247] In this embodiment, the spatiotemporal capsule routing and cross-scale connection module operates as follows:

[0248] Step 1: Capsule Information Parsing and Index Building:

[0249] Extract core parameters at each scale: extract global coordinate system, scene partition coordinates, timeline stages, and keyframes from macro capsules; extract action / effect IDs, local coordinates, time intervals, and size parameters from meso capsules; extract texture / lighting IDs, associated elements, dynamic rules, and spatiotemporal parameters from micro capsules;

[0250] Establish a cross-scale index table: Assign a unique association ID to each element according to the hierarchy of "macro scene partitioning → meso action / effects → micro texture / lighting", ensuring that each meso and micro element can be traced back to the corresponding macro partition.

[0251] Step 2: Spatial coordinate unification and range verification:

[0252] Mesoscopic coordinate mapping: Convert the local coordinates of mesoscopic actions / effects into macroscopic global coordinates. For example, the local coordinates of the mesoscopic "dragon wings flapping" are (-50 to +50, 0, 0) (with the dragon's head as the origin). The position of the dragon in the macroscopic view at 3-3.5 seconds is (350, 450, 260). Then the global coordinates = local coordinates + macroscopic position, that is, the range of the wings flapping is (300-400, 450, 260).

[0253] Macroscopic range verification: Check whether the mapped mesoscopic coordinates fall within the corresponding macroscopic partition. For example, if the wing flapping range (300-400, 450, 260) corresponds to the macroscopic core area (400-700, 300-600, 200-500), and the left boundary 300-400 exceeds the left boundary 400 of the core area, the deviation is 0-100m. The macroscopic position of the dragon needs to be adjusted to (450, 450, 260) so that the wing flapping range becomes (400-500, 450, 260) and falls completely within the core area.

[0254] Micro coordinate binding: Bind the coordinates of micro textures / lighting to the macro coordinates of meso motions / effects - for example, if the micro "scale texture" is associated with the meso wings and the macro range of the wings is (400-500, 450, 260), then the scale texture will only be generated within that coordinate range.

[0255] Step 3: Timeline Synchronization and Trigger Linkage:

[0256] Frame-level time conversion: Converts time intervals of all scales into frame units (30fps);

[0257] Timing deviation verification: Check the synchronization of time intervals between scales. For example, the original time of the micro-scale "snow splash" effect is 4.6-5.1 seconds = 139-154 frames, and the meso-scale "claw waving" action is 4.5-5 seconds = 136-150 frames. The difference between the two is 3 frames. The effect time needs to be adjusted to 4.5-5 seconds = 136-150 frames to be completely synchronized with the action.

[0258] Trigger Link Establishment: Establish time trigger links in the order of "Action → Special Effects → Details → Lighting and Shadow". For example, the mid-level "Dragon Opens Its Mouth" action lasts 4.8 seconds = 144 frames → after a 3-frame delay (0.1 seconds), the mid-level "Flame Jet" is triggered at 147 frames → after another 3-frame delay (0.1 seconds), the micro-level "Flame Lighting" is triggered at 150 frames → the micro-level "Snow Mountain Texture Brightness Change" within the lighting range is initiated at 150 frames, forming a complete time linkage chain. Record the frame number and delay time of each trigger node.

[0259] Step 4: Dynamic Dependency Routing and Consistency Adjustment

[0260] Top-down constraint propagation: Macro constraints are propagated to meso and micro levels. For example, the macro "moonlight direction from the upper left to the lower right" is propagated to the meso "dragon shadow direction from the lower right". The meso shadow parameters (length 160m, blur radius 2m) are then propagated to the micro "shadow texture details" (a 1m wide gradient blur is added to the shadow edge, matching the blur radius).

[0261] Bottom-up issue feedback: Micro-level resource issues are reported to the meso-level for adjustment. For example, if the micro-level issue is "4K scale texture total size 120MB, exceeding the memory limit by 100MB", it is reported to the meso-level, and the resolution of scale texture in non-core areas (wing edges) is reduced to 2K, and the total size is reduced to 80MB to ensure resource compatibility.

[0262] Synchronous adjustment at the same level: synchronize the dynamic parameters of the meso- and micro-levels. For example, in the meso-level, "wing flapping angle changes from -35° to +35°" (frames 91-105), the micro-level "scale stretching ratio changes from 10% to 15%" is adjusted simultaneously. For every 1° change in flapping angle per frame, the scale stretching ratio changes by 0.25%, ensuring that the dynamic rhythm of the action and details is consistent.

[0263] Step 5: Capsule Integration

[0264] Capsule Set Integration: The "macro-partition - meso-action / effects - micro-texture / lighting" units are integrated into a spatiotemporal consistent capsule set. Each unit contains an association index, unified spatiotemporal parameters, dynamic dependency links, and verification flags. For example, the "core area unit" contains: association index (LINK-core area-ACT001-TEX001), unified spatiotemporal parameters (400-700, 300-600, 200-500, 91-360 frames), dependency links (macro core area → meso wing flapping → micro scale stretching), and verification flags (passed).

[0265] In this embodiment, the multi-scale content fusion and preliminary rendering module operates as follows:

[0266] Step 1: Preprocessing and hierarchical loading of spatiotemporal consistency capsule sets:

[0267] After receiving the spatiotemporal consistency capsule set, the data is classified according to "macro scene layer → meso dynamic layer → micro detail layer". For example, the macro layer extracts the spatial range of the core area (400-700, 300-600, 200-500), the global illumination (moonlight 200 lux, direction from upper left to lower right), and the association rules and parameters of the time axis key frames (90 / 360 frames) to ensure that no data is omitted in each layer.

[0268] Layered loading and resource allocation: A "priority loading" strategy is adopted. First, macro scene layer data (such as the basic model of the snow mountain peak and the sky background) is loaded, and 20% of the memory is allocated to store the scene framework. Then, the meso dynamic layer (dragon skeleton model, flame particle effect template) is loaded, and 40% of the memory is allocated to carry dynamic data. Finally, the micro detail layer (scale texture map, light and shadow calculation shader) is loaded, and 30% of the memory is allocated to store detail resources. 10% of the memory is reserved for real-time calculation to avoid resource conflicts caused by disordered loading order (such as loading micro textures first but having no meso model to carry them).

[0269] Step 2: Multi-scale content space fusion and dynamic binding:

[0270] Macro-meta spatial binding: Anchoring meso-level dynamic elements to their corresponding positions in the macro scene. Taking the dragon model as an example, based on the unified parameter "dragon position 450, 450, 260" in the capsule set, the meso-level dragon skeleton model is loaded onto the (450, 450, 260) coordinate point of the macro coordinate system to ensure that the model's center point completely coincides with the parameter.

[0271] Meso-micro detail attachment: Bind the micro texture to the specific parts of the meso dynamic elements. According to the rule of "scale texture associated with wings", the 4K scale texture map is accurately attached to the triangular face model of the meso dragon's wings. The "texture UV animation" technology is used to make the scale texture stretch synchronously with the flapping angle of the wings (-35° to +35°). The stretching ratio is increased from 10% to 15% to match the dynamic movement of the meso.

[0272] Cross-scale spatial conflict detection: Traverse the fused spatial data to check for overlapping or contradictory elements. For example, check for conflicts between the mesoscopic flame jet range (440-470, 450, 270-300) and the microscopic snow mountain texture range (400-700, 300-600, 200-500) in terms of "flames penetrating snow mountains". Use a collision detection algorithm to determine the distance between the flame particles and the snow mountain model. If the distance between the particles and the snow mountain surface is ≤0.5m, automatically adjust the flame jet direction (from horizontal to 10° upward) to ensure the spatial logic is reasonable.

[0273] Step 3: Multi-scale content temporal fusion and temporal synchronization:

[0274] Frame-level unified timing scheduling: Based on the macroscopic time axis (30fps), frame-level execution instructions are assigned to dynamic elements at each scale. For example, the mesoscopic wing flapping action is initiated at frame 91 (3 seconds), and the microscopic scale texture stretching calculation is triggered simultaneously to ensure that the timing is completely synchronized with the capsule set.

[0275] Dynamic rhythm coordination adjustment: Based on the rhythm requirements of the macro timeline stage, optimize the dynamic parameters of the meso and micro levels. For example, in the macro opening stage (0-3 seconds, 0-90 frames), the rhythm is slow, so the speed of the meso dragon flying in is reduced from 10m / s to 5m / s, and the stretching rate of the micro scale texture is reduced simultaneously (from 0.25% / frame to 0.1% / frame).

[0276] Timing conflict correction: If a timing deviation is detected in multi-scale dynamics (such as a 2-frame delay in the start of microscopic flame illumination), it is adjusted through the "frame compensation" mechanism.

[0277] Step 4: Global Illumination Blending and Style Unification Rendering:

[0278] Multi-source superposition calculation: Integrating macroscopic main light source and microscopic local light source to generate a global illumination field.

[0279] Stylized rendering parameter configuration: Based on the macro-level "fantasy dark tone" constraint, set global rendering parameters to ensure a consistent style across the entire scene.

[0280] Layered rendering and layer compositing: A layered rendering strategy of "from far to near" is used to render, thereby forming a complete single frame.

[0281] Step 5: Initial rendering output and output

[0282] Full-frame sequence rendering: The rendering process is executed frame by frame according to the macro timeline (0-15 seconds, 0-450 frames). After each frame is completed, it is stored in PNG format (the alpha channel is retained for subsequent optimization). During the rendering process, resource usage (such as graphics card computing power and memory usage) is monitored in real time. If the computing power usage of a certain frame exceeds 90% (such as the peak of the flame jet at 180 frames), the rendering precision of non-core elements in that frame is automatically reduced (such as reducing the texture of distant rocks from 2K to 1K) to ensure smooth rendering (stable frame rate of 30fps).

[0283] The qualified PNG frame sequence (0-450 frames) is integrated into a "preliminary rendered video frame sequence", a unique identifier is marked on the sequence, and the frame sequence is submitted to the physical-semantic joint constraint verification module for subsequent verification and optimization.

[0284] In this embodiment, the physical-semantic joint constraint verification module includes:

[0285] A joint constraint field construction module is used to generate a joint constraint field model based on the multi-scale spatiotemporal capsule modeling information and an intent graph with priority and constraint conditions.

[0286] The simulated annealing optimization module is used to optimize the joint constraint field model using a simulated annealing algorithm, thereby obtaining the optimized generation parameters.

[0287] A real-time physics engine verification module is used to verify the optimized generation parameters, thereby generating physics violation markers and correction suggestions.

[0288] A real-time semantic discriminator verification module is used to perform semantic scoring and evaluation on the optimized generated parameters, thereby obtaining semantic violation tags and correction suggestions.

[0289] A parameter adjustment module is provided, which is used to adjust the optimized generation parameters according to the physical violation markers and correction suggestions and the semantic violation markers and correction suggestions, so as to obtain the adjusted generation parameters.

[0290] A re-rendering module is used to adjust the multi-scale spatiotemporal capsule modeling information according to the adjusted generation parameters, thereby obtaining verified modeling information.

[0291] In this embodiment, the physical-semantic joint constraint verification module can perform the following operations:

[0292] Physical constraint subfield construction: Based on physical verification parameters and intention constraints, a physical law constraint model is built. Taking the "Dragon Snow Mountain Battle" scenario as an example, a gravity constraint (gravitational acceleration 9.8 m / s²) is constructed. 2 Allows fantasy scenes to be adjusted to 8m / s 2 Constraints such as collision constraints (impact force when the dragon collides with the snow mountain ≤ snow mountain compressive strength 30MPa);

[0293] Semantic constraint subfield construction: Based on semantic verification parameters and intent constraints, a semantic logic constraint model is built. For example, a style constraint subfield is constructed (all element hues must be derived from RGB(30,40,80), and realistic metallic textures are prohibited) and an association constraint subfield (the fire effect must always deviate from the coordinates of the dragon's mouth by ≤5m, and the snow mountain texture must maintain the fantastic mixed texture of ice crystals and rocks). Each semantic subfield establishes a matching standard through semantic vector encoding (such as using the CLIP model to encode "fantasy dark-toned fire" into a 768-dimensional vector).

[0294] By linking physical constraints with semantic constraints to form a unified joint constraint field, such as in the "collision of flames and snow mountains" scenario, the physical constraint requires that "the temperature of the flames causes the snow mountains to melt," while the semantic constraint requires that "the melting effect must conform to the fantasy style (such as producing light purple steam rather than realistic white steam)." In the joint constraint field, it is clearly stated that "the melting rate of the snow mountains must satisfy the physical temperature difference law (physical subfield), and the steam color must be RGB(100,100,180) (semantic subfield)." At the same time, the weights of the two are defined (physical constraint weight 0.4, semantic constraint weight 0.6, because the semantic priority is slightly higher in fantasy scenarios) to avoid validation bias caused by single-dimensional constraints.

[0295] Through simulated annealing optimization and parameter iterative adjustment:

[0296] Initial parameter deviation calculation: Substitute the modeling parameters corresponding to the initial rendered frame sequence into the joint constraint field to calculate the parameter deviation value. For example, the calculated lift value of the dragon's wings flapping is: 0.5 × 1.2 × 5. 2 ×1200×1.2=21600N, which is far below the requirement of 64000N. The physical deviation value is (64000-21600) / 64000=0.6625 (66.25%). Record the parameters and corresponding frame numbers of all deviations exceeding the threshold (deviation>20%) (e.g., the lift deviation corresponds to the wing flapping stage of frames 91-105).

[0297] Simulated annealing algorithm optimization: Aiming to minimize the joint deviation value, simulated annealing optimization is initiated—setting the initial temperature T = 100 (to control the probability of accepting deviation), with 100 iterations. In each iteration, 1-2 out-of-tolerance parameters are adjusted (e.g., in the first iteration, the wing flapping speed is increased from 5 m / s to 8 m / s, and the lift is recalculated as 0.5 × 1.2 × 8). 2 ×1200×1.2=55296N, the deviation is reduced to (64000-55296 / 64000=0.136(13.6%), which meets the threshold; in the second iteration, the coordinates of the flame jet starting point are adjusted 3m towards the dragon's mouth, and the deviation is reduced to 5m, which meets the semantic constraints); after each iteration, the temperature T=0.9×T is reduced until T<1 or the deviation value<5%, the iteration is stopped, and the optimized generation parameters are output (such as the wing flapping speed of 8m / s and the adjusted value of the flame starting point coordinates).

[0298] Real-time physics engine verification: Input the optimized generation parameters into the real-time physics engine (such as NVIDIA PhysX) and perform physical simulation verification frame by frame on the initial rendered frame sequence. For frames where the physical parameters exceed the threshold (such as a frame where the flame temperature suddenly rises to 1500℃, exceeding the allowable range of 1200℃), mark them as "physical violation frames" and record the violation type (temperature exceeding the limit), frame number (such as frame 180), violation parameter value (1500℃), and correction suggestion (revert the temperature back to 1100℃).

[0299] Real-time semantic discriminator verification: A dual-model approach of "CLIP semantic matching + style discriminator" is used for semantic verification. For example, the rendering frames corresponding to the optimized parameters (such as 150 frames after adjusting the flame color) are input into the CLIP model to calculate the similarity with the semantic vector of "fantasy dark flame" (e.g., similarity 0.85 ≥ threshold 0.7, which meets the semantics); the frames are input into the style discriminator (pre-trained fantasy / realistic classification model) to judge the style matching degree of the elements (e.g., the style matching degree of the dark gold reflection of the dragon scales is 0.92, which meets the requirements). For frames with semantic mismatch (e.g., a bright area of ​​RGB(220,220,220) appears in a snow mountain in a certain frame), they are marked as "semantic violation frames", and the violation type (hue exceeding the standard), frame number (e.g., frame 250), violation parameters (RGB value) and correction suggestions (adjust the hue to RGB(180,180,200)) are recorded.

[0300] Generate parameters, adjust and re-render to verify:

[0301] Batch adjustment of violation parameters: Targeted corrections are made to the optimized generated parameters. For example, for physical violation parameters, such as reducing the flame temperature from 1500℃ to 1100℃ in frame 180, the melting rate in the thermodynamic constraints is adjusted simultaneously (from 0.1m / s to 0.08m / s); for semantic violation parameters, such as adjusting the RGB value of the snow mountain area in frame 250 from (220,220,220) to (180,180,200), the illumination reflectance coefficient of the area is corrected simultaneously (from 0.8 to 0.6 to avoid excessive brightness); After adjustment, a "final correction parameter set" is generated, and the violation report ID corresponding to each adjustment is marked to ensure traceability.

[0302] Partial re-rendering: Only frames with violations are partially rendered (instead of re-rendering the entire sequence, saving resources). For example, for frames 180-185 (flame temperature violation range) and 250-255 (snow mountain tone violation range), the corrected parameters are loaded, and layered rendering is re-executed (first rendering the background layer, then overlaying the dynamic layer and detail layer) to generate corrected PNG frames. During the rendering process, it is monitored whether the parameters still have violations (e.g., the temperature in frame 180 is 1100℃ after correction, and it needs to be checked in real time whether it meets the constraints). If there are still violations, the parameters are adjusted again (e.g., the temperature is reduced to 1050℃) until there are no violations in the locally rendered frames.

[0303] Final verification and modeling information output: The locally corrected frames are integrated with the original qualified frames into a complete frame sequence, and joint constraint verification is performed again (20 frames are randomly selected, including 10 original qualified frames and 10 corrected frames). The physical parameter deviation is checked to be ≤20%, the semantic matching degree is ≥0.7, and there are no new violations. After the verification is passed, the corrected generated parameters are integrated with the complete frame sequence to form "verified modeling information". The verification time, number of violation corrections, and final parameter version (e.g., V2.0, corresponding to two parameter adjustments) are marked and submitted to the spatiotemporal consistency optimization and content fusion module. At the same time, a "verification summary report" is generated, which includes violation type statistics (e.g., 2 physical violations and 1 semantic violation) and correction effect (e.g., temperature deviation decreased from 66.25% to 8.3%) for reference by subsequent modules.

[0304] In this embodiment, the spatiotemporal consistency optimization and content fusion module includes:

[0305] A spatiotemporal consistency detection and marking module is used to perform spatiotemporal consistency detection on the verified modeling information to obtain spatiotemporal inconsistency markers, which include the frame number to be optimized and the region coordinates.

[0306] An optical flow prediction and motion trajectory smoothing module is used to process the spatiotemporal inconsistency marker to form a smoothed motion vector field, which includes corrected inter-frame motion parameters.

[0307] A timing discriminator verification and coherence optimization module is used to generate a sequence of video frames that pass timing coherence verification based on a smoothed motion vector field.

[0308] Multi-scale content fusion and final rendering are used to generate a spatiotemporally consistent video frame sequence based on the video frame sequence that has passed the temporal coherence verification, macro scene capsules, meso action and special effects capsules, and micro detail and light and shadow capsules.

[0309] In this embodiment, the spatiotemporal consistency optimization and content fusion module can perform the following operations:

[0310] The system receives validated modeling information from the physical-semantic joint constraint verification module and archives it into three categories: "frame sequence data → parameter data → verification records." It focuses on extracting the coordinate trajectories and motion parameters (such as the inter-frame variation of wing flapping angles) of dynamic elements (dragon, flames, snow splashes) in the frame sequence, as well as the original spatiotemporal constraints of macroscopic / mesoscopic / microscopic capsules (such as macroscopic core area coordinates and mesoscopic action time intervals), ensuring no critical dynamic data is missing (e.g., if there is no inter-frame motion trajectory of flame particles, it is returned for supplementation). The complete frame sequence is then formatted and its coordinates are calibrated.

[0311] Spatiotemporal consistency detection and abnormal region marking:

[0312] Spatial consistency detection: The method of "dynamic element coordinate trajectory tracking + collision conflict judgment" is adopted to detect whether dynamic elements conform to macroscopic spatial constraints and inter-frame spatial continuity frame by frame. For example, for the dragon model, the center coordinates of each frame are extracted (such as frame 91 (400,450,260) and frame 92 (402,450,261)), and the inter-frame coordinate offset is calculated (such as x-axis offset of 2m and z-axis offset of 1m). If the offset of a certain frame suddenly increases (such as the dragon coordinates in frame 95 jumping from (410,450,263) to (450,450,263), an offset of 40m, far exceeding the 5m / frame offset caused by normal fan-out), it is marked as a "spatial abnormal frame".

[0313] Temporal consistency detection: The temporal coherence of dynamic elements is detected by "inter-frame motion parameter continuity analysis + temporal constraint matching". For example, for the flapping action of a dragon's wings, the change in flapping angle in each frame is calculated (e.g., 91 frame -35°, 92 frame -32°, 93 frame -29°, normal change is 3° / frame). If the angle suddenly changes from -20° to +10° in 98 frame (change is 30° / frame, far exceeding the normal range), it is marked as "abnormal temporal motion mutation".

[0314] All detected anomalies are organized according to the structure of "anomaly type (spatial / temporal) → frame number range → anomaly region coordinates → preliminary judgment of anomaly cause" to generate a "spatiotemporal anomaly report".

[0315] Optical flow prediction and motion trajectory smoothing: For abrupt changes in temporal motion (such as a sudden change in wing angle in frame 98), an optical flow prediction model (such as FlowNet2.0) is used to calculate the motion vector field of the frames before and after the abrupt frame (frames 97 and 99). For example, the wing flapping angle in frame 97 is -23° and in frame 99 it is -17°. The optical flow model predicts that the reasonable angle in frame 98 should be -20° (3° change between frames), generating the corrected motion parameters (angle -20°, flapping speed 8m / s). The resulting smoothed motion vector field contains the corrected motion parameters (angle, speed, coordinates) of the dynamic elements in each frame.

[0316] Temporal discriminator verification and coherence optimization: The corrected frame sequence is input into a pre-trained temporal discriminator (built based on LSTM+CNN to judge the dynamic coherence between frames). The discriminator scores the motion parameters and temporal logic of each frame (0-1 points, 1 point is complete coherence). For example, if the scores of the corrected frames 95, 98, and 182 are all ≥0.9 (the original abnormal frame scores ≤0.3), it means that the temporal coherence meets the standard. For frames that still do not meet the standard (such as the flame in frame 211 being overdue, with a score of 0.6), the flame particle layer of that frame is directly deleted to ensure that the temporal sequence is completely matched with the mesoscopic constraints (frames 151-210). At the same time, a gradient effect is added to the start / end transition frames of dynamic elements (such as the flame starting in frame 151 and ending in frame 210) to avoid the temporal inconsistency of "sudden appearance / disappearance". Finally, a "video frame sequence that passes the temporal coherence verification" is generated.

[0317] Multi-scale content layering and fusion: Deep fusion is performed on verified frame sequences in the order of "macro scene layer → meso dynamic layer → micro detail layer → lighting and shadow effect layer." For example, the macro scene layer loads the basic models of the snow-capped mountain peak and the moon in the sky, rendering the background texture according to macro style constraints (dark RGB(30,40,80)); the meso dynamic layer overlays the corrected dragon, flame, and snow splash models, ensuring that the coordinates of dynamic elements match the macro space (e.g., the dragon always moves in the core area); the micro detail layer integrates scale textures, ice crystal textures, and other elements. The detailed textures of the flame particles are accurately attached to the corresponding models (e.g., the scale texture stretches synchronously with the flapping angle of the wings); the lighting effect layer integrates the macroscopic moonlight (200 lux) and the microscopic flame lighting (1100℃ corresponds to 200 lux local light), calculates the global lighting distribution for each frame (e.g., the brightness of the flame area is 400 lux, and the background area is 180 lux), and adds dynamic shadows (e.g., the dragon's shadow shifts synchronously with the position change between frames, and the shadow length ratio is maintained at 1:2), ensuring that the content at all scales is completely integrated in space and style, without any sense of layering or separation.

[0318] For the merged frame sequence, detail enhancement and style unification adjustments are performed. The merged and optimized frame sequence is rendered as a 1920×1080 resolution MP4 format frame sequence (30fps). Spatiotemporal consistency (e.g., smooth dynamic element coordinate trajectory, timing and mesoscopic constraints matching), style consistency (tone and lighting conform to macroscopic constraints), and detail integrity (clear microscopic textures and reasonable special effects) are checked frame by frame. 30 frames are randomly selected (covering the opening, battle, and ending stages), and 3 testers are invited to give subjective coherence scores (1-5 points, 5 points being the best). An average score of ≥4.5 points is considered qualified. After passing, a "Spatiotemporal Consistency Optimized Video Frame Sequence" is generated, marked with a unique sequence identifier (e.g., "Optimized Frame Sequence - Dragon Snow Mountain - 001") and optimization records (e.g., correcting 3 spatial anomalies and 2 temporal anomalies). It is submitted to the dynamic bitrate video synthesis and output module, and an "Optimization Summary Report" is generated at the same time, including a comparison chart of the frame sequence before and after optimization and records of key parameter adjustments (e.g., wing flapping angle correction value) for subsequent modules to trace.

[0319] In this embodiment, the dynamic bitrate video synthesis and output module includes:

[0320] The content importance analysis and region segmentation module is used to identify the video frame sequence after spatiotemporal consistency optimization, thereby obtaining a content importance map, which includes the importance score of each region in each frame;

[0321] A dynamic bitrate allocation strategy formulation module is used to dynamically allocate bitrate based on the content importance graph, thereby generating a dynamic bitrate allocation table, which includes the bitrate values ​​of each region in each frame.

[0322] Video encoding and dynamic bitrate compression are used to encode and compress the spatiotemporally consistent optimized video frame sequence according to the dynamic bitrate allocation table, thereby obtaining a dynamically bitrate compressed video stream.

[0323] In this embodiment, the dynamic bitrate video synthesis and output module can perform the following processing:

[0324] The system receives video frame sequences optimized for spatiotemporal consistency, extracts the basic attributes of the frame sequences (resolution, frame rate, and RGB color space), and obtains the spatiotemporal distribution information of key dynamic elements from the optimization report (e.g., the core area of ​​the dragon battle is concentrated in frame pixel coordinates (800-1100, 400-700), and the fire effect area is concentrated in (850-950, 450-550), thus clarifying the key and non-key areas for subsequent bitrate allocation.

[0325] The PNG format frame sequences are batch converted to the YUV420 color space. The converted frame sequences are then subjected to noise removal processing to eliminate any residual tiny pixel noise in the optimized frame sequences, thus avoiding wasted coding bitrate due to noise. At the same time, the frame sequences are divided into 30 coding units (each 15 frames (0.5 seconds) as a "coding unit") to facilitate subsequent analysis of content importance and bitrate allocation by unit.

[0326] For each frame within a coding unit, a region importance score (0-10 points, with 10 points being the highest importance) is calculated from three dimensions: "visual salience, motion intensity, and semantic relevance." For example, visual salience uses a salience map detection algorithm (such as the Itti-Koch model) to calculate the visual attractiveness of each region within the frame. The dragon battle core region (800-1100, 400-700) scores 9-10 points due to its high color contrast (red dragon against a dark background) and clear outline; the flame effect region (850-950, 450-550) scores 8-9 points due to its dynamic luminescence characteristics.

[0327] Motion intensity: The pixel change rate of each region is calculated by the inter-frame difference algorithm. The inter-frame pixel change rate of the flame effect area reaches 60%-80% (particle dynamic motion), scoring 8-9 points; the change rate of the dragon wing flapping area (880-920, 480-520) is 40%-60%, scoring 7-8 points.

[0328] Semantic relevance: Combining the priority intent map, the core intent "Dragon battle" related area (dragon body, battle interaction area) scored 10 points, and the background area related to the auxiliary intent "snow mountain scene" scored 3-4 points.

[0329] The final importance score for each region is calculated as "visual salience × 40% + motion intensity × 30% + semantic relevance × 30%", rounded to one decimal place.

[0330] Each frame is divided into grid regions of 16×16 pixels (120×67.5≈8100 grid regions in 1920×1080 resolution). Based on the importance score, each grid region is classified as follows: a score ≥8 is marked as "core important region", a score 5≤score<8 is marked as "medium important region", and a score<5 is marked as "low important region". A "content importance map" is generated for each frame, and the coordinates (e.g. (800-816, 400-416)), importance score (e.g., 9.2), and region type (core / medium / low important) of each region are marked. For example, "Frame 150 (flame jet peak frame): Block region (850-866, 450-466) score 9.1 (core important region), block region (200-216, 100-116) score 2.3 (low important region)".

[0331] The importance maps of adjacent frames are compared. If the importance score of the same spatial region fluctuates by more than 2 points (e.g., a region in frame 150 scores 9.1 points, but drops sharply to 6.8 points in frame 151), the score is smoothed by weighted averaging (60% weight of the previous frame and 40% weight of the current frame) (adjusted score = 9.1 × 0.6 + 6.8 × 0.4 = 8.3 points) to avoid excessive bitrate fluctuations caused by frequent switching of region types in adjacent frames. At the same time, "dynamic important regions" (e.g., the trajectory of the flame effect area as it moves with the spray action) are marked to maintain the bitrate continuity of the region in subsequent bitrate allocation.

[0332] Total bitrate budget and interval division: Calculate the total bitrate budget based on user output requirements (e.g., total video length 15 seconds, target file size ≤ 100MB). For example, 100MB = 800Mbps, the total bitrate budget for a 15-second video is 800Mbps / 15≈53.3Mbps. Considering encoding loss and reserving 10% redundancy, the actual usable bitrate budget is 48Mbps (average bitrate per frame at 30fps ≈ 48Mbps / 30≈ 1.6Mbps). Set bitrate intervals based on the importance of the regions: core important regions: 7-8Mbps (per frame per region), medium important regions: 4-5Mbps, low important regions: 1-2Mbps, ensuring sufficient bitrate in high important regions and that the total bitrate does not exceed the budget.

[0333] Frame-level dynamic bitrate allocation table generation: Combining the content importance map and bitrate range of each frame, a specific bitrate value is assigned to each region block of each frame. For example, in frame 150 (the peak frame of the flame jet): 7.8Mbps is allocated to each block of the core important region (such as flames and the dragon's head), 4.5Mbps is allocated to each block of the medium important region (non-action parts of the dragon's body), and 1.2Mbps is allocated to each block of the low important region (the distant snow mountain). The total bitrate of each frame is calculated (the sum of the bitrates of each region block). If the total bitrate of a frame exceeds the budget (e.g., the original total bitrate of frame 150 is 2.1Mbps > 1.6Mbps), the bitrate is reduced in the order of "low important region → medium important region" until the total bitrate of the frame is ≤ 1.6Mbps. A "dynamic bitrate allocation table" is generated by organizing the frames, including the frame number, region block coordinates, allocated bitrate, region type, and adjustment record (e.g., "Frame 150: the bitrate of block (200-216, 100-116) is adjusted from 1.2Mbps to 1.0Mbps").

[0334] Differentiated coding parameters are configured for regions of different importance. For example, core important regions use "high bitrate + low compression ratio" parameters (e.g., quantization parameter QP=20 for H.265 encoding, and omnidirectional prediction in intra-frame prediction mode) to preserve detailed textures (e.g., dragon scales, flame particles); medium important regions use "medium bitrate + medium compression ratio" parameters (QP=26, and bidirectional prediction mode); low important regions use "low bitrate + high compression ratio" parameters (QP=32, enabling inter-frame reference compression to reduce redundant data); at the same time, "B-frame optimization" (inserting 1 B-frame every 5 frames) is enabled for core regions with drastic dynamic changes (e.g., flames) to improve coding efficiency while avoiding loss of detail.

[0335] Dynamic bitrate encoding and compression execution: Load the dynamic bitrate allocation table and the preprocessed YUV frame sequence, and call the H.265 / HEVC encoder (such as x265) to perform frame-by-frame encoding. For example, during the encoding process, read the allocated bitrate of each region in each frame in real time. Configure the encoder parameters for the core important region (such as the flame area in frame 150) at a bitrate of 7.8 Mbps to ensure that the bitrate deviation of the video stream in this region after encoding is ≤5% (e.g., target 7.8 Mbps, actual 7.5-8.1 Mbps); compress the less important region (such as static snow mountain) at a bitrate of 1.0 Mbps, with an allowable deviation of ≤10%; monitor the CPU / GPU resource usage of the encoder during the encoding process (e.g., CPU usage ≤80%, GPU computing power ≤70%). If the encoding time of a certain frame exceeds 33ms (30fps single frame time limit), temporarily reduce the bitrate of the less important region of that frame (e.g., from 1.0 Mbps to 0.8 Mbps) to ensure smooth encoding without stuttering; after encoding is completed, generate the dynamically bitrate compressed video stream (initially in TS format).

[0336] In this embodiment, the hierarchical intent parsing and dynamic graph construction module further includes:

[0337] The intent conflict detection and self-repair module detects and repairs conflicts based on the dynamically adjusted intent graph, thereby obtaining a repaired intent graph; the intent priority sorting and constraint generation module processes the repaired intent graph.

[0338] In this embodiment, the intent contradiction detection and self-repair module includes:

[0339] An intent conflict detection module is used to detect intent conflicts using the ICOF formula, thereby obtaining conflict markers.

[0340] An intent conflict repair module is used to repair intent conflicts based on the conflict markers, thereby obtaining a repaired intent graph.

[0341] In this embodiment, the ICOF formula is as follows:

[0342]

[0343] Among them, I i ,I j Let v(I) be two intent nodes in the dynamic graph (e.g., "Dragon Battle" and "Clear Sky"). v(I) is the semantic vector of intent I, generated by a pre-trained language model (e.g., CLIP-Text), containing keywords, extended words, and weights. α and β are the weight coefficients of the semantic and physical contradictions, satisfying α + β = 1 (determined through dynamic programming optimization). C k (I) represents the required value of physical constraint k for intention I (e.g., the "dragon mass" requirement is 500 kg). R k This represents the actual resource limit for the physical constraint k (e.g., "maximum load of the physics engine" is 800 kg).

[0344] In this embodiment, all intent pairs (I) in the graph are traversed. i ,I j ), calculate the intention contradiction correlation optimization function, if ICOF(I i ,I j )>θ (the threshold θ is determined through user preference learning, such as θ = 0.5), labeling intent pairs I i ,I j It is a contradiction.

[0345] This application replaces the traditional expert rule base with a self-designed ICOF formula, which can adaptively combine any intent (such as "dragon breathing fire" + "clear sky"). It also considers semantic and physical contradictions to avoid new contradictions caused by local repairs (e.g., repairing only semantic contradictions may violate physical laws). Through vector computation and GAN generation, the repair speed is 3-5 times faster than traditional rule matching (experimental data).

[0346] This application has the following advantages:

[0347] Traditional methods rely on static rule bases to parse intent, failing to handle emerging combinations (such as the implicit contradiction of "dragon breathing fire + clear sky"). This method dynamically fuses multimodal features (text, visual, and audio) through a residual adjustment mechanism, and automatically adjusts intent dependencies by combining dynamic priority ranking to construct a scalable intent graph. For example, when a user inputs "fantasy-style dragon battle," the system can automatically identify the contradiction between "breathing fire requires clouds" and "cloudless sky," prioritizing the resolution of the core conflict (such as changing "clear sky" to "partly cloudy"), avoiding the limitations of hard-coded rules.

[0348] Traditional methods employ sequential verification (physical first, then semantic), which is prone to introducing new problems due to localized fixes (such as violating physical laws after fixing semantic contradictions). This method innovatively proposes the Intent Contradiction Relevance Optimization Function (ICOF), which combines semantic contradiction terms (detected through semantic vector cosine similarity) and physical contradiction terms (detected through the difference between physical constraint requirements and resource limits) to dynamically detect and fix intent contradictions. For example, when "dragon mass + fire-breathing energy" exceeds the physics engine's load, the ICOF formula can simultaneously adjust "dragon mass" and "fire-breathing energy," ensuring a balance between physical feasibility and semantic expressiveness.

[0349] Traditional methods generate scenes, actions, and details step by step and then stitch them together, which can easily lead to spatiotemporal misalignment (such as the dragon's position abruptly changing between adjacent frames). This method identifies anomalous regions through spatiotemporal inconsistency detection (optical flow prediction + temporal discriminator), combines Bézier curve fitting to smooth motion trajectories, and dynamically aligns macroscopic scenes, mesoscopic actions, and microscopic details (such as the dragon's position aligning with the space of the fire effects) through a multi-scale content fusion processor. For example, when generating "dragon wings flapping," the system can automatically adjust the wing flapping frequency to ensure smooth action and aerodynamic compliance.

[0350] Traditional methods use fixed bitrate compression, which easily leads to the loss of detail in core areas (such as a dragon breathing fire) due to low bitrate, while background areas (such as static mountains) waste file size. This method generates a region importance map through content importance analysis (visual salience + motion intensity), combined with a dynamic bitrate allocation strategy (high bitrate for core areas, low bitrate for background areas), and supports multi-format / multi-platform adaptation (such as TikTok portrait mode, YouTube landscape mode). For example, when generating a "dragon breathing fire" image, the system can automatically allocate a high bitrate of 8Mbps to the fire effect and a low bitrate of 2Mbps to the background mountains, balancing image quality and file size.

[0351] This application also provides an automated video generation method based on AIAgentic multimodal cooperative control, the automated video generation method based on AIAgentic multimodal cooperative control comprising:

[0352] Obtain multimodal generation requirements information provided by the user;

[0353] The multimodal generation requirement information is parsed to generate an intent graph with priorities and constraints.

[0354] Multi-scale spatiotemporal capsule modeling information is generated based on the dynamically evolving intent map;

[0355] The modeling information of the multi-scale spatiotemporal capsule is validated to obtain validated modeling information;

[0356] The validated modeling information is subjected to spatiotemporal consistency optimization and content fusion to obtain a spatiotemporally consistent optimized video frame sequence;

[0357] Video files are generated based on the video frame sequence optimized for spatiotemporal consistency.

[0358] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. An automated video generation system with AI Agentic multimodal collaborative control, characterized in that, The AIAgentic multimodal collaborative control automated video generation system includes: A requirement acquisition module, which is used to acquire multimodal generation requirement information provided by the user; A hierarchical intent parsing and dynamic graph construction module is used to parse the multimodal generation requirement information, thereby generating an intent graph with priority and constraints. A multi-scale spatiotemporal capsule scene generation module is used to generate multi-scale spatiotemporal capsule modeling information based on a dynamically evolving intent graph. The physical-semantic joint constraint verification module is used to verify the multi-scale spatiotemporal capsule modeling information, thereby obtaining the verified modeling information. The spatiotemporal consistency optimization and content fusion module is used to perform spatiotemporal consistency optimization and content fusion on the verified modeling information, thereby obtaining a spatiotemporally consistent optimized video frame sequence; A dynamic bitrate video synthesis and output module, which is used to generate video files based on a video frame sequence optimized for spatiotemporal consistency; The hierarchical intent parsing and dynamic graph construction module includes: An intent classification module is used to generate a core intent set based on the multimodal generation requirement information, the core intent set including at least one core intent information; A confidence scoring module is used to score the confidence of each core intent information. The sub-intent recursive decomposition module is used to decompose each core intent information separately to obtain executable sub-intents. A core intent information includes at least one sub-intent. A hierarchical intent graph construction module is used to generate and construct a hierarchical intent graph based on core intent information and sub-intents; wherein, the nodes of the constructed hierarchical intent graph include core intent information and sub-intents, and the edges represent dependencies; A dynamic intent graph evolution and adjustment module is used to generate a dynamically adjusted intent graph based on a hierarchical intent graph. An intent priority sorting and constraint generation module is used to adjust the priority and constraints of each node in the dynamically adjusted intent graph, thereby generating an intent graph with priorities and constraints. The hierarchical intent parsing and dynamic graph construction module further includes: The intent conflict detection and self-repair module detects and repairs conflicts based on the dynamically adjusted intent graph, thereby obtaining a repaired intent graph; the intent priority sorting and constraint generation module processes the repaired intent graph. The intent contradiction detection and self-repair module includes: An intent conflict detection module is used to detect intent conflicts using the ICOF formula, thereby obtaining conflict markers. An intent conflict repair module is used to repair intent conflicts based on the conflict markers, thereby obtaining a repaired intent graph. The ICOF formula is as follows: ; in, These are two intent nodes in the dynamic graph; α is the semantic vector of intent I, generated by a pre-trained language model, containing keywords, extended words, and weights; α and β are the weight coefficients of semantic contradictions and physical contradictions, satisfying α+β=1; Let I be the requirement value of physical constraint k. The actual resource ceiling for physical constraint k; Traverse all intention pairs in the graph ( ),calculate ,like The marking intent is to As a contradictory pair, the threshold θ is determined through user preference learning.

2. The automated video generation system with AI Agentic multimodal collaborative control as described in claim 1, characterized in that, The requirement acquisition module includes: A text information acquisition module, which is used to acquire text information provided by the user; A visual information acquisition module, wherein the visual information acquisition module is used to acquire visual information provided by the user; An audio information acquisition module, wherein the audio information acquisition module is used to acquire audio information provided by the user; The text processing module is used to extract text features from the text information and expand the text features to obtain text modal features, which include text semantic vectors, expanded keyword lists and corresponding semantic weights. A visual processing module is used to extract features from visual information to obtain visual modal features, which include object-level visual feature vectors and visual dynamic trajectory feature vectors. An audio processing module is used to extract the audio information to obtain audio modal features, which include acoustic feature vectors and emotion tags. The dynamic modal weight allocation module is used to allocate weights to text modal features, visual modal features and audio modal features respectively, so as to obtain a multimodal feature vector after dynamic weight allocation. A cross-modal residual learning and alignment module is used to perform residual adjustment on the multimodal feature vectors after dynamic weight allocation, thereby obtaining residual-adjusted multimodal feature vectors, which serve as the required information.

3. The automated video generation system with AI Agentic multimodal collaborative control as described in claim 2, characterized in that, The multi-scale spatiotemporal capsule scene generation module includes: A macro-scale scene layout generation module is used to generate macro-scale scene capsules based on the intent graph with priority and constraints. A mesoscale character motion and special effects generation module, which is used to generate mesoscale motion and special effects capsules based on the macroscale scene capsule and the requirement information; The microscale detail texture and lighting generation module is used to generate microscale detail and lighting capsules based on macroscale scene capsules, mesoscale action and special effects capsules, and requirement information. The spatiotemporal capsule routing and cross-scale connection module is used to merge macroscopic scene capsules, mesoscopic action and special effects capsules, and microscopic detail and light and shadow capsules into a spatiotemporal consistent capsule set. The multi-scale content fusion and preliminary rendering module is used to perform capsule fusion and preliminary rendering on the spatiotemporal consistency capsule set to obtain a preliminary rendered video frame sequence, which serves as multi-scale spatiotemporal capsule modeling information.

4. The automated video generation system with AI Agentic multimodal collaborative control as described in claim 3, characterized in that, The physical-semantic joint constraint verification module includes: A joint constraint field construction module is used to generate a joint constraint field model based on the multi-scale spatiotemporal capsule modeling information and an intent graph with priority and constraint conditions. The simulated annealing optimization module is used to optimize the joint constraint field model using a simulated annealing algorithm, thereby obtaining the optimized generation parameters. A real-time physics engine verification module is used to verify the optimized generation parameters, thereby generating physics violation markers and correction suggestions. A real-time semantic discriminator verification module is used to perform semantic scoring and evaluation on the optimized generated parameters, thereby obtaining semantic violation tags and correction suggestions. A parameter adjustment module is provided, which is used to adjust the optimized generation parameters according to the physical violation markers and correction suggestions and the semantic violation markers and correction suggestions, so as to obtain the adjusted generation parameters. A re-rendering module is used to adjust the multi-scale spatiotemporal capsule modeling information according to the adjusted generation parameters, thereby obtaining verified modeling information.

5. The automated video generation system with AI Agentic multimodal collaborative control as described in claim 4, characterized in that, The spatiotemporal consistency optimization and content fusion module includes: A spatiotemporal consistency detection and marking module is used to perform spatiotemporal consistency detection on the verified modeling information to obtain spatiotemporal inconsistency markers, which include the frame number to be optimized and the region coordinates. An optical flow prediction and motion trajectory smoothing module is used to process the spatiotemporal inconsistency marker to form a smoothed motion vector field, which includes corrected inter-frame motion parameters. A timing discriminator verification and coherence optimization module is used to generate a sequence of video frames that pass timing coherence verification based on a smoothed motion vector field. Multi-scale content fusion and final rendering are used to generate a spatiotemporally consistent video frame sequence based on the video frame sequence that has passed the temporal coherence verification, macro scene capsules, meso action and special effects capsules, and micro detail and light and shadow capsules.

6. The automated video generation system with AI Agentic multimodal collaborative control as described in claim 5, characterized in that, The dynamic bitrate video synthesis and output module includes: The content importance analysis and region segmentation module is used to identify the video frame sequence after spatiotemporal consistency optimization, thereby obtaining a content importance map, which includes the importance score of each region in each frame; A dynamic bitrate allocation strategy formulation module is used to dynamically allocate bitrate based on the content importance graph, thereby generating a dynamic bitrate allocation table, which includes the bitrate values ​​of each region in each frame. Video encoding and dynamic bitrate compression are used to encode and compress the spatiotemporally consistent optimized video frame sequence according to the dynamic bitrate allocation table, thereby obtaining a dynamically bitrate compressed video file.

7. An automated video generation method based on AI Agentic multimodal collaborative control, applied to the automated video generation system based on AI Agentic multimodal collaborative control as described in any one of claims 1 to 6, characterized in that, The automated video generation method of AIAgentic multimodal collaborative control includes: Obtain multimodal generation requirements information provided by the user; The multimodal generation requirement information is parsed to generate an intent graph with priorities and constraints. Generate multi-scale spatiotemporal capsule modeling information based on dynamically evolving intent maps; The modeling information of the multi-scale spatiotemporal capsule is validated to obtain validated modeling information; The validated modeling information is subjected to spatiotemporal consistency optimization and content fusion to obtain a spatiotemporally consistent optimized video frame sequence; Video files are generated based on the video frame sequence optimized for spatiotemporal consistency.

Images