Video film generation system based on AI

Abstract

The invention belongs to the technical field of multimedia processing, and discloses a video film generation system based on AI. Through cooperative work of a script analysis module, a scene transition analysis module, a role dynamic coding module, a video frame generation module, a time sequence consistency evaluation module and a frame optimization repair module, intelligent conversion from a script text to a complete video movie is realized. According to the method, the core problems of script understanding deviation, stiff scene conversion, unnatural role expression, inconsistent inter-frame quality and the like in traditional video production are solved, and the efficiency and the quality of content creation are improved. According to the method, complex script semantics can be accurately understood, a scene transition effect is intelligently controlled based on content relevance, role actions and emotions are accurately expressed, and video coherence and smoothness are ensured through an advanced frame evaluation and optimization technology.

Description

Technical Field

[0001] This invention relates to the field of multimedia processing technology, and more specifically, to an AI-based video movie generation system. Background Technology

[0002] In traditional video production workflows, the transition from script to finished product involves a complex process of manual interpretation and transformation. Inconsistent understandings of the script by the creative team often lead to inaccurate interpretations, especially in large-scale collaborative productions. Existing AI video generation systems perform poorly when handling lengthy narratives, struggling to deeply understand the subtle relationships between complex scene descriptions and character behavior, often resulting in a significant disconnect between the generated output and the creative intent. Scene transitions frequently appear abrupt and jarring, failing to intelligently adjust to the narrative rhythm and emotional tone, thus disrupting the viewer's immersive experience. Regarding character performance, current technologies struggle to simultaneously coordinate action semantics and emotional expression, resulting in mechanical movements, exaggerated expressions, or emotions inconsistent with the context, creating the "uncanny valley" effect for viewers. The quality instability during video frame generation is particularly prominent; long video sequences often exhibit inter-frame flickering, character feature drift, and abrupt changes in background elements, and the lack of an effective temporal consistency assessment mechanism makes these problems difficult to systematically identify and correct. Traditional frame optimization methods struggle with complex scenes and dynamic content, particularly in maintaining the continuity of character features. The repaired content often exhibits noticeable flaws or lost details. These technical bottlenecks, coupled with high production costs and lengthy production cycles, create a vicious cycle, making it difficult for small production teams and independent creators to achieve high-quality visual storytelling. Ultimately, this leads to severe content homogenization, a scarcity of innovative works, and an inability to meet increasingly diverse market demands and audience expectations.

[0003] In view of this, the present invention proposes an AI-based video movie generation system to solve the above problems. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: an AI-based video movie generation system, comprising: The script parsing module is used to acquire the script text to be processed, perform semantic parsing on the script text, and obtain scene description sequences and character behavior description sequences. The scene transition analysis module is used to obtain the transition smoothing factor for each scene transition node based on the semantic association strength and visual style difference between adjacent scene descriptions in the scene description sequence. The character dynamic encoding module is used to obtain the character dynamic performance vector corresponding to each scene based on the action semantic encoding and emotional polarity value of each character behavior description in the character behavior description sequence. The video frame generation module is used to generate a sequence of video frame images corresponding to a scene frame by frame through a frame-level generation model, based on the transition smoothing factor of each scene transition node and the dynamic performance vector of the character in the corresponding scene. The temporal consistency evaluation module is used to obtain inter-frame temporal consistency scores based on the degree of optical flow continuity and structural consistency between adjacent frames in a video frame image sequence; and to filter frames that need optimization based on the inter-frame temporal consistency scores. The frame optimization and repair module is used to perform inter-frame interpolation repair and local feature reconstruction on the frames that need to be optimized, so as to obtain the target video movie.

[0005] The technical effects and advantages of the AI-based video movie generation system of this invention: This invention represents a breakthrough in digital content creation, reshaping the boundaries of video film production possibilities. Creators can seamlessly transform written ideas into fluid, coherent visual narratives without needing complex video production techniques, unleashing creative potential and imagination. The content production cycle is reduced from the traditional months to a very short time, allowing creators to rapidly iterate, experiment, and refine their work, focusing on the story itself rather than technical implementation. For independent filmmakers, small studios, and educational institutions, this invention eliminates the high barriers of equipment investment and professional team configuration, democratizing the allocation of creative resources. In terms of content quality, works generated by this invention feature natural and harmonious scene transitions, compelling character performances, and viewers can immerse themselves in a complete and coherent visual experience without being distracted by technical flaws. This combination of efficiency and high quality fosters entirely new creative possibilities, making experimental narratives, multi-version testing, and personalized content production a reality. This invention redefines production relations, reduces repetitive technical work, and allows creators to return to the essence of art. Attached Figure Description

[0006] Figure 1 This is a schematic diagram of the AI-based video movie generation system of the present invention. Detailed Implementation

[0007] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0008] This application provides an AI-based video and movie generation system. The system's execution entities include, but are not limited to, entities that run the system, such as film production platforms, AI video generation services, digital content creation centers, and multimedia production workstations, which can be considered general computing nodes in this application. The video generation system includes, but is not limited to, at least one cloud-based video generation engine, distributed rendering system, and intelligent script parser.

[0009] Please see Figure 1 In this embodiment of the invention, the AI-based video movie generation system includes: The script parsing module is used to acquire the script text to be processed, perform semantic analysis on the script text, and obtain scene description sequences and character behavior description sequences. The script parsing module receives the input script text and uses natural language processing technology to identify and extract key information such as scene boundary markers, scene content descriptions, character behavior markers, and corresponding behavior descriptions. The scene description sequences record background information such as environment, time, and atmosphere for each scene in the film, arranged in chronological order. The character behavior description sequences record in detail the dialogue, actions, expressions, and emotional states of each character in different scenes, arranged chronologically. This data provides the basic semantic material for subsequent scene generation and character dynamic performance, ensuring the consistency and integrity of the generated content with the original script.

[0010] The scene transition analysis module is used to obtain the transition smoothing factor for each scene transition node based on the semantic association strength and visual style difference between adjacent scene descriptions in the scene description sequence. The module calculates the semantic association strength by measuring the cosine similarity between the semantic feature vectors of adjacent scene descriptions; simultaneously, it extracts the visual style tags from each scene description to calculate the visual style difference between adjacent scenes. Semantic association strength reflects the coherence of scene content, while visual style difference reflects the magnitude of change in scene visual presentation. The transition smoothing factor is calculated as a weighted combination of the negative correlation mapping values ​​of semantic association strength and visual style difference, and is used to control the smooth transition effect during subsequent scene transitions, ensuring natural and fluid scene switching.

[0011] The character dynamic encoding module is used to obtain the character dynamic performance vector corresponding to each scene based on the action semantic encoding and emotional polarity value of each character's behavior description in the character behavior description sequence. This module first performs action semantic analysis on the character behavior description, extracting action category identifiers and action amplitude parameters; then it performs emotion recognition on the behavior description, obtaining emotion type labels and emotion intensity values; finally, it concatenates the action semantic encoding and emotional polarity values ​​of all characters in the same scene according to preset rules to form a complete character dynamic performance vector. These vectors accurately describe the dynamic performance characteristics of the character in each scene, providing key parameters for subsequently generating realistic character behavior videos.

[0012] The video frame generation module generates a sequence of video frames for each scene transition based on the transition smoothing factor and the character dynamic performance vector of that scene, using a frame-level generation model. First, the module generates initial keyframes for the current scene based on the character dynamic performance vector. Then, at each scene transition node, it determines the number of transition frames to be generated based on the transition smoothing factor. Next, based on the initial keyframes and the number of transition frames for both scenes, it obtains a sequence of transition frames using interpolation techniques. Finally, it concatenates the initial keyframes and transition frame sequences of each scene sequentially to form a complete sequence of video frames. This process ensures the continuity of video frame generation and the natural smoothness of scene transitions.

[0013] The temporal consistency evaluation module is used to obtain an inter-frame temporal consistency score based on the degree of optical flow continuity and structural consistency between adjacent frames in a video frame image sequence. Frames requiring optimization are then selected based on this score. This module first estimates the optical flow of adjacent frames in the video frame sequence to obtain the optical flow field. Then, it calculates the directional dispersion and amplitude change rate of the optical flow vector in the optical flow field to evaluate the degree of optical flow continuity. Simultaneously, it calculates the structural similarity index between adjacent frames as a measure of structural consistency. Finally, it uses the weighted sum of the optical flow continuity and structural consistency scores as the inter-frame temporal consistency score and selects frames requiring optimization based on a preset threshold. This evaluation process ensures the temporal coherence of the generated video, providing a precise target for frame optimization.

[0014] The frame optimization and repair module performs inter-frame interpolation repair and local feature reconstruction on frames requiring optimization to obtain the target video movie. This module first acquires the preceding and following frames for each frame requiring optimization and performs bidirectional optical flow estimation. Then, based on the optical flow estimation results, it performs optical flow-guided frame content reconstruction on the frames requiring optimization. Finally, it extracts edge and texture features from the reconstructed frames and aligns and optimizes them with corresponding features from adjacent frames to obtain optimized frames with reconstructed local features. This repair process effectively improves the quality of problematic frames, eliminates discontinuities in the video, and ultimately generates a high-quality, smooth, and natural target video movie.

[0015] The modules are connected via wired and / or wireless means to enable data transmission between them.

[0016] In this embodiment of the invention, the detailed implementation steps for obtaining the transition smoothing factor of each scene transition node include: Each scene description in the scene description sequence is text-encoded to obtain a semantic feature vector for each scene description. Text encoding is a fundamental step in extracting semantic features from scene description text, converting natural language descriptions into computer-processable vector representations. The encoding process uses a pre-trained language model, such as BERT or T5, inputting the scene description text into the encoder to obtain context-dependent semantic representations. In specific implementation, the scene description text is first segmented and converted into the model input format, then features are extracted through a multi-layer Transformer network, and finally, the output of the last layer's [CLS] tag or the output of the average pooling layer is taken as the semantic feature vector for the entire scene. These feature vectors are typically 768-1024 dimensions, comprehensively capturing the semantic content, sentiment tone, and environmental features of the scene description, providing a semantic foundation for subsequent scene association analysis.

[0017] The cosine similarity between semantic feature vectors describing adjacent scenes is calculated as the semantic association strength. Cosine similarity is a classic method for evaluating vector similarity and can effectively measure the semantic coherence between scenes. The calculation process directly applies the cosine similarity formula to the standardized semantic feature vectors. The cosine similarity calculation formula is: ; in, and These are the semantic feature vectors of adjacent scenes. For vector dot product, and It is the vector norm.

[0018] The similarity value ranges from -1 to 1. A value closer to 1 indicates a stronger semantic connection between scenes and a more natural content transition; a value closer to 0 or negative indicates a weaker semantic connection between scenes and a more abrupt transition. The strength of semantic connection directly affects the smoothness of scene transitions, providing a key semantic basis for calculating the transition smoothness factor.

[0019] Visual style tags are extracted from each scene description, and the visual style difference is obtained based on the category distance between the visual style tags of adjacent scene descriptions. Visual style tags are a high-level abstraction of the visual representation characteristics of a scene, reflecting its artistic style such as tone, lighting, and composition. The extraction process first establishes a pre-defined visual style dictionary, containing multiple pre-defined visual style categories and their corresponding keyword sets; then, keyword matching is performed on the scene descriptions, identifying the most matching visual style category as the style tag for that scene. The visual style difference between adjacent scenes is calculated using the shortest path length of the style category in the pre-defined style category hierarchy tree; the larger the path length, the more significant the visual style difference. This difference metric provides a visual reference for transition effect design, ensuring a smooth transition or conscious contrast of visual styles during scene transitions.

[0020] The transition smoothing factor for each scene transition node is obtained by normalizing the weighted product of the negative correlation mapping values ​​between semantic association strength and visual style difference. The transition smoothing factor is a core parameter for controlling scene transition effects, comprehensively considering both semantic coherence and visual style changes. The calculation process first performs a negative correlation mapping on the visual style difference, converting it into a visual coherence index in the same direction as the semantic association strength; then, the two are weighted and multiplied to reflect scene coherence in both semantic and visual dimensions; finally, the result is mapped to the [0,1] interval through Min-Max normalization to obtain the standardized transition smoothing factor. The calculation formula is: ; in, As a transition smoothing factor, For semantic association strength, For visual style differences, For the maximum possible style difference value, and These are the weighting coefficients, and , This indicates a normalization operation.

[0021] The transition smoothness factor directly determines the frame rate and transition effect of scene transitions. The larger the factor value, the smoother and more gradual the transition; the smaller the factor value, the faster and more direct the transition. This mechanism ensures the harmony and unity between scene transition effects and content semantics and visual style.

[0022] In this embodiment of the invention, the detailed implementation steps of the method for obtaining the character's dynamic performance vector include: Action semantic analysis is performed on each character behavior description in the character behavior description sequence to extract action category identifiers and action amplitude parameters, which are then encoded to obtain action semantic codes. Action semantic analysis is a fundamental step in understanding character behavior characteristics, converting text descriptions into structured action representations using natural language processing techniques. The analysis process combines semantic role annotation and action ontology mapping. First, verbs and modifiers in the description are identified to determine basic action types; then, parameters such as intensity, speed, and amplitude of the action are extracted based on context and modifiers; finally, the identified actions are mapped to a predefined action ontology library to obtain standardized action category identifiers. Action amplitude parameters are obtained through intensity analysis of modifiers and contextual understanding, quantified as continuous values ​​between 0 and 1. The complete action semantic coding uses a hybrid representation combining multi-hot coding and continuous parameters, which includes both discrete information about action categories and retains continuous parameters such as amplitude and speed, providing precise behavioral guidance for generating dynamic character performances.

[0023] For each character's behavioral description, sentiment recognition is performed to obtain sentiment polarity values, which include sentiment type labels and sentiment intensity values. Sentiment recognition is a crucial step in capturing a character's emotional state, quantifying sentiment information in the text through sentiment analysis techniques. The recognition process combines sentiment dictionary matching and deep learning classification. First, sentiment words and expressions in the text are identified through dictionary matching; then, a pre-trained sentiment classification model, such as a BERT-based sentiment analyzer, is used to perform sentiment analysis on the entire sentence to determine the dominant sentiment type; finally, sentiment intensity values ​​are calculated based on the intensity and distribution of sentiment words. Sentiment type labels typically include basic categories such as joy, sadness, anger, fear, surprise, and disgust, while sentiment intensity values ​​are quantified as continuous values ​​from 0 to 1, representing the degree of sentiment expression. Sentiment polarity values ​​provide the emotional basis for generating character facial expressions, tone of voice, and body language, ensuring consistency between the character's performance and the sentiment described in the text.

[0024] The action semantic codes and emotional polarity values ​​of all characters in each scene are concatenated into vectors according to a preset concatenation rule to obtain the character dynamic performance vector for that scene. Vector concatenation is a key step in integrating multi-dimensional character information, organizing scattered character dynamic features into a unified representation. The concatenation process first constructs a combined vector of action semantic codes and emotional polarity values ​​for each character appearing in the scene; then, these vectors are arranged according to character importance or order of appearance; finally, they are concatenated into a high-dimensional vector according to preset rules. To handle variations in the number of characters in different scenes, padding or truncation strategies are used to ensure consistent output vector dimensions. The final character dynamic performance vector is a comprehensive representation that fully encodes the dynamic behavior and emotional state of all characters in the scene, providing rich character performance parameters for video frame generation and ensuring accurate correspondence between the generated content and the script description.

[0025] In this embodiment of the invention, the detailed implementation steps of the method for obtaining inter-frame temporal consistency scores include: Optical flow estimation is performed on adjacent frames in a video frame image sequence to obtain the optical flow field between adjacent frames. Optical flow estimation is a fundamental technique for analyzing inter-frame motion relationships, describing object motion by calculating pixel-level displacement fields. The estimation process employs deep learning optical flow algorithms, such as PWC-Net or RAFT, to perform pixel matching on adjacent frames and generate a high-precision bidirectional optical flow field. Specifically, adjacent frames are first normalized and adjusted to the network input format; then, a multi-scale feature pyramid is extracted using a multi-layer convolutional network; next, feature matching and displacement estimation are performed at each scale; finally, upsampling and thinning are used to obtain the full-resolution optical flow field. The optical flow field is a two-dimensional vector field, where each vector represents the displacement direction and distance of the corresponding pixel from the current frame to the next frame, comprehensively capturing inter-frame motion information and providing a motion foundation for subsequent continuous analysis.

[0026] The directional dispersion and amplitude change rate of the optical flow vector in the optical flow field are calculated, and the product of the negative correlation mapping values ​​of directional dispersion and amplitude change rate is taken as the degree of optical flow continuity. Optical flow continuity analysis is a key step in evaluating motion smoothness, quantifying the degree of motion coherence by statistically analyzing the consistency characteristics of the optical flow field. The analysis process first calculates the directional standard deviation of the optical flow vector in a local region as the directional dispersion, reflecting the consistency of motion direction; then, it calculates the relative change rate of optical flow amplitude between adjacent regions, reflecting the smoothness of motion speed; finally, it combines the negative correlation mapping values ​​of the two (i.e., low dispersion and low change rate correspond to high continuity) to obtain a comprehensive optical flow continuity measure. The optical flow continuity degree ranges from [0,1], with higher values ​​indicating more coherent and natural motion, and lower values ​​indicating the presence of inconsistent and abrupt motion. This indicator directly reflects the smoothness of motion in a video frame sequence, providing a key basis for the motion dimension of temporal consistency assessment.

[0027] The structural similarity index between adjacent frames is calculated as a measure of structural consistency. Structural similarity analysis is an important method for assessing content stability, quantifying the continuity of content changes by comparing image structural features. The calculation process uses the SSIM (Structural Similarity Index) algorithm, which comprehensively considers the similarity across three dimensions: brightness, contrast, and structure. The SSIM calculation formula is: ; in, , and These represent the brightness, contrast, and structural similarity components, respectively. , and These are weighting coefficients, which are usually set to 1. and These are two adjacent frames of images.

[0028] The SSIM value ranges from [-1, 1] and is typically normalized to the [0, 1] interval to represent the degree of structural consistency. A higher value indicates a more stable inter-frame content structure, while a lower value indicates discontinuous content changes. Structural consistency and optical flow continuity together constitute the two main dimensions of temporal consistency assessment, comprehensively capturing the continuity characteristics of video quality.

[0029] The inter-frame temporal consistency score is calculated by weighting the degree of optical flow continuity and structural consistency. Frames with inter-frame temporal consistency scores below a preset consistency threshold are marked as frames requiring optimization. The temporal consistency score is a quality metric that integrates both motion and structure dimensions, directly reflecting the coherence of the video frame sequence. The score calculation uses a weighted sum method, dynamically adjusting the weights according to different scene characteristics: increasing the weight of optical flow continuity for moving scenes and increasing the weight of structural consistency for static scenes. The calculation formula is as follows: ; in, The score is given for inter-frame temporal consistency. For the degree of optical flow continuity, For the degree of structural consistency, and These are the weighting coefficients, and .

[0030] The score ranges from [0,1], with higher values ​​indicating smoother and more natural transitions between frames. The system sets a preset consistency threshold (usually 0.7-0.8), marking frames below the threshold as needing optimization and serving as targets for subsequent repairs. This filtering mechanism ensures that system resources are concentrated on improving the quality of problematic frames, effectively enhancing the overall video coherence and viewing experience.

[0031] In this embodiment of the invention, the detailed implementation steps for generating the video frame image sequence corresponding to the scene frame by frame include: Initial keyframes for the current scene are generated based on the character's dynamic performance vectors. Initial keyframe generation is a fundamental step in constructing the video frame sequence, transforming semantic features into visual representations through a conditional generation model. The generation process employs conditional image generation techniques based on diffusion models, such as Stable Diffusion or DALL-E, using scene description text and character dynamic performance vectors as conditional inputs. A high-quality scene image is generated through an iterative noise reduction process. Specifically, the character dynamic performance vectors are first fused with scene description features to construct a conditional embedding; then, a random noise image is initialized and progressively refined according to a preset number of noise reduction steps; finally, an additional detail enhancement network is used to improve image quality, resulting in the initial keyframes of the scene. Special emphasis is placed on the accuracy of character performance during keyframe generation; action poses, facial expressions, and scene composition are precisely controlled based on the character's dynamic performance vectors to ensure consistency between the visual presentation and the script description. These initial keyframes serve as visual anchors for the scene, providing a foundational reference for the generation of subsequent transition frames.

[0032] At scene transition nodes, the number of transition frames is determined based on the corresponding transition smoothing factor, and the number of transition frames is positively correlated with the transition smoothing factor. Determining the number of transition frames is a crucial step in controlling the scene transition rate, dynamically adjusting the gradualness of the transition through the transition smoothing factor. The determination process employs a non-linear mapping function, mapping the transition smoothing factor in the [0,1] interval to the actual required number of transition frames. The mapping formula is: ; in, For the number of transition frames, The minimum number of transition frames (usually 1-3). This is the maximum number of transition frames (usually 15-30). As a transition smoothing factor, The shape parameter (usually 0.5-2) controls the shape of the response curve.

[0033] When the transition smoothness factor is close to 0, it indicates significant differences between scenes or abrupt transitions, and the system generates fewer transition frames to achieve rapid switching. When the factor is close to 1, it indicates that the scenes are highly related and require a smooth transition, and the system generates more transition frames to achieve a gradual effect. This dynamic adjustment mechanism ensures the harmonious unity between scene transition effects and content semantics, enhancing the fluency of video narrative and artistic expression.

[0034] Based on the number of initial keyframes and transition frames in the preceding and following scenes, a sequence of transition frames is generated through interpolation. Transition frame generation is a core step in achieving smooth scene transitions, constructing a continuously changing path in visual space using advanced interpolation techniques. The generation process employs a controlled interpolation method based on a diffusion model, considering not only direct interpolation of image pixels but also emphasizing natural semantic transitions. Specifically, the implementation first extracts features from the initial keyframes of the preceding and following scenes in the latent space to obtain semantic-level representations; then, a non-uniform interpolation path is designed, increasing sampling density in key regions based on content characteristics; next, for each interpolation point, a corresponding transition frame is generated using a conditional diffusion model, guided by structural information from the preceding and following frames; finally, the generated frame sequence is optimized for temporal consistency to ensure the continuity of the transition. This semantically guided interpolation generation method, compared to traditional pixel blending or optical flow distortion, produces a more natural and reasonable scene transition effect, effectively avoiding common visual artifacts and content fragmentation problems.

[0035] The initial keyframes and transition frame sequences of each scene are sequentially stitched together to obtain a video frame image sequence. Sequential stitching is the final step in constructing a complete video sequence, forming a continuous video stream by orderly combining various frame sequences. The stitching process first determines the precise location of each scene and transition segment according to the script's timeline; then, the initial keyframes and transition frame sequences are arranged sequentially according to time; finally, global frame rate adjustment is applied to ensure that the video rhythm is consistent with the narrative requirements. During the stitching process, the system maintains a strict temporal correspondence to ensure the synchronization of video content with the script's narrative. To enhance the overall coherence of the video, global tone and brightness consistency processing is performed after stitching to eliminate visual style gaps between different scenes. The final generated video frame image sequence is a complete visual narrative flow, accurately expressing the script content and achieving natural transitions between scenes and a coherent display of character performances.

[0036] In this embodiment of the invention, the detailed implementation steps for inter-frame interpolation repair and local feature reconstruction of the frame to be optimized include: For each frame requiring optimization, the preceding and following frames are acquired, and bidirectional optical flow estimation is performed on both. Bidirectional optical flow estimation is a fundamental step in frame inpainting, providing motion guidance for content reconstruction by calculating the pixel correspondence between the preceding and following reference frames. The estimation process employs high-precision optical flow algorithms, such as RAFT or FlowNet 2.0, simultaneously calculating forward optical flow (from the previous frame to the next frame) and backward optical flow (from the next frame to the previous frame). Specifically, the input frames are first preprocessed, including noise suppression and edge enhancement; then, multi-scale feature representations are extracted using a feature pyramid; next, iterative feature matching and flow field refinement are performed at each scale; finally, outliers are removed through consistency checks to obtain an accurate bidirectional optical flow field. The key advantage of bidirectional estimation lies in its ability to detect occlusion regions and motion discontinuities. Through consistency verification of forward and backward flow, reliable matching regions and problem regions requiring special handling are identified, providing precise motion guidance and reliability indicators for subsequent frame content reconstruction.

[0037] Optical flow-guided frame content reconstruction is performed on frames requiring optimization based on bidirectional optical flow estimation results. Optical flow-guided reconstruction is the core step in repairing frame content, reconstructing problematic frames through reliable information from a reference frame and optical flow guidance. The reconstruction process employs a combination of optical flow-based spatiotemporal interpolation and neural network inpainting. Specifically, firstly, based on the bidirectional optical flow field, the ideal position and visibility weight of each pixel in the frame requiring optimization are calculated; then, forward and backward warping guided by optical flow are applied to reliable regions to obtain content from the reference frame; next, occluded and unreliable regions are filled in using a conditional generation network; finally, the warping and generation results are fused to obtain the preliminary reconstructed frame content. Optical flow guidance ensures motion continuity, while generation and inpainting guarantee visual plausibility. The combination of these two methods achieves high-quality reconstruction of frames requiring optimization, effectively eliminating common problems such as temporal inconsistencies, flickering, and jitter.

[0038] Edge and texture features are extracted from the reconstructed frames requiring optimization, and aligned with corresponding features in adjacent frames to obtain optimized frames after local feature reconstruction. Local feature reconstruction is a fine-grained step in frame optimization, improving the detail quality and temporal consistency of the reconstructed frames through feature-level alignment and enhancement. The reconstruction process first uses a dedicated feature extraction network to extract edge features (representing structural contours) and texture features (representing surface details); then, it compares these features with corresponding features in adjacent frames to identify inconsistencies; next, for these areas, feature transfer and style matching techniques are used to adjust the local features of the reconstructed frames; finally, the optimized features are re-integrated into the frames to generate the final optimized frames. Feature alignment optimization pays special attention to key areas such as human faces and object boundaries, ensuring the temporal consistency and stability of these visually significant parts. Through this local fine-grained processing, the system not only restores the macroscopic motion coherence between frames but also ensures the temporal consistency of microscopic details, significantly improving the quality and viewing experience of the final video.

[0039] In this embodiment of the invention, the detailed implementation steps for semantic parsing of the script text include: The script text undergoes sentence segmentation and semantic role labeling to identify scene boundary markers and character behavior markers. Sentence segmentation and semantic role labeling are fundamental steps in script parsing, transforming unstructured text into structured information using natural language processing techniques. The process begins with a syntactic analyzer segmenting the script text into sentences, identifying sentence boundaries and syntactic structure. Then, named entity recognition technology is applied to label key elements such as characters, locations, and times. Next, a semantic role labeling algorithm identifies the actor, object, and modifiers of each sentence. Finally, based on specific text patterns and marker symbols, scene boundary markers (such as scene titles and scene description start symbols) and character behavior markers (such as character names and dialogue introductory symbols) are identified. This step decomposes the original script text into basic semantic units, laying the foundation for subsequent semantic aggregation and sequence construction.

[0040] Consecutive sentences with the same scene boundary markers are aggregated into scene descriptions and arranged chronologically to form a scene description sequence. Scene aggregation is a crucial step in constructing the scene description sequence, organizing scattered sentences into coherent scene units through semantic association. The aggregation process first segments the script text into preliminary scene blocks based on identified scene boundary markers; then, it analyzes the sentences within each scene block to extract content describing the scene's environment, atmosphere, and temporal and spatial characteristics; next, it semantically fuses these descriptive contents to form complete scene descriptions; finally, it arranges the scene descriptions according to the chronological order in the script to form a scene description sequence. During the aggregation process, the system filters out dialogue and action instructions irrelevant to the scene descriptions, focusing on capturing the visual and atmospheric features of the scenes to provide rich semantic references for subsequent scene generation. The scene description sequence, as the spatiotemporal framework of the film, defines the visual flow and environmental changes of the entire story.

[0041] All sentences with character behavior markers are extracted as character behavior descriptions and arranged chronologically to form a character behavior description sequence. Character behavior extraction is the core step in constructing the character behavior description sequence, capturing dynamic character performances by accurately identifying character activity information in the script. The extraction process first locates the character activity parts in the script based on the identified character behavior markers; then, it analyzes the semantic structure of each part, extracting character names, dialogue content, action descriptions, and emotional expressions; next, it semantically associates the behavior descriptions of the same character in consecutive situations to form complete behavior units; finally, it arranges the character behavior descriptions according to the chronological order in the script to form a character behavior description sequence. During the extraction process, the system pays special attention to the semantics of verbs and modifiers, accurately capturing the character's action characteristics, emotional state, and interaction patterns, providing comprehensive behavior descriptions for subsequent character dynamic encoding. As dynamic elements of the film, the character behavior description sequence defines the characters' activities and emotional transitions in the story, serving as the foundational data for character performance generation.

[0042] In this embodiment of the invention, the detailed implementation steps of the visual style tag extraction method include: A pre-defined visual style dictionary is established, comprising multiple pre-defined visual style categories and corresponding keyword sets. This dictionary serves as a knowledge base for style recognition, assisting in the mapping from text to visual representation through a systematic style classification system. The construction process begins by identifying core visual style categories based on film studies and visual art theory, such as "realism," "expressionism," "minimalism," and "surrealism." Then, feature descriptions and visual attributes are defined for each style category. Next, a keyword set is established for each style category, including direct style terms, related descriptive words, typical visual elements, and emotional association words. Finally, this information is organized into a structured dictionary, supporting efficient style matching and querying. The dictionary also includes hierarchical relationships and similarity measures between style categories, forming a style category hierarchical tree for subsequent style difference calculations. As prior knowledge for the system, the pre-defined visual style dictionary provides a standardized reference for mapping scene text descriptions to visual expression styles, ensuring stylistic consistency and artistry in the generated content.

[0043] For each scene description, keyword matching is performed, and the preset visual style category with the highest matching frequency is used as the visual style tag for the corresponding scene description. Keyword matching is the core step in style tag extraction, mapping the scene description to the most suitable visual style through text analysis. The matching process first preprocesses the scene description text, including word segmentation, part-of-speech tagging, and stop word filtering; then, keywords and phrases are extracted, with particular attention to words describing environment, lighting, color, composition, and atmosphere; next, these keywords are matched with keyword sets of each category in the preset visual style dictionary, and the matching degree score is calculated; finally, the style category with the highest matching frequency is selected as the visual style tag for that scene. The matching algorithm not only considers direct word matching but also applies semantic similarity calculation, handling synonyms and related expressions to improve the flexibility and accuracy of matching. When the matching degree of multiple style categories is close, the system will select the tag that is more consistent with or more distinctive to the context scene style, maintaining the coherence and artistic expression of the visual style.

[0044] Category distance is the shortest path length of visual style labels describing adjacent scenes within a predefined style category hierarchy tree. Calculating category distance is a crucial step in measuring style difference, quantifying the magnitude of visual change through the semantic relationships between style categories. The calculation process is based on a predefined style category hierarchy tree, a tree structure representing the relationships between style categories, where nodes represent style categories and edges represent the strength of association between categories. Specifically, the implementation first locates the style label nodes of adjacent scenes in the hierarchy tree; then calculates the shortest path length between two nodes, i.e., the minimum number of edges required to travel from one node to another; finally, the path length is used as the distance metric between style categories. The distance is 0 when two scenes have the same style label; a smaller distance when they belong to similar style branches; and a larger distance when they belong to completely different style systems. Category distance intuitively reflects the magnitude of visual style change, providing a quantitative basis for the style dimension in calculating the transition smoothing factor, ensuring the coordination and unity between scene transition effects and the degree of style change.

[0045] This invention achieves fully automated generation from script text to video film through script parsing, scene transition analysis, dynamic character encoding, video frame generation, temporal consistency assessment, and frame optimization and repair. The AI ​​video generation system of this invention can accurately understand script content, naturally handle scene transitions, and realistically present character performances, providing an efficient and intelligent technical solution for digital content creation.

[0046] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0047] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0048] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An AI-based video movie generation system, characterized in that, include: The script parsing module is used to acquire the script text to be processed, perform semantic parsing on the script text, and obtain scene description sequences and character behavior description sequences. The scene transition analysis module is used to obtain the transition smoothing factor of each scene transition node based on the semantic association strength and visual style difference between adjacent scene descriptions in the scene description sequence. The character dynamic encoding module is used to obtain the character dynamic performance vector corresponding to each scene based on the action semantic encoding and emotional polarity value of each character behavior description in the character behavior description sequence. The video frame generation module is used to generate a sequence of video frame images corresponding to a scene frame by frame through a frame-level generation model, based on the transition smoothing factor of each scene transition node and the character dynamic performance vector of the corresponding scene. The temporal consistency evaluation module is used to obtain inter-frame temporal consistency scores based on the degree of optical flow continuity and structural consistency between adjacent frames in the video frame image sequence; and to filter frames that need optimization based on the inter-frame temporal consistency scores. The frame optimization and repair module is used to perform inter-frame interpolation repair and local feature reconstruction on the frames to be optimized, so as to obtain the target video movie.

2. The AI-based video movie generation system according to claim 1, characterized in that, The process of obtaining the transition smoothing factor for each scene transition node includes: Each scene description in the scene description sequence is text-encoded to obtain a semantic feature vector for each scene description. Calculate the cosine similarity between semantic feature vectors describing adjacent scenes, as the semantic association strength; Extract the visual style labels from each scene description, and obtain the visual style difference based on the category distance between the visual style labels of adjacent scene descriptions; The weighted product of the negative correlation mapping values ​​between the semantic association strength and the visual style difference is normalized to obtain the transition smoothing factor for each scene transition node.

3. The AI-based video movie generation system according to claim 1, characterized in that, The method for obtaining the character's dynamic performance vector includes: Perform action semantic analysis on each character behavior description in the character behavior description sequence, extract the action category identifier and action amplitude parameter, and encode them to obtain the action semantic code; Emotion recognition is performed on the behavior description of each character to obtain the emotion polarity value, which includes the emotion type label and the emotion intensity value; The action semantic codes and emotional polarity values ​​of all characters in each scene are concatenated into vectors according to a preset concatenation rule to obtain the dynamic performance vector of the characters in the corresponding scene.

4. The AI-based video movie generation system according to claim 1, characterized in that, The method for obtaining the inter-frame temporal consistency score includes: Optical flow estimation is performed on adjacent frames in the video frame image sequence to obtain the optical flow field between adjacent frames; Calculate the directional dispersion and amplitude change rate of the optical flow vector in the optical flow field, and take the product of the negative correlation mapping value of the directional dispersion and the amplitude change rate as the degree of optical flow continuity; Calculate the structural similarity index between adjacent frames as a measure of structural consistency; The weighted sum of the optical flow continuity degree and the structural consistency degree is used as the inter-frame temporal consistency score; Frames with inter-frame temporal consistency scores lower than a preset consistency threshold are marked as frames that need optimization.

5. The AI-based video movie generation system according to claim 1, characterized in that, The video frame image sequence corresponding to the scene is generated frame by frame, including: Generate the initial keyframes for the corresponding scene based on the character's dynamic performance vector in the current scene; At the scene transition node, the number of transition frames is determined according to the corresponding transition smoothing factor, and the number of transition frames is positively correlated with the transition smoothing factor. Based on the number of initial keyframes and transition frames in the preceding and following scenes, a transition frame sequence is obtained by interpolation. The initial keyframes and transition frame sequences of each scene are spliced ​​together in time sequence to obtain a video frame image sequence.

6. The AI-based video movie generation system according to claim 4, characterized in that, The step of performing inter-frame interpolation repair and local feature reconstruction on the frame to be optimized includes: Obtain the previous and next frames for each frame to be optimized, and perform bidirectional optical flow estimation on the previous and next frames; Based on the bidirectional optical flow estimation results, the frame content of the frame to be optimized is reconstructed using optical flow guidance. Edge and texture features are extracted from the reconstructed frame to be optimized, and aligned and optimized with the corresponding features of adjacent frames to obtain the optimized frame after local feature reconstruction.

7. The AI-based video movie generation system according to claim 1, characterized in that, The semantic parsing of the script text includes: The script text is segmented and semantically labeled to identify scene boundary markers and character behavior markers; Consecutive sentences with the same scene boundary markers are aggregated into scene descriptions and arranged in chronological order to form a scene description sequence; All sentences with character behavior tags are extracted as character behavior descriptions and arranged in chronological order to form a character behavior description sequence.

8. The AI-based video movie generation system according to claim 2, characterized in that, The method for extracting the visual style tags includes: Establish a preset visual style dictionary, which includes multiple preset visual style categories and corresponding keyword sets; For each scene description, perform keyword matching and use the preset visual style category with the highest matching frequency as the visual style tag for the corresponding scene description. The category distance is the shortest path length of the visual style labels describing adjacent scenes in a preset style category hierarchy tree.

Images