Dynamic video generation system driven by artificial intelligence vision and implementation method

Abstract

The invention discloses a dynamic video generation system driven by artificial intelligence vision and an implementation method, and relates to the technical field of AI video generation. The system comprises an input layer, a model layer, a rendering layer and an output layer, wherein the input layer supports multi-modal input of a single image, a video clip, text description and the like; the model layer integrates a visual feature extraction module, an AI generation model module and a dynamic adaptation module, realizes accurate feature extraction and inter-frame coherence guarantee through a self-attention mechanism and a time sequence attention mechanism, and aligns texts and visual semantics in combination with cross-modal feature mapping; the rendering layer completes frame smoothing, format conversion and resolution adaptation; and the output layer provides preview and secondary editing functions. According to the method, the limitations of single input, time sequence disjunction, insufficient customization and the like in the traditional technology are broken through, the convenience, reality sense and universality of video generation are improved, and multi-field scene requirements are met.

Description

Technical Field

[0001] This invention relates to dynamic video generation, specifically to an AI vision-driven dynamic video generation system and its implementation method. Background Technology

[0002] With the digital upgrade of media, e-commerce, education and other fields, the demand for dynamic video as an efficient information transmission carrier is surging. Ordinary users and professional creators are increasingly eager for "low-threshold, high-quality" dynamic video generation tools. Current dynamic video generation technologies mainly suffer from the following limitations: First, the input format is limited; most systems only support images or video clips as input, with weak cross-modal driving capabilities such as text descriptions, and a problem of "disconnect between text semantics and visual content." Second, temporal continuity is insufficient; generated videos often exhibit inter-frame jitter and broken motion logic, making it difficult to conform to real-world dynamic patterns. Third, customization is low; the flexibility in adjusting core parameters such as frame rate, motion speed, and resolution is poor, making it difficult to adapt to the differentiated needs of multiple scenarios. Fourth, model deployment has high barriers to entry; traditional AI generation models have large parameter sets and high inference latency, relying on high-performance hardware, making it difficult to popularize on edge devices. Fifth, cross-modal feature alignment accuracy is insufficient; the semantic conversion efficiency from text to vision is low, making it impossible to accurately reproduce the dynamic details in the description. These issues make it difficult for existing technologies to balance convenience, high quality, and universality, limiting the large-scale application of dynamic video generation technology. There is an urgent need for a multimodal compatible, time-optimized, lightweight, and highly customizable solution. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the defects of the above-mentioned technologies and provide an artificial intelligence vision-driven dynamic video generation system and implementation method.

[0004] To address the aforementioned technical problems, the present invention provides an AI vision-driven dynamic video generation system and implementation method: the AI ​​vision-driven dynamic video generation system includes:

[0005] An input layer is used to receive multimodal visual input data, which includes at least one of a single image, multiple images, video clips, text descriptions, and keyframe annotations.

[0006] The model layer, connected to the input layer, includes a visual feature extraction module, an AI generation model module, and a dynamic adaptation module. The visual feature extraction module is used to extract the texture, color, and structural features of the input visual data. The AI ​​generation model module is used to generate an initial frame sequence based on the extracted features. The dynamic adaptation module is used to optimize the inter-frame transition logic.

[0007] The rendering layer, connected to the model layer, is used to smooth the initial frame sequence, convert its format, and adapt its resolution.

[0008] The output layer, connected to the rendering layer, is used to output the processed dynamic video and provides a preview interface and a secondary editing interface.

[0009] As an improvement, the visual feature extraction module employs a pre-trained CNN model or a VisionTransformer model, and integrates a self-attention mechanism to focus on key visual regions in the input visual data. The self-attention mechanism achieves feature focusing by calculating attention weights, with the following formula:

[0010]

[0011] in, The query matrix (a set of query vectors representing the current feature). It is the key matrix (the set of key vectors representing all features). A value matrix (a set of value vectors representing all features). Given the dimension of the query vector, this formula calculates the correlation weights between features through scaling dot product operations, thereby enhancing the features of key regions.

[0012] As an improvement, the AI ​​generation model module adopts a diffusion model or a generative adversarial network, and integrates a temporal attention mechanism to ensure the inter-frame coherence of the generated frame sequence; the temporal attention mechanism achieves temporal consistency control by calculating inter-frame correlation weights, and the calculation formula is as follows:

[0013]

[0014] in, For the first Frame and the Temporal attention weights of frames, For the first Frame and the Frame similarity score, The formula normalizes the similarity score using the softmax function to represent the total number of reference frames, making the generated frames more relevant to temporally related reference frames.

[0015] As an improvement, the dynamic adaptation module includes a motion trend analysis unit and a parameter adjustment unit. The motion trend analysis unit generates a dynamic path based on the motion features of the input visual data. The parameter adjustment unit receives user-inputted custom parameters and adjusts the video generation parameters accordingly. The custom parameters include frame rate, motion speed, and camera switching mode. The motion speed is calculated using the inter-frame pixel change rate, using the following formula:

[0016]

[0017] in, For the speed of movement, The change in position of corresponding pixels in adjacent frames (unit: pixels). The time interval between adjacent frames (in seconds) is used to quantify the speed of motion and provide a velocity reference for dynamic path generation.

[0018] As an improvement, the output layer supports video formats including MP4 and AVI, and resolutions including 720P, 1080P, and 4K; the output layer achieves multi-resolution adaptation through a resolution conversion algorithm, the conversion formula being:

[0019]

[0020] in, The size (width or height) of the output resolution. The corresponding size for the input resolution. This refers to the scaling ratio (e.g., when converting 1080P to 720P). This formula is used for linear scaling of image size, ensuring the effectiveness of resolution conversion.

[0021] An AI-driven vision-based method for generating dynamic videos includes the following steps:

[0022] Input parsing steps: Preprocess the received multimodal visual input data, including size unification and noise reduction of image or video data, and feature vector conversion of text descriptions;

[0023] Feature modeling steps: Extract features from the preprocessed input data using a visual feature extraction model, fuse the extracted visual features with temporal information, and construct the input feature map for the generation model;

[0024] Video generation steps: Input the input feature map into the AI ​​generation model to generate the initial frame sequence, and optimize the inter-frame transition and motion logic of the initial frame sequence through the dynamic adaptation module;

[0025] Optimized output steps: The rendering module smooths and converts the optimized frame sequence to output dynamic video and provides preview and secondary editing functions.

[0026] As an improvement, the input parsing step includes the following process for transforming the text description into a feature vector: using a pre-trained language model to convert the text description into a semantic vector, and then using feature mapping to convert the semantic vector into a feature vector that matches the visual feature dimension; the feature mapping is implemented using a linear transformation, and the formula is:

[0027]

[0028] in, The transformed visual feature vector. For the mapping weight matrix, For text semantic vectors, As the bias vector, this formula maps the semantic vector to the visual feature space through a linear transformation, achieving cross-modal feature alignment.

[0029] As an improvement, in the feature modeling step, the visual feature extraction model adopts the ResNet series model, and the key region feature weights are strengthened through a self-attention mechanism. The key regions include the main object and dynamically changing parts; the formula for strengthening the key region weights is:

[0030]

[0031] in, For the enhanced version of the first One characteristic, Original features This is the enhancement coefficient (value range 0-1). For the first The attention weights of each feature (ranging from 0 to 1) are calculated using this formula. By adding weights to key regional features, the influence of these features in the subsequent generation process is enhanced.

[0032] As an improvement, in the video generation step, the AI ​​generation model adopts a lightweight diffusion model, and inference latency is reduced through model distillation technology. The dynamic adaptation module generates continuous motion paths based on motion trajectories predicted from the input visual data. The motion trajectory prediction uses weighted smoothing processing, with the following formula:

[0033]

[0034] in, For the first Predicted location at time For the first Weight of time position (satisfying) ), For historical position, The formula uses a weighted average of historical frames to make the predicted trajectory smoother and more continuous.

[0035] As an improvement, in the optimized output step, the rendering module smooths the frame sequence using a frame interpolation algorithm. This frame interpolation algorithm includes optical flow estimation interpolation or deep learning interpolation, and supports automatic adaptation to video resolution based on output device parameters. The intermediate frame calculation for frame interpolation uses a linear interpolation formula.

[0036]

[0037] in, Intermediate frames generated for interpolation, For the previous keyframe, For the next keyframe, The interpolation coefficients (ranging from 0 to 1) are used in this formula to generate smooth intermediate frames by linearly fusing pixel values ​​from adjacent frames, thus reducing inter-frame jumps.

[0038] The advantages of this invention compared to existing technologies are as follows: This technical solution has significant practical value and application advantages. It supports multimodal inputs such as single images, video clips, and text descriptions, and can initiate generation without professional skills, greatly reducing the barrier to entry. Through precise visual feature extraction and a self-attention mechanism, it can focus on the main object and dynamic parts, ensuring the core expressiveness of the generated video. The temporal attention mechanism and dynamic path optimization technology effectively avoid inter-frame jitter and motion breaks, making the video dynamics conform to real-world patterns and enhancing realism.

[0039] The system supports customization of parameters such as frame rate, motion speed, and resolution, adapting to diverse scenarios including education, advertising, and e-commerce, with a wide range of applications. Through cross-modal feature alignment technology, it accurately transforms textual semantics into visual content, bridging the "text-visual" semantic gap. Its lightweight design, achieved through model distillation, reduces hardware dependence, enables rapid inference, and is compatible with both edge and cloud deployments. Coupled with an iterative mechanism for previewing and secondary editing, it eliminates the need for rework, significantly reducing creation time and costs, and providing efficient, high-quality dynamic video generation solutions for users with different needs. Attached Figure Description

[0040] Figure 1 This is a system framework diagram of the AI ​​vision-driven dynamic video generation system and implementation method of the present invention.

[0041] Figure 2 This is a flowchart illustrating the implementation method of the AI ​​vision-driven dynamic video generation system and method of the present invention. Detailed Implementation

[0042] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be more thorough and complete.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0044] It is understood that spatial relation terms such as "below," "under," "below," "below," "above," "over," etc., can be used here to describe the relationship between one element or feature shown in the figure and other elements or features. It should be understood that, in addition to the orientation shown in the figure, spatial relation terms also include different orientations of the device in use and operation. For example, if the device in the figure is flipped, the element or feature described as "below" or "under" or "below" of the other element or feature will be oriented "over" the other element or feature. Therefore, the exemplary terms "below" and "under" can include both upper and lower orientations. Furthermore, the device may also include other orientations, such as being rotated 90 degrees or other orientations, and the spatial descriptive terms used herein will be interpreted accordingly.

[0045] It should be noted that when one element is considered to be "connected" to another element, it can be directly connected to the other element or connected to the other element through an intermediary element. In the following embodiments, "connection" should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have the transmission of electrical signals or data between them.

[0046] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” or “having,” etc., specify the presence of the stated feature, whole, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof.

[0047] Referring to the attached diagram, the AI ​​vision-driven dynamic video generation system and its implementation method are described. The AI ​​vision-driven dynamic video generation system includes:

[0048] An input layer is used to receive multimodal visual input data, which includes at least one of a single image, multiple images, video clips, text descriptions, and keyframe annotations.

[0049] The model layer, connected to the input layer, includes a visual feature extraction module, an AI generation model module, and a dynamic adaptation module. The visual feature extraction module is used to extract the texture, color, and structural features of the input visual data. The AI ​​generation model module is used to generate an initial frame sequence based on the extracted features. The dynamic adaptation module is used to optimize the inter-frame transition logic.

[0050] The rendering layer, connected to the model layer, is used to smooth the initial frame sequence, convert its format, and adapt its resolution.

[0051] The output layer, connected to the rendering layer, is used to output the processed dynamic video and provides a preview interface and a secondary editing interface.

[0052] As an improvement, the visual feature extraction module employs a pre-trained CNN model or a VisionTransformer model, and integrates a self-attention mechanism to focus on key visual regions in the input visual data. The self-attention mechanism achieves feature focusing by calculating attention weights, with the following formula:

[0053]

[0054] in, The query matrix (a set of query vectors representing the current feature). It is the key matrix (the set of key vectors representing all features). A value matrix (a set of value vectors representing all features). Given the dimension of the query vector, this formula calculates the correlation weights between features through scaling dot product operations, thereby enhancing the features of key regions.

[0055] As an improvement, the AI ​​generation model module adopts a diffusion model or a generative adversarial network, and integrates a temporal attention mechanism to ensure the inter-frame coherence of the generated frame sequence; the temporal attention mechanism achieves temporal consistency control by calculating inter-frame correlation weights, and the calculation formula is as follows:

[0056]

[0057] in, For the first Frame and the Temporal attention weights of frames, For the first Frame and the Frame similarity score, The formula normalizes the similarity score using the softmax function to represent the total number of reference frames, making the generated frames more relevant to temporally related reference frames.

[0058] As an improvement, the dynamic adaptation module includes a motion trend analysis unit and a parameter adjustment unit. The motion trend analysis unit generates a dynamic path based on the motion features of the input visual data. The parameter adjustment unit receives user-inputted custom parameters and adjusts the video generation parameters accordingly. The custom parameters include frame rate, motion speed, and camera switching mode. The motion speed is calculated using the inter-frame pixel change rate, using the following formula:

[0059]

[0060] in, For the speed of movement, The change in position of corresponding pixels in adjacent frames (unit: pixels). The time interval between adjacent frames (in seconds) is used to quantify the speed of motion and provide a velocity reference for dynamic path generation.

[0061] As an improvement, the output layer supports video formats including MP4 and AVI, and resolutions including 720P, 1080P, and 4K; the output layer achieves multi-resolution adaptation through a resolution conversion algorithm, the conversion formula being:

[0062]

[0063] in, The size (width or height) of the output resolution. The corresponding size for the input resolution. This refers to the scaling ratio (e.g., when converting 1080P to 720P). This formula is used for linear scaling of image size, ensuring the effectiveness of resolution conversion.

[0064] An AI-driven vision-based method for generating dynamic videos includes the following steps:

[0065] Input parsing steps: Preprocess the received multimodal visual input data, including size unification and noise reduction of image or video data, and feature vector conversion of text descriptions;

[0066] Feature modeling steps: Extract features from the preprocessed input data using a visual feature extraction model, fuse the extracted visual features with temporal information, and construct the input feature map for the generation model;

[0067] Video generation steps: Input the input feature map into the AI ​​generation model to generate the initial frame sequence, and optimize the inter-frame transition and motion logic of the initial frame sequence through the dynamic adaptation module;

[0068] Optimized output steps: The rendering module smooths and converts the optimized frame sequence to output dynamic video and provides preview and secondary editing functions.

[0069] As an improvement, the input parsing step includes the following process for transforming the text description into a feature vector: using a pre-trained language model to convert the text description into a semantic vector, and then using feature mapping to convert the semantic vector into a feature vector that matches the visual feature dimension; the feature mapping is implemented using a linear transformation, and the formula is:

[0070]

[0071] in, The transformed visual feature vector. For the mapping weight matrix, For text semantic vectors, As the bias vector, this formula maps the semantic vector to the visual feature space through a linear transformation, achieving cross-modal feature alignment.

[0072] As an improvement, in the feature modeling step, the visual feature extraction model adopts the ResNet series model, and the key region feature weights are strengthened through a self-attention mechanism. The key regions include the main object and dynamically changing parts; the formula for strengthening the key region weights is:

[0073]

[0074] in, For the enhanced version of the first One characteristic, Original features This is the enhancement coefficient (value range 0-1). For the first The attention weights of each feature (ranging from 0 to 1) are calculated using this formula. By adding weights to key regional features, the influence of these features in the subsequent generation process is enhanced.

[0075] As an improvement, in the video generation step, the AI ​​generation model adopts a lightweight diffusion model, and inference latency is reduced through model distillation technology. The dynamic adaptation module generates continuous motion paths based on motion trajectories predicted from the input visual data. The motion trajectory prediction uses weighted smoothing processing, with the following formula:

[0076]

[0077] in, For the first Predicted location at time For the first Weight of time position (satisfying) ), For historical position, The formula uses a weighted average of historical frames to make the predicted trajectory smoother and more continuous.

[0078] As an improvement, in the optimized output step, the rendering module smooths the frame sequence using a frame interpolation algorithm. This frame interpolation algorithm includes optical flow estimation interpolation or deep learning interpolation, and supports automatic adaptation to video resolution based on output device parameters. The intermediate frame calculation for frame interpolation uses a linear interpolation formula.

[0079]

[0080] in, Intermediate frames generated for interpolation, For the previous keyframe, For the next keyframe, The interpolation coefficients (ranging from 0 to 1) are used in this formula to generate smooth intermediate frames by linearly fusing pixel values ​​from adjacent frames, thus reducing inter-frame jumps.

[0081] This invention discloses an AI vision-driven dynamic video generation system and its implementation method. It drives an AI model to generate dynamic videos through multimodal visual input, and combines feature extraction, temporal modeling, and dynamic optimization techniques to achieve highly coherent and customizable video generation. The system structure and implementation process are described in detail below with reference to the accompanying drawings and embodiments.

[0082] I. Specific Implementation of System Structure:

[0083] 1.1 Input Layer:

[0084] The input layer is equipped with a data receiving interface and a preprocessing unit, supporting the acquisition and standardized processing of multimodal visual input data. Specifically, the multimodal data that the input layer can receive includes: single images (such as still images in JPG or PNG format), multiple images (such as image sequences containing temporal relationships), video clips (such as short videos in MP4 format), text descriptions (such as natural language commands such as "a cat jumps off a chair"), and keyframe annotations (such as user-marked starting and ending points of object motion).

[0085] The preprocessing unit performs unified processing on the input data: for image or video data, the size is unified to a preset resolution (e.g., 512×512 pixels) through interpolation algorithms, and Gaussian filtering is used for noise reduction to remove high-frequency noise; for text descriptions, natural language processing steps such as word segmentation and stop word removal are used to convert them into text sequences that the model can recognize.

[0086] 1.2 Model Layer:

[0087] The model layer is the core processing unit of the system, which includes a visual feature extraction module, an AI generation model module, and a dynamic adaptation module. These modules work together to complete the process of generating an initial frame sequence from input features.

[0088] 1.2.1 Visual Feature Extraction Module:

[0089] This module uses a pre-trained VisionTransformer model (such as ViT-Base) as the base network and integrates a self-attention mechanism to achieve feature focusing on key visual regions. The self-attention mechanism strengthens the feature representation of the main object (such as people or animals) and dynamically changing parts (such as limbs or vehicle wheels) by calculating the association weights between features. Its calculation formula is as follows:

[0090]

[0091] in, The query matrix consists of the feature vectors to be processed and is used to represent "what needs to be focused on". (Key matrix) and The (value matrix) is composed of all the feature vectors of the input data, which respectively represent the "identifier of the feature" and the "specific content of the feature"; The dimension of the query vector (in this embodiment) The formula is used to scale the dot product result to avoid gradient vanishing. Through this formula, the module can calculate the correlation between each feature and other features. For example, when processing an image of a person waving, features in the hand region receive higher attention weights, thus preserving their details preferentially in subsequent generation.

[0092] 1.2.2 AI Generative Model Module:

[0093] This module employs an improved diffusion model (such as StableDiffusion) as its basic generation framework, integrating a temporal attention mechanism to ensure inter-frame coherence. The diffusion model generates frame sequences through a progressive denoising process, while the temporal attention mechanism ensures the continuity of motion trajectories by calculating the association weights between the current frame and historical frames. The calculation formula is as follows:

[0094]

[0095] in, For the first Frame and the Temporal attention weights of frames, This refers to the feature similarity score between two frames calculated using cosine similarity. For reference, the total number of historical frames (in this embodiment) For example, when generating a video of a "ball falling", the first... The system uses this formula to focus on the ball's positional characteristics in the first 5 frames, ensuring that the falling trajectory conforms to the laws of physics and avoiding jumps or reverse movements.

[0096] 1.2.3 Dynamic Adaptation Module:

[0097] This module includes a motion trend analysis unit and a parameter adjustment unit, enabling the generation of dynamic paths and personalized parameter adjustments. The motion trend analysis unit extracts motion features (such as object displacement direction and velocity) from the input video using an optical flow estimation algorithm to generate continuous dynamic paths; the parameter adjustment unit provides a user interface that supports custom frame rates (such as 24fps, 30fps), motion speeds (such as 1.2x speed), and camera switching methods (such as push-pull, rotation).

[0098] The motion speed is calculated using the inter-frame pixel change rate quantization, and the formula is as follows:

[0099]

[0100] In the formula, The change in pixel position of the centroid of the same object in adjacent frames (unit: pixels). The time interval between adjacent frames (unit: seconds, e.g., 30fps) For example, when the calculation yields... At a speed of pixels per second, the system can automatically adjust to 80 pixels per second based on the user-set speed parameters (such as 0.8x speed) to achieve precise control over the speed of motion.

[0101] 1.3 Rendering Layer:

[0102] The rendering layer connects to the model layer and is responsible for post-processing the initial frame sequence, including frame smoothing, format conversion, and resolution adaptation. Frame smoothing uses a frame interpolation algorithm to fill gaps between frames and reduce screen jumps; format conversion supports encoding the frame sequence into common video formats such as MP4 and AVI; resolution adaptation achieves multi-device compatibility through a linear scaling algorithm, with the conversion formula as follows:

[0103]

[0104] in, Width or height for output resolution. The corresponding size of the input frame sequence. For scaling ratios (e.g., when converting a 1080P video to 720P), Using this formula, the system can automatically adjust the video size according to the output device (such as a mobile phone screen at 720P or a monitor at 1080P) to ensure display quality.

[0105] 1.4 Output Layer:

[0106] The output layer is configured with a video storage unit, a preview interface, and a secondary editing interface. The video storage unit supports saving the processed video to local storage or the cloud; the preview interface displays the generated effect in real time through a web page or client; the secondary editing interface allows users to adjust the frame rate, crop the image, etc., and feed the modified parameters back to the model layer for regeneration, achieving iterative optimization.

[0107] II. Specific steps for implementing the method:

[0108] 2.1 Input parsing steps:

[0109] This step preprocesses the multimodal visual input data to prepare for subsequent feature extraction. Specifically, it includes:

[0110] (1) Image / video preprocessing: For the input single image, multiple images or video clips, the size is unified to 512×512 pixels by bilinear interpolation, and then a 3×3 Gaussian kernel is used to perform convolution operation on the image to remove noise. The standard deviation of the Gaussian kernel is set to 0.5.

[0111] (2) Text description transformation: The input text description (such as "the puppy ran towards the tree") is first transformed into a 768-dimensional semantic vector through a pre-trained language model (such as BERT). Then, the semantic vector is transformed into a feature vector that matches the visual feature dimension (1024 dimensions in this embodiment) through feature mapping. The mapping process is implemented using linear transformation, and the formula is as follows:

[0112]

[0113] in, The mapping weight matrix is ​​1024×768 (obtained through training and optimization). It is a 1024-dimensional bias vector. This is the transformed visual feature vector. This process maps textual semantics to the visual feature space through linear transformation, achieving the alignment of "text-visual" cross-modal features. For example, the textual semantics of "puppy" can be accurately associated with the visual features of a dog in an image.

[0114] 2.2 Feature modeling steps:

[0115] This step extracts features from the preprocessed input data using a visual feature extraction model and integrates temporal information to construct the input feature map for the generation model. Specifically, it includes:

[0116] (1) Feature extraction: The ResNet-50 model is used as the backbone network for visual feature extraction. The preprocessed image / video frames are convolved layer by layer to extract a deep feature map of 512×512×2048. This feature map contains the texture, color and structural information of the input data.

[0117] (2) Key region enhancement: The attention weight of each position in the feature map is calculated through a self-attention mechanism. And use the following formula to enhance the features of key areas (such as the puppy's head and limbs):

[0118]

[0119] in, These are the original eigenvalues. For the enhancement coefficient (in this embodiment) ), Attention weights (values ​​range from 0 to 1, for key regions) (Approximately 1). This formula amplifies the feature values ​​of key areas; for example, the features of a puppy's head will stand out more than the features of the background grass, ensuring the clarity of the subject in the generated video.

[0120] (3) Temporal fusion: For multiple images or video clips, the extracted feature maps are arranged in chronological order, and the temporal information is fused through a 3D convolutional layer (3×3×3 convolutional kernel) to generate an input feature map containing spatiotemporal features (size 512×512×256), which provides temporal constraints for the generation of subsequent frame sequences.

[0121] 2.3 Video generation steps:

[0122] This step involves feeding the input feature map into the AI ​​generation model to generate an initial frame sequence, and then optimizing the inter-frame logic through a dynamic adaptation module. Specifically, this includes:

[0123] (1) Initial frame generation: A lightweight diffusion model (based on StableDiffusion for model distillation, reducing the number of parameters by 40%) is adopted. Using the input feature map as a condition, an initial frame sequence is generated through a 50-step denoising process (e.g., 300 frames for a 10-second 30fps video). Model distillation transfers knowledge from the large model to the small model through a knowledge distillation loss function (e.g., KL divergence), reducing inference latency (in this embodiment, the single generation latency is ≤2 seconds).

[0124] (2) Dynamic path optimization: The dynamic adaptation module predicts continuous motion paths based on the motion trajectory of the input data (such as the running path of the dog in the video) using a weighted smoothing algorithm. The formula is:

[0125]

[0126] in, For the first The predicted location of the puppy in the frame. Weights for historical positions (in this embodiment) ,satisfy ), This shows the puppy's actual position in the first three frames. This formula predicts location by incorporating historical movement trends, resulting in a smoother running path for the puppy and preventing sudden changes in direction.

[0127] 2.4 Optimize output steps:

[0128] This step uses the rendering module to post-process the frame sequence, outputting the final video and providing interactive functionality. Specifically, it includes:

[0129] (1) Frame smoothing: An optical flow estimation interpolation algorithm is used to generate intermediate frames to fill the gaps in the initial frame sequence. For adjacent keyframes... and intermediate frame The calculation uses a linear interpolation formula:

[0130]

[0131] in, These are interpolation coefficients (values ​​ranging from 0 to 1, for example, when generating an intermediate frame between two frames). Using this formula, the pixel values ​​of the intermediate frames are linearly blended from those of adjacent keyframes. For example, as the puppy runs from position A to position B, the intermediate frames will show a gradual displacement effect, reducing screen jumps.

[0132] (2) Format and resolution adaptation: The frame sequence is encoded by H.264 encoder according to the output format selected by the user (such as MP4); the video size is automatically adjusted by applying the resolution conversion formula according to the output device parameters (such as the detected mobile phone screen resolution of 720P), and finally outputs a video file that meets the device requirements.

[0133] (3) Preview and Edit: Users can view the video effect through the preview interface of the output layer. If adjustments are needed, the parameters can be modified through the secondary editing interface (such as changing the frame rate from 24fps to 30fps). After receiving the parameters, the system returns to the video generation step to reprocess until a video that meets the requirements is generated.

[0134] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0135] Through core designs such as multimodal input fusion, AI-generated model optimization, and dynamic adaptation technology innovation, significant beneficial effects are demonstrated in multiple dimensions, including technical performance, user experience, and application scenarios, as detailed below:

[0136] Multimodal input compatibility significantly improves system applicability and convenience:

[0137] The system supports multimodal visual input, including single images, multiple images, video clips, text descriptions, and keyframe annotations. This breaks through the reliance of traditional video generation systems on a single input format, significantly lowering the barrier to entry for users. For users with different needs, such as designers with only static images, who can generate coherent animations from multiple images; creators lacking visual materials, who can directly drive video generation through text descriptions (such as "the tide is gradually rising under the setting sun"); and video editors, who can expand upon existing video clips—this "full-scene input coverage" capability allows the system to adapt to the entire spectrum of needs, from professional creation to amateur expression, without requiring users to master complex video shooting or modeling skills, greatly improving the convenience of dynamic video generation.

[0138] Accurate visual feature extraction ensures the core expressiveness of the generated content:

[0139] The system's visual feature extraction module uses a pre-trained CNN or VisionTransformer model, combined with a self-attention mechanism (such as the formula). The feature-focused feature extraction system accurately captures texture, color, and structural features from input visual data, while prioritizing the feature weights of the main subject (such as people or products) and dynamically changing parts (such as body movements or object trajectories). This precise feature extraction capability ensures that the generated video always focuses on the core content, avoiding interference from redundant background information. For example, when generating a video of an athlete running, the system prioritizes preserving the details of the athlete's body movements rather than over-rendering irrelevant background information, thereby significantly improving the video's content expressiveness and information delivery efficiency.

[0140] Timing continuity optimization results in generated videos that more closely resemble real-world dynamic patterns.

[0141] Traditional AI video generation often faces problems such as inter-frame jitter and broken motion logic. This system, however, ensures temporal continuity through a dual mechanism: on the one hand, the AI ​​generation model integrates a temporal attention mechanism (such as the formula). The inter-frame correlation weights are calculated so that the current frame actively references the features of historical frames when it is generated, ensuring the continuity of the motion trajectory. On the other hand, the dynamic adaptation module uses a motion trend analysis unit (such as formulas) to analyze the motion trend. Quantified motion speed) predicts a reasonable dynamic path, and combines it with a weighted smoothing algorithm (such as formula) The generated continuous trajectory optimizes the inter-frame transition. This dual protection makes the generated video dynamics closer to the laws of physics and human visual expectations. For example, when generating a video of a "ball falling freely," the ball's position changes strictly follow the laws of acceleration, avoiding phenomena such as jumping or hovering that are contrary to common sense, greatly improving the realism and viewing experience of the video.

[0142] Customizable parameters allow for diverse customization needs across various scenarios:

[0143] The system's dynamic adaptation module has a parameter adjustment unit that supports user-defined frame rate (e.g., 24fps, 30fps), motion speed (e.g., 0.5x slow motion, 2x fast forward), and camera switching methods (e.g., push-pull, rotation, pan), etc., combined with the rendering layer's resolution adaptation function (e.g., formula). The system enables multi-resolution conversion, flexibly meeting the customized needs of different scenarios. For example, in the education field, users can generate slow-motion videos that clearly demonstrate experimental steps by reducing motion speed and increasing frame rate; in the advertising field, it can create visually impactful product promotional videos by quickly switching between shots and adapting to 4K resolution; and in mobile applications, it can automatically adapt to 720P resolution to reduce bandwidth usage. This high degree of customization allows the system to cover the differentiated needs of multiple fields such as education, media, entertainment, and e-commerce, expanding the application boundaries of the technology.

[0144] Cross-modal feature alignment bridges the text-visual semantic gap:

[0145] To address the challenges of text-based video generation, the system employs feature mapping techniques (such as formulas). The system achieves a semantic vector-to-visual feature conversion, accurately mapping the text semantic vectors output by the pre-trained language model to the visual feature space, realizing cross-modal alignment of "text semantics - visual features". This technological breakthrough solves the problem of "disconnect between description and content" in traditional text-to-video generation. For example, when the input text is "a red bird flies from a tree into the sky", the system can not only accurately generate the visual subject "red bird", but also accurately reproduce the dynamic process of "taking off from the tree → flying into the sky", ensuring that the semantics of the text description are completely and accurately transformed into visual content, providing reliable support for creative scenarios with "zero visual materials".

[0146] Lightweight design and efficient rendering enhance the system's usability and accessibility.

[0147] The system uses model distillation technology to lightweight the AI-generated model (reducing the number of parameters by more than 40%), significantly reducing inference latency (single generation latency ≤ 2 seconds). This makes it adaptable to various deployment environments, including edge devices (such as mobile phones and tablets) and cloud servers, avoiding the dependence of traditional AI video generation on high-performance hardware. Simultaneously, the rendering layer employs an efficient frame interpolation algorithm (such as the formula...). The system achieves a smooth transition and format conversion technology, enabling rapid conversion of frame sequences to common video formats (MP4, AVI), reducing post-processing time costs. This "efficient and lightweight" design allows ordinary users to quickly generate high-quality videos without professional equipment, promoting the widespread application of AI video generation technology.

[0148] Iterative optimization mechanisms reduce creation costs and time consumption:

[0149] The system output layer provides a preview interface and secondary editing functions, allowing users to view the generated effect in real time and adjust parameters (such as resetting motion speed and modifying keyframes) to feed back to the model layer for regeneration, forming a closed-loop iterative mechanism of "generation-preview-optimization". This mechanism avoids the drawback of traditional video generation where "one-time output and modifications require starting from scratch". For example, if a user finds that the turning angle in the generated "car driving" video is too large, they can directly adjust the camera switching parameters through the secondary editing interface. The system does not need to reprocess the entire frame sequence, but only optimizes the local dynamics, greatly reducing the time and manpower required for repeated creation and improving creative efficiency.

[0150] In summary, this technical solution, through innovations such as multimodal compatibility, accurate feature extraction, temporal optimization, personalized customization, cross-modal alignment, lightweight design, and iterative mechanisms, not only breaks through many limitations of traditional video generation at the technical level, but also provides users in different fields with efficient, convenient, and high-quality dynamic video generation solutions at the application level, demonstrating significant technological advancement and practical value.

[0151] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. An AI-driven vision-based dynamic video generation system, characterized in that: include: An input layer is used to receive multimodal visual input data, which includes at least one of a single image, multiple images, video clips, text descriptions, and keyframe annotations. The model layer, connected to the input layer, includes a visual feature extraction module, an AI generation model module, and a dynamic adaptation module. The visual feature extraction module is used to extract the texture, color, and structural features of the input visual data. The AI ​​generation model module is used to generate an initial frame sequence based on the extracted features. The dynamic adaptation module is used to optimize the inter-frame transition logic. The rendering layer, connected to the model layer, is used to smooth the initial frame sequence, convert the format, and adapt the resolution. The output layer, connected to the rendering layer, is used to output the processed dynamic video and provides a preview interface and a secondary editing interface.

2. The AI ​​vision-driven dynamic video generation system according to claim 1, characterized in that: The visual feature extraction module employs a pre-trained CNN model or a VisionTransformer model, and integrates a self-attention mechanism to focus on key visual regions in the input visual data. The self-attention mechanism achieves feature focusing by calculating attention weights, with the following formula: in, The query matrix (a set of query vectors representing the current feature). It is the key matrix (the set of key vectors representing all features). A value matrix (a set of value vectors representing all features). Given the dimension of the query vector, this formula calculates the correlation weights between features through scaling dot product operations, thereby enhancing the features of key regions.

3. The AI ​​vision-driven dynamic video generation system according to claim 1, characterized in that: The AI ​​generation model module employs a diffusion model or a generative adversarial network, and integrates a temporal attention mechanism to ensure the inter-frame coherence of the generated frame sequence. The temporal attention mechanism achieves temporal consistency control by calculating inter-frame correlation weights, with the calculation formula as follows: in, For the first Frame and the Temporal attention weights of frames, For the first Frame and the Frame similarity score, The formula normalizes the similarity score using the softmax function to represent the total number of reference frames, making the generated frames more relevant to temporally related reference frames.

4. The AI ​​vision-driven dynamic video generation system according to claim 1, characterized in that: The dynamic adaptation module includes a motion trend analysis unit and a parameter adjustment unit. The motion trend analysis unit generates a dynamic path based on the motion features of the input visual data. The parameter adjustment unit receives user-inputted custom parameters and adjusts the video generation parameters accordingly. The custom parameters include frame rate, motion speed, and camera switching mode. The motion speed is calculated using the inter-frame pixel change rate, using the following formula: in, For the speed of movement, The change in position of corresponding pixels in adjacent frames (unit: pixels). The time interval between adjacent frames (in seconds) is used to quantify the speed of motion and provide a speed benchmark for dynamic path generation.

5. The AI ​​vision-driven dynamic video generation system according to claim 1, characterized in that: The output layer supports video formats including MP4 and AVI, and resolutions including 720P, 1080P, and 4K. The output layer achieves multi-resolution adaptation through a resolution conversion algorithm, the conversion formula of which is: in, The size (width or height) of the output resolution. The corresponding size for the input resolution. This refers to the scaling ratio (e.g., when converting 1080P to 720P). This formula is used for linear scaling of image size, ensuring the effectiveness of resolution conversion.

6. A method for generating dynamic videos driven by artificial intelligence vision, characterized in that: Includes the following steps: Input parsing steps: Preprocess the received multimodal visual input data, including size unification and noise reduction of image or video data, and feature vector conversion of text descriptions; Feature modeling steps: Extract features from the preprocessed input data using a visual feature extraction model, fuse the extracted visual features with temporal information, and construct the input feature map for the generation model; Video generation steps: Input the input feature map into the AI ​​generation model to generate the initial frame sequence, and optimize the inter-frame transition and motion logic of the initial frame sequence through the dynamic adaptation module; Optimized output steps: The rendering module smooths and converts the optimized frame sequence to output dynamic video and provides preview and secondary editing functions.

7. The method for generating dynamic video driven by artificial intelligence vision according to claim 6, characterized in that: The input parsing step, specifically the feature vector transformation of the text description, includes: converting the text description into a semantic vector using a pre-trained language model, and then converting the semantic vector into a feature vector matching the visual feature dimensions through feature mapping; the feature mapping is implemented using a linear transformation, with the following formula: in, The transformed visual feature vector, For the mapping weight matrix, For text semantic vectors, As the bias vector, this formula maps the semantic vector to the visual feature space through a linear transformation, achieving cross-modal feature alignment.

8. The method for generating dynamic video driven by artificial intelligence vision according to claim 6, characterized in that: In the feature modeling step, the visual feature extraction model adopts the ResNet series model, and the feature weights of key regions are enhanced through a self-attention mechanism. The key regions include the main object and dynamically changing parts; the formula for enhancing the weights of the key regions is: in, For the enhanced version of the first One characteristic, Original features This is the enhancement coefficient (value range 0-1). For the first The attention weights of each feature (ranging from 0 to 1) are calculated using this formula. By adding weights to key regional features, the influence of these features in the subsequent generation process is enhanced.

9. The method for generating dynamic video driven by artificial intelligence vision according to claim 6, characterized in that: In the video generation step, the AI ​​generation model employs a lightweight diffusion model, reducing inference latency through model distillation technology. The dynamic adaptation module predicts and generates continuous motion paths based on motion trajectories in the input visual data. The motion trajectory prediction uses weighted smoothing, with the following formula: in, For the first Predicted location at time For the first Weight of time position (satisfying) ), For historical position, The formula uses a weighted average of historical frames to make the predicted trajectory smoother and more continuous.

10. The method for generating dynamic video driven by artificial intelligence vision according to claim 6, characterized in that: In the optimized output step, the rendering module smooths the frame sequence using a frame interpolation algorithm. This algorithm includes optical flow estimation interpolation or deep learning interpolation, and supports automatic adaptation to video resolution based on output device parameters. The intermediate frame calculation for frame interpolation uses a linear interpolation formula. in, Intermediate frames generated for interpolation, For the previous keyframe, For the next keyframe, The interpolation coefficients (ranging from 0 to 1) are used in this formula to generate smooth intermediate frames by linearly fusing pixel values ​​from adjacent frames, thus reducing inter-frame jumps.

Images