Video generation system based on video large model

Abstract

The invention discloses a video generation system based on a video large model, and belongs to the technical field of video generation. A trajectory control unit; a physical rule constraint unit; the scene generation unit is used for constructing a high-fidelity dynamic environment and enhancing the scene immersion of the video; a timing consistency unit; a multi-modal fusion unit; the user interaction unit is used for realizing man-machine collaborative creation and real-time control and improving the generation efficiency; a quality evaluation unit; the post-processing optimization unit is used for improving the terminal applicability of the output video and adapting to multi-platform requirements; and a distributed deployment unit. Millimeter-level regulation and control of the object motion trail are realized through a multi-dimensional accurate control mechanism, and physical rule constraints are combined to ensure that the generated video conforms to the reality mechanics law, so that the content authenticity is remarkably improved; the multi-modal fusion technology integrates text, voice and image input, accurately restores the intention of the user and realizes lip shape synchronization and emotion migration.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and in particular to a video generation system based on a large video model. Background Art

[0002] Traditional video generation systems have significant defects in following physical laws and matching user intentions: existing technologies mostly rely on end-to-end generation models and lack precise control modules for object motion trajectories, resulting in physical distortions such as sudden speed changes and uncontrolled suspension in generated videos; at the same time, existing systems are not able to integrate multimodal inputs (such as text and voice), making it difficult to accurately transform semantic instructions into visual actions. For example, there are widespread problems such as asynchrony between lip shape and voice rhythm, and deviation between emotional expression and text description. In addition, existing methods usually ignore the optimization of temporal coherence, and it is easy to have sudden changes in character appearance or logical contradictions in scenes when generating long videos, which requires a lot of manual correction.

[0003] Although the current improvement scheme attempts to introduce a physical engine or a single modality optimization module, it has failed to build a systematic solution: most physical constraint modules are disconnected from the generation model and are only used as post-processing tools, and cannot correct the illegal frames in the generation process in real time; the existing multimodal fusion technology mostly adopts a simple cascade method, and has not established a cross-modal semantic alignment and dynamic weight adjustment mechanism, resulting in a low match between the generated content and complex instructions. At the same time, the traditional system lacks a distributed architecture design and cannot support large-scale industrial production needs. The copyright protection mechanism is weak and it is difficult to deal with the infringement risks in commercial applications. Therefore, we propose a video generation system based on a large video model to solve this problem. Summary of the invention

[0004] The purpose of the present invention is to provide a video generation system based on a large video model to solve the problems raised in the above background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: A video generation system based on a large video model, comprising: Multimodal data preprocessing unit integrates and cleans multi-source data, extracts structured information, and provides high-quality input for video generation; The trajectory control unit accurately plans and controls the object's motion path to ensure the accuracy of the trajectory of the generated video; Physical rule constraint unit, which forces the generated content to conform to the real physical laws and improve the authenticity of the video; The scene generation unit builds a high-fidelity dynamic environment to enhance the scene immersion of the video; Timing consistency unit ensures dynamic continuity between video frames and avoids jumps and logical contradictions; The multi-modal fusion unit integrates multi-source input signals to drive video generation and improve the user intention matching degree; The user interaction unit realizes human-computer collaborative creation and real-time control to improve the generation efficiency; The quality assessment unit quantitatively evaluates the quality of the generated video in multiple dimensions to ensure high standards of the output content; The post-processing optimization unit improves the terminal applicability of the output video to adapt to the requirements of multiple platforms; The distributed deployment unit supports industrial production with high concurrency and low latency to ensure the scalability of the system.

[0006] Preferably, the multi-modal data preprocessing unit includes: The cross-modal alignment module: uses the CLIP model to align text, speech, and images, generates a spatio-temporal semantic mapping table, and ensures the timestamp synchronization and semantic association of the input data; The key object detection module: identifies the core dynamic objects in the video through the improved YOLOv7 model, outputs the object category, bounding box, and motion trend vector, and improves the detection accuracy of small targets; The dynamic data cleaning module: combines the optical flow method and frame difference analysis to filter out low-quality frames and static scenes, retains the significant motion regions, and reduces the amount of invalid data processing.

[0007] Preferably, the trajectory control unit includes: The trajectory planning module supports generating smooth paths by hand-drawing or physical engines, and optimizes the trajectory curvature using Bezier curves and dynamic constraints; The trajectory encoding module compresses the trajectory data into a latent space vector through a variational autoencoder and cascades it with the noise latent variables of the video large model to enhance the stability of trajectory control; The dynamic weight adjustment module adjusts the trajectory constraint weights using the PID algorithm based on the real-time feedback pressure sensor data to ensure the convergence of motion errors.

[0008] Preferably, the physical rule constraint unit includes: The rigid body dynamics module integrates the Bullet physics engine to simulate gravity, collisions, and joint motions, and corrects abnormal frames that do not conform to the law of conservation of momentum; The fluid simulation module generates fluid effects such as water flow and flames based on the SPH algorithm, and ensures the rationality of the interaction between the fluid and the scene through the particle-mesh model; The material response module dynamically adjusts the deformation and reflection parameters according to the material properties, such as the realistic rendering of cloth wrinkles and metal reflections.

[0009] Preferably, the scene generation unit includes: 3D scene reconstruction module, which uses NeRF technology to reconstruct a 3D model with lighting from a monocular video and supports free switching of camera positions; Lighting matching module, which generates virtual light source parameters through GAN to ensure the lighting consistency between the synthesized object and the background; Dynamic weather simulation module, which superimposes weather effects such as rain and snow in real time, adjusts the raindrop trajectory according to the wind speed field, and simulates the snow accumulation effect.

[0010] Preferably, the temporal consistency unit includes: Temporal memory module, which embeds an LSTM network to store historical frame features to prevent sudden changes in the appearance of people or abnormal disappearance of objects; Motion interpolation module, which inserts intermediate frames based on an optical flow-guided deformation grid and outputs a smooth video at 60 FPS.

[0011] Preferably, the multimodal fusion unit includes: Text-driven module, which parses natural language instructions into skeletal action parameters and generates limb movement trajectories through inverse kinematics; Speech lip-sync module, which uses the Wav2Lip model to achieve millisecond-level synchronization of lip shapes and speech, and supports multi-language input; Emotion transfer module, which drives the amplitude of the character's expression based on speech emotion analysis, such as the eyebrows pressing down and the corners of the mouth tightening when angry.

[0012] Preferably, the user interaction unit includes: Real-time trajectory editing module, which supports users to select objects by bounding boxes and drag to adjust the path, and achieves interactive correction through low-latency transmission; Bio-signal feedback module, which integrates an eye tracker and an electroencephalogram device to capture the attention hotspots and emotional fluctuations, and dynamically optimizes the generated details; Multi-version management module, which saves the generated intermediate states, supports branch backtracking and differential visualization comparison, and assists users in making decisions on the optimal version.

[0013] Preferably, the quality assessment unit includes: Physical compliance detection module, which verifies anomalies such as sudden speed changes and uncontrolled suspension, marks the violation frames and triggers regeneration; Aesthetic scoring module, which evaluates the picture composition, color balance and dynamic rhythm through a deep aesthetic network and outputs a score from 0 to 100; Semantic consistency module, which calculates the semantic similarity between the generated video and the input text, and automatically iteratively optimizes the missing actions.

[0014] Preferably, the post-processing optimization unit includes: Super-resolution enhancement module, which uses the ESRGAN model to upscale the video to 4K resolution and restore high-frequency texture details; Space-time artifact removal module, which uses a diffusion model to repair and generate defects, ensuring that the repair results are dynamically consistent with the context; Cross-platform adaptation module, which automatically compresses the bit rate and converts the format to generate VR stereoscopic videos or social media adaptation versions.

[0015] Preferably, the distributed deployment unit includes: Computing acceleration module, which optimizes the inference speed based on TensorRT, and a single card supports real-time generation of 30FPS video streams; Dynamic resource scheduling module, which uses Kubernetes cluster management to elastically allocate GPU resources and preferentially process high-priority tasks; Security and copyright module, which embeds invisible digital watermarking and blockchain evidence storage technologies to prevent content tampering and infringement.

[0016] The beneficial effects of the present invention are as follows: 1. In the present invention, the video generation system based on a large video model realizes millimeter-level precision control of the movement path of an object through the dual effects of the trajectory control unit and the physical rule constraint unit. The trajectory planning module combines Bezier curves and dynamic constraints to optimize the trajectory smoothness, and the dynamic weight adjustment module automatically corrects the movement deviation based on real-time feedback data to ensure that the movement trajectory of the object in the generated video strictly conforms to physical laws. At the same time, the coordinated operation of the rigid body dynamics module and the fluid simulation module effectively simulates the collision, gravity, and fluid movement effects in the real world, making the physical rationality of the generated video reach the film and television industry standard, significantly reducing the manual correction cost; 2. In the present invention, the video generation system based on a large video model integrates multi-source input signals such as text, voice, and images, and realizes cross-modal semantic alignment through the multi-modal fusion unit. The text-driven module parses natural language instructions into skeletal action parameters, the speech lip synchronization module ensures millisecond-level matching of lip shapes and speech rhythms, and the emotion transfer module dynamically adjusts the character's expression according to the speech emotion. This multi-modal cooperation mechanism enables the generated video to accurately restore the user's intention, such as converting the instruction of "a person waves angrily" into coherent facial expressions and body movements, significantly improving the efficiency and accuracy of content creation; 3. In the present invention, the video generation system based on the large video model, through the optical flow constraint and motion interpolation technology of the timing consistency unit, the system effectively eliminates the frame jump and logical contradiction in the video generation. The optical flow constraint module forces the displacement field of the generated video to match the predicted optical flow to reduce the object jitter; the timing memory module stores the historical frame features through the LSTM network to avoid the sudden disappearance or morphological mutation of objects in long videos, and the motion interpolation module can intelligently supplement the low frame rate video to 60FPS to ensure dynamic smoothness. This technical combination makes the generated video have the same coherence as the real video, meeting the high-standard film and television production requirements; 4. In the present invention, the video generation system based on the video big model, the quality assessment unit builds a full-dimensional quality monitoring system through physical compliance detection, aesthetic scoring and semantic consistency analysis. The physical compliance detection module automatically identifies abnormal frames such as speed mutation and uncontrolled suspension and triggers regeneration; the aesthetic scoring module quantifies the visual performance from multiple angles such as composition, color, rhythm, etc., and guides the model to optimize the aesthetics of the picture; the semantic consistency module ensures a high degree of match between the generated content and the input instructions through multimodal comparative learning. Combined with the super-resolution enhancement and de-artifact restoration of the post-processing optimization module, the system can automatically output 4K videos that meet professional standards, greatly reducing the cost of manual review; 5. In the present invention, a video generation system based on a large video model adopts a distributed deployment unit to realize the elastic scheduling of computing resources and parallel processing of tasks. The computing acceleration module optimizes TensorRT and multi-GPU parallelism to increase the single-card reasoning speed to real-time generation of 30FPS video streams; the dynamic resource scheduling module is based on Kubernetes cluster management, giving priority to real-time interaction requests and automatically expanding computing nodes to ensure low-latency response in high-concurrency scenarios. The security and copyright module uses invisible digital watermarks and blockchain evidence storage technology to ensure the traceability of content copyright. The architecture supports the simultaneous generation of thousands of video streams, meeting the large-scale production needs of film and television industrialization, batch production of advertisements, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a system block diagram of a video generation system based on a large video model proposed by the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0019] Reference Figure 1 , a video generation system based on a video macro model, comprising: Multimodal Data Preprocessing Unit, which integrates and cleans multi-source data, extracts structured information, and provides high-quality input for video generation; Trajectory Control Unit, which accurately plans and regulates the movement path of objects to ensure the trajectory accuracy of the generated video; Physical Rule Constraint Unit, which enforces the generated content to conform to real-world physical laws and enhances the authenticity of the video; Scene Generation Unit, which constructs a high-fidelity dynamic environment and enhances the scene immersion of the video; Temporal Consistency Unit, which ensures the dynamic coherence between video frames, avoiding jumps and logical contradictions; Multimodal Fusion Unit, which integrates multi-source input signals to drive video generation and improves the matching degree of user intentions; User Interaction Unit, which realizes human-machine collaborative creation and real-time control, improving the generation efficiency; Quality Assessment Unit, which quantifies the quality of the generated video in multiple dimensions to ensure high standards of the output content; Post-processing Optimization Unit, which improves the terminal applicability of the output video and adapts to the requirements of multiple platforms; Distributed Deployment Unit, which supports high-concurrency and low-latency industrial production and ensures the scalability of the system.

[0020] In this embodiment, the Multimodal Data Preprocessing Unit includes: Cross-modal Alignment Module: Using the CLIP model to align text, speech, and images, generating a spatio-temporal semantic mapping table to ensure the timestamp synchronization and semantic correlation of the input data; Key Object Detection Module: Identifying the core dynamic objects in the video through an improved YOLOv7 model, outputting object categories, bounding boxes, and motion trend vectors, and improving the detection accuracy of small targets; Dynamic Data Cleaning Module: Combining optical flow method and frame difference analysis to filter low-quality frames and static scenes, retaining significant motion regions, and reducing the amount of invalid data processing.

[0021] In this embodiment, the Trajectory Control Unit includes: Trajectory Planning Module, which supports generating smooth paths by hand-drawing or physical engines, and optimizing the trajectory curvature using Bezier curves and dynamic constraints; Trajectory Encoding Module, which compresses trajectory data into latent space vectors through a variational autoencoder and cascades them with the noise latent variables of the video large model to enhance the stability of trajectory control; Dynamic Weight Adjustment Module, which adjusts the trajectory constraint weights using the PID algorithm based on the real-time feedback pressure sensor data to ensure the convergence of motion errors.

[0022] In this embodiment, the Physical Rule Constraint Unit includes: The rigid body dynamics module integrates the Bullet physics engine to simulate gravity, collisions, and joint movements, and corrects abnormal frames that do not conform to the law of conservation of momentum; The fluid simulation module generates fluid effects such as water flow and flames based on the SPH algorithm, and ensures the rationality of the interaction between the fluid and the scene through the particle-mesh model; The material response module dynamically adjusts the deformation and reflection parameters according to the material properties, such as the realistic rendering of cloth wrinkles and metal reflections.

[0023] In this embodiment, the scene generation unit includes: The 3D scene reconstruction module uses NeRF technology to reconstruct a lighted 3D model from a monocular video, supporting free switching of the camera position; The lighting matching module generates virtual light source parameters through GAN to ensure the lighting consistency between the synthesized object and the background; The dynamic weather simulation module superimposes weather effects such as rain and snow in real time, adjusts the raindrop trajectory according to the wind speed field, and simulates the snow accumulation effect.

[0024] In this embodiment, the temporal consistency unit includes: The temporal memory module embeds an LSTM network to store the historical frame features, preventing sudden changes in the appearance of characters or abnormal disappearance of objects; The motion interpolation module inserts intermediate frames based on an optical flow-guided deformation grid and outputs a smooth 60FPS video.

[0025] In this embodiment, the multi-modal fusion unit includes: The text-driven module parses natural language instructions into skeletal action parameters and generates limb movement trajectories through inverse kinematics; The speech lip-sync module uses the Wav2Lip model to achieve millisecond-level synchronization of lip shapes and voices, supporting multi-language input; The emotion transfer module drives the amplitude of the character's expression based on speech emotion analysis, such as the eyebrows pressing down and the corners of the mouth tightening when angry.

[0026] In this embodiment, the user interaction unit includes: The real-time trajectory editing module supports users to select objects by bounding boxes and drag to adjust the path, and realizes interactive correction through low-latency transmission; The bio-signal feedback module integrates an eye tracker and an EEG device to capture the attention hotspots and emotional fluctuations, and dynamically optimizes the generated details; The multi-version management module saves the generated intermediate states, supports branch backtracking and differential visualization comparison, and assists users in making decisions on the optimal version.

[0027] In this embodiment, the quality assessment unit includes: Physical compliance detection module, which verifies anomalies such as sudden speed changes and uncontrolled suspension, marks the offending frames and triggers regeneration; Aesthetic scoring module, which uses a deep aesthetic network to evaluate the composition, color balance and dynamic rhythm of the picture and outputs a score of 0-100; The semantic consistency module calculates the semantic similarity between the generated video and the input text, and automatically iterates and optimizes the missing actions.

[0028] In this embodiment, the post-processing optimization unit includes: The super-resolution enhancement module uses the ESRGAN model to upscale the video to 4K resolution and restore high-frequency texture details; The spatiotemporal artifact removal module uses a diffusion model to repair generated defects and ensure that the repair results are consistent with the context dynamics; The cross-platform adaptation module automatically compresses the bit rate and converts the format to generate VR binocular video or social media adaptation versions.

[0029] In this embodiment, the distributed deployment unit includes: Computing acceleration module, based on TensorRT to optimize inference speed, a single card supports real-time generation of 30FPS video stream; Dynamic resource scheduling module, using Kubernetes cluster management, flexibly allocates GPU resources, and prioritizes high-priority tasks; The security and copyright module embeds invisible digital watermarks and blockchain evidence storage technology to prevent content tampering and infringement.

[0030] In this embodiment, a multi-dimensional precise control mechanism is used to achieve millimeter-level control of the object's motion trajectory, and physical rule constraints are combined to ensure that the generated video conforms to the laws of real mechanics, significantly improving the authenticity of the content; multimodal fusion technology integrates text, voice and image inputs, accurately restores user intentions and achieves lip synchronization and emotional migration; the timing consistency guarantee module eliminates frame jumps and logical contradictions, and outputs smooth and coherent high-frame rate videos; the automated quality assessment system achieves full-dimensional optimization through physical compliance detection, aesthetic scoring and semantic analysis, reducing the cost of manual review; the distributed architecture supports flexible resource scheduling and high-concurrency processing, combined with secure watermarking and blockchain evidence storage technology, to meet the needs of industrialized production and ensure copyright traceability, providing efficient and reliable video generation solutions for film and television, advertising and virtual reality fields. The above has introduced in detail a video generation system based on a video large model provided by the present invention. Specific embodiments are applied in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications can still be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A video generation system based on a large video model, characterized in that, Including: A multimodal data preprocessing unit that integrates and cleans multi-source data, extracts structured information, and provides high-quality input for video generation; A trajectory control unit that accurately plans and regulates the movement path of an object to ensure the trajectory accuracy of the generated video; A physical rule constraint unit that enforces the generated content to conform to real-world physical laws and enhances the authenticity of the video; A scene generation unit that constructs a high-fidelity dynamic environment and enhances the scene immersion of the video; A temporal consistency unit that ensures the dynamic coherence between video frames, avoiding jumps and logical contradictions; A multimodal fusion unit that integrates multi-source input signals to drive video generation and improves the matching degree of user intentions; A user interaction unit that realizes human-computer collaborative creation and real-time control and improves the generation efficiency; A quality assessment unit that quantitatively evaluates the quality of the generated video in multiple dimensions to ensure high standards of the output content; A post-processing optimization unit that improves the terminal applicability of the output video and adapts to the requirements of multiple platforms; A distributed deployment unit that supports high-concurrency and low-latency industrial production and ensures the scalability of the system.

2. The video generation system based on a video large model according to claim 1, wherein, The multimodal data preprocessing unit includes: A cross-modal alignment module: Using the CLIP model to align text, speech, and images, generating a spatio-temporal semantic mapping table to ensure the timestamp synchronization and semantic association of the input data; A key object detection module: Identifying the core dynamic objects in the video through an improved YOLOv7 model, outputting object categories, bounding boxes, and motion trend vectors, and improving the detection accuracy of small targets; A dynamic data cleaning module: Combining optical flow method and frame difference analysis to filter low-quality frames and static scenes, retaining significant motion regions, and reducing the amount of invalid data processing.

3. The video generation system based on a large video model according to claim 1, wherein The trajectory control unit includes: A trajectory planning module that supports generating smooth paths by hand-drawing or physical engines, and optimizing the trajectory curvature using Bezier curves and dynamic constraints; A trajectory encoding module that compresses trajectory data into latent space vectors through a variational autoencoder and cascades them with the noise latent variables of the video large model to enhance the stability of trajectory control; A dynamic weight adjustment module that adjusts the trajectory constraint weights using the PID algorithm based on the pressure sensor data of real-time feedback to ensure the convergence of motion errors.

4. The video generation system based on a large video model according to claim 1, wherein, The physical rule constraint unit includes: A rigid body dynamics module that integrates the Bullet physics engine to simulate gravity, collisions, and joint movements, and corrects abnormal frames that do not conform to the law of conservation of momentum; A fluid simulation module that generates fluid effects such as water flow and flames based on the SPH algorithm, and ensures the rationality of the interaction between the fluid and the scene through a particle-grid model; A material response module that dynamically adjusts deformation and reflection parameters according to material properties, such as the realistic rendering of cloth wrinkles and metal reflections.

5. The video generation system based on a video large model according to claim 1, wherein The scene generation unit includes: A three-dimensional scene reconstruction module that reconstructs a three-dimensional model with lighting from a monocular video using NeRF technology, supporting free switching of camera positions; A lighting matching module that generates virtual light source parameters through GAN to ensure the lighting consistency between the synthesized object and the background; A dynamic weather simulation module that superimposes weather effects such as rain and snow in real time, adjusts the raindrop trajectories according to the wind speed field, and simulates the snow accumulation effect.

6. The video generation system based on a large video model according to claim 1, wherein, The temporal consistency unit includes: Temporal Memory Module, which embeds an LSTM network to store historical frame features and prevent sudden changes in human appearance or abnormal disappearance of objects; Motion Interpolation Module, which inserts intermediate frames based on an optical flow-guided deformation grid and outputs a 60FPS smooth video.

7. The video generation system based on a large video model according to claim 1, wherein The multimodal fusion unit includes: Text-driven Module, which parses natural language instructions into skeletal action parameters and generates limb movement trajectories through inverse kinematics; Speech Lip Sync Module, which uses the Wav2Lip model to achieve millisecond-level synchronization of lip shapes and speech, supporting multilingual input; Emotion Transfer Module, which drives the amplitude of the character's expression based on speech emotion analysis, such as the eyebrows pressing down and the corners of the mouth tightening when angry.

8. The video generation system based on a large video model according to claim 1, wherein The user interaction unit includes: Real-time Trajectory Editing Module, which supports users to select objects by bounding boxes and drag to adjust the path, and achieves interactive correction through low-latency transmission; Bio-signal Feedback Module, which integrates an eye tracker and an electroencephalogram device to capture attention hotspots and emotional fluctuations, and dynamically optimizes the generated details; Multi-version Management Module, which saves the generated intermediate states, supports branch backtracking and differential visualization comparison, and assists users in making decisions on the optimal version.

9. The video generation system based on a large video model according to claim 1, wherein The quality assessment unit includes: Physical Compliance Detection Module, which verifies anomalies such as sudden speed changes and uncontrolled suspension, marks the violating frames and triggers regeneration; Aesthetic Scoring Module, which evaluates the picture composition, color balance and dynamic rhythm through a deep aesthetic network and outputs a score from 0 to 100; Semantic Consistency Module, which calculates the semantic similarity between the generated video and the input text and automatically iteratively optimizes the missing actions.

10. The video generation system based on a large video model according to claim 1, wherein, The post-processing optimization unit includes: Super-resolution Enhancement Module, which uses the ESRGAN model to upscale the video to 4K resolution and restore high-frequency texture details; Spatio-temporal Artifact Removal Module, which uses a diffusion model to repair the generated defects and ensures that the repair results are dynamically consistent with the context; Cross-platform Adaptation Module, which automatically compresses the bitrate and converts the format to generate a VR stereoscopic video or a version adapted for social media; The distributed deployment unit includes: Computing Acceleration Module, which optimizes the inference speed based on TensorRT, and a single card supports real-time generation of a 30FPS video stream; Dynamic Resource Scheduling Module, which uses a Kubernetes cluster for management, elastically allocates GPU resources, and preferentially processes high-priority tasks; Security and Copyright Module, which embeds invisible digital watermarking and blockchain certification technologies to prevent content tampering and infringement.

Images