Video resolution enhancement system and method based on pre-trained video generation model

Abstract

The video resolution enhancement system and method based on the pre-trained video generation model relates to the field of image processing technology, including: S1, obtaining the low-resolution video to be processed and its related text description and pre-processing through the pre-processing module; S2, processing the pre-processed low-resolution video and its related text description through the pre-trained video generation model to obtain high-level features; S3, extracting additional visual features from the low-resolution video through a feature extractor; S4, inputting the high-level features and the additional visual features into the super-resolution module to obtain a preliminary high-resolution video; S5, inputting the high-level features and the preliminary super-resolution image into the fusion module to obtain the final high-resolution video. The present invention not only effectively improves the spatial resolution of the video, but also maintains the temporal consistency and semantic integrity of the video, can handle various complex scenes and dynamic content, and overcomes the limitations of traditional methods in processing fast motion and complex textures.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video resolution enhancement system and method based on a pre-trained video generation model. Background Art

[0002] With the rapid development and popularization of high-definition display technology, ultra-high-definition televisions including 4K and 8K, high-resolution mobile devices, and large-screen displays are becoming increasingly mainstream. This trend has led to increasing requirements for the resolution of video content. However, a large amount of existing low-resolution video content, such as early-produced movies, TV shows, and home videos, cannot directly meet the needs of these new display devices. Therefore, video super-resolution technology has become an extremely important research field, aiming to convert low-resolution videos into high-resolution versions to enhance the viewing experience.

[0003] Traditional video super-resolution methods mainly include interpolation-based methods, reconstruction-based methods, and learning-based methods. Interpolation methods (such as bilinear interpolation, bicubic interpolation, etc.) are simple to calculate but have limited effects and often result in blurred images. Reconstruction-based methods restore details by establishing a mapping relationship from low resolution to high resolution, but they are not effective when dealing with complex textures. Learning-based methods, especially deep learning methods, have made significant progress in recent years and are able to learn more complex nonlinear mapping relationships and generate higher quality high-resolution images.

[0004] However, these methods still face many challenges when processing videos. The first is the problem of temporal consistency, that is, how to ensure that the processing results between adjacent frames remain consistent and avoid visual artifacts such as flickering or jitter. The second is the ability to handle complex dynamic scenes, how to accurately reconstruct the details and contours of fast-moving objects. In addition, there are difficulties in maintaining the overall style and semantic information of the video, especially for works of art or video content with a specific style.

[0005] In recent years, the emergence of large-scale pre-trained video generation models (such as Open-Sora) has provided new possibilities for video super-resolution tasks. These models have learned rich video content representation and generation capabilities by pre-training on massive video data. They can not only understand the semantic content of the video, but also capture complex spatiotemporal relationships and dynamic changes. This powerful video understanding and generation capability provides new ideas for solving the challenges faced by traditional video super-resolution methods.

[0006] However, how to effectively utilize the capabilities of these pre-trained video generation models and combine them with specialized super-resolution techniques is still a problem that needs further research. It is against this background that this paper proposes an innovative video resolution enhancement method, which aims to give full play to the advantages of pre-trained video generation models while overcoming the limitations of existing technologies to achieve high-quality, temporally coherent video resolution enhancement.

[0007] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of this application and are not admitted to be prior art by inclusion in this section. Summary of the invention

[0008] In order to overcome the deficiencies in the background technology, the present invention discloses a video resolution enhancement system and method based on a pre-trained video generation model.

[0009] In order to achieve the above-mentioned invention object, the present invention adopts the following technical scheme:

[0010] In a first aspect, the present invention provides a video resolution enhancement system based on a pre-trained video generation model, comprising:

[0011] A pre-trained video generation model is used to extract high-level features from low-resolution videos and their related text descriptions; the pre-trained video generation model adopts the Diffusion Transformer architecture, uses the PixArt-a open source high-quality text-to-image generation model, and is expanded into a video generation model by adding a temporal attention layer; the high-level features include: video content, video structure, and motion pattern; the related text description includes: video title, introduction, tags, or content description;

[0012] A feature extractor, used to extract additional visual features from the low-resolution video; the additional visual features include: multi-scale features, edge details, texture information, and motion information features;

[0013] A super-resolution module that generates preliminary high-resolution videos from high-level features and additional visual features;

[0014] The super-resolution module includes the following submodules:

[0015] Super-resolution input submodule: used to receive high-level features and additional visual features;

[0016] Feature extraction and processing submodule: used to extract high-level features and additional visual features through three consecutive convolutional layers, including three convolutional layers, where the first two convolutional layers are followed by ReLU activation functions to introduce nonlinearity, while the third convolutional layer is not followed by ReLu activation function;

[0017] Upsampling and detail reconstruction submodule: used to gradually increase the spatial resolution of the feature map through convolution and upsampling operations, including two PixelShuffle upsampling layers and three convolution layers. The first PixelShuffle upsampling layer is used for upsampling, which can reconstruct high-frequency details without significantly increasing the computational complexity. It is followed by two convolution layers, in which the first convolution layer is followed by a ReLU activation function, while the second convolution layer is not followed by an activation function. Then, PixelShuffle upsampling is applied again to further improve the resolution. The last convolution layer is used for final detail adjustment to generate a preliminary high-resolution video.

[0018] Super-resolution output submodule: used to output preliminary high-resolution video frames after the above processing;

[0019] The fusion module is used to fuse the high-level features and the preliminary high-resolution video to generate the final high-resolution video.

[0020] Specifically, the video resolution enhancement system further includes:

[0021] A preprocessing module that checks whether the input low-resolution video can be read correctly and verifies the validity of its associated text description;

[0022] A training module, used to train the pre-trained video generation model, the super-resolution module, and the models in the fusion module through a preset training strategy;

[0023] The inference optimization module is used to improve the efficiency of model inference in the pre-trained video generation model, super-resolution module and fusion module through video segmentation processing, feature caching mechanism and dynamic resolution adjustment.

[0024] Specifically, the feature extractor in the feature extractor includes the following units:

[0025] Multi-scale feature extraction unit: used to extract multi-scale features using convolution kernels of different sizes, and uses VGGNet's multi-layer convolution-pooling structure to extract multi-scale features at different depths;

[0026] Edge detail extraction unit: used to capture edge and contour information features in the image using a specific edge detection algorithm; the specific edge detection algorithm includes Sobel and Canny;

[0027] Texture information extraction unit: used to extract texture features in the video using texture analysis technology or to model texture features by using convolutional layers and fully connected layers; the texture analysis technology includes LBP and GLCM;

[0028] Motion information extraction unit: used to predict the motion between adjacent frames using an optical flow estimation algorithm or a convolutional neural network, and output the contained motion information features.

[0029] Specifically, the fusion module includes the following submodules:

[0030] Fusion input submodule: used to receive high-level features and preliminary high-resolution video frames;

[0031] Feature processing submodule: used to fuse high-level features and preliminary high-resolution video frames; including feature mapping layer, multi-head attention mechanism and feature fusion layer. The feature mapping layer maps the output of the input submodule to the same feature space, the multi-head attention mechanism captures the relationship between the features output by the feature mapping layer, and the feature fusion layer integrates the output of the multi-head attention mechanism.

[0032] Detail enhancement submodule: used to enhance the detail information of the image; including at least two convolutional layers and ReLU activation functions and residual connections, wherein the convolutional layer receives the output of the feature processing submodule and performs a convolution operation, the ReLU activation function performs a nonlinear transformation on the output of the convolutional layer, and the residual connection directly adds the preliminary high-resolution video frame to the output of the detail enhancement part;

[0033] Fusion output submodule: used to generate the final high-resolution video frame through an output convolutional layer.

[0034] Specifically, the preset training strategy includes phased training, progressive resolution increase, and dynamic batch size adjustment;

[0035] Phased training: First, freeze the pre-trained video generation model and only train the super-resolution module and fusion module; then gradually unfreeze the layers of the pre-trained video generation model and perform global fine-tuning;

[0036] Progressive resolution increase: Start training from a low resolution, gradually increase to the target resolution, and gradually learn multi-scale features;

[0037] Dynamic batch size adjustment: Dynamically adjust the batch size according to the current training resolution to optimize memory usage and training efficiency.

[0038] Specifically, the training module trains the pre-trained video generation model, the super-resolution module and the model in the fusion module through a preset training strategy, including:

[0039] 1) Data preparation: prepare low-resolution video datasets, high-resolution video datasets and related text descriptions, preprocess the datasets, including normalization, cropping, and rotation operations, and then split the datasets into training sets and validation sets;

[0040] 2) Model initialization: Initialize model parameters and set hyperparameters, including learning rate and batch size;

[0041] 3) Training cycle, including: a. Loading batch data; b. Performing forward propagation to calculate model output; c. Loss calculation; d. Performing back propagation to calculate gradient; e. Updating model parameters to reduce loss; f. Determining whether the preset validation interval has been reached. If it is within the validation interval, the model performance is evaluated on the validation set. If the model performance has improved, the current model is saved as the best model. If the performance has not improved, check whether the early stopping condition is met; if the early stopping condition is not met, return to step a, continue loading batch data, and enter the next round of training cycle;

[0042] 4) Final evaluation: After the early stopping condition is met, the training ends, the final evaluation is performed on the test set, and the final model is output.

[0043] Specifically, the loss calculation includes:

[0044] Mean square error loss: ensures overall pixel-level accuracy;

[0045] Perceptual loss: Use the pre-trained VGG network to extract features and compare the difference between the generated results and the target in the feature space to improve the visual quality;

[0046] Temporal consistency loss: ensures the temporal coherence of the video by comparing the optical flow between adjacent frames;

[0047] Pre-trained video generation model consistency loss: Ensure that the generated high-resolution video is consistent with the semantic understanding of the pre-trained video generation model.

[0048] Specifically, the video segmentation processing, feature caching mechanism and dynamic resolution adjustment in the inference optimization module are as follows:

[0049] Video segmentation processing: split long videos into segments of appropriate size for processing, and then merge the processing results;

[0050] Feature caching mechanism: Establish a feature caching mechanism for processing between adjacent frames to avoid repeated calculations;

[0051] Dynamic resolution adjustment: Dynamically adjust the processing resolution based on the characteristics of the input video and user needs.

[0052] Specifically, when the preprocessing module detects that the low-resolution video cannot be read correctly, or an error occurs in verifying the validity of its related text description, it will provide a corresponding error prompt to guide the user to make corrections.

[0053] In a second aspect, the present invention provides a method for improving video resolution based on a pre-trained video generation model, which is based on the video resolution improvement system based on a pre-trained video generation model described in any one of the first aspects, and the method comprises:

[0054] S1, obtaining the low-resolution video to be processed and its related text description and preprocessing it through a preprocessing module;

[0055] S2, processing the pre-processed low-resolution video and its related text description through a pre-trained video generation model to obtain high-level features;

[0056] S3, extracts additional visual features from low-resolution videos through feature extractor;

[0057] S4, inputting high-level features and additional visual features into the super-resolution module to obtain a preliminary high-resolution video;

[0058] S5. Input the high-level features and preliminary super-resolution images into the fusion module to obtain the final high-resolution video.

[0059] The method of the present invention utilizes the powerful capabilities of a large-scale pre-trained video generation model, combined with a specially designed super-resolution module, to achieve high-quality video resolution enhancement. The core steps of the method include: first receiving a low-resolution video input, and extracting high-level features through a pre-trained video generation model, which contain the semantic information and temporal relationships of the video. Then, a special feature extractor is used to extract additional visual features from the low-resolution video to capture more detailed information. Next, these extracted features are input into a specially designed super-resolution module, which generates a preliminary high-resolution video through a multi-layer convolutional neural network and an upsampling layer. Finally, through an innovative fusion module, the output of the pre-trained model and the output of the super-resolution module are intelligently merged to generate the final high-resolution video output.

[0060] The method of the present invention also includes a series of innovative technical details, such as the special design of the super-resolution module, the multi-head attention mechanism used in the fusion module, and the loss function design that comprehensively considers multiple factors. In addition, the present invention also proposes a unique training strategy, including phased training, progressive resolution increase, and dynamic batch size adjustment techniques to optimize model performance and training efficiency.

[0061] In the inference stage, the present invention uses optimization techniques such as video segmentation processing and feature caching mechanism to improve the efficiency of processing long videos. This method can be applied to a variety of practical scenarios, such as restoration and enhancement of old videos, quality improvement of video streaming services, image enhancement of video surveillance systems, and resolution improvement of medical images.

[0062] The main purpose of the present invention is to provide an innovative video resolution enhancement method and system, which fully utilizes the powerful capabilities of pre-trained video generation models and combines them with specially designed super-resolution techniques to generate high-quality, temporally coherent high-resolution videos.

[0063] The method of the present invention can not only effectively improve the spatial resolution of the video, but also maintain the temporal consistency and semantic integrity of the video. By integrating the advanced feature extraction capabilities of the pre-trained model and the customized super-resolution module, the method can handle various complex scenes and dynamic content, overcoming the limitations of traditional methods in dealing with fast motion and complex textures.

[0064] In addition, the present invention also includes a complete system implementation, covering the entire process from input processing to final output, and provides innovative training strategies and inference optimization techniques to improve processing efficiency and output quality. This method and system is expected to bring important application value in multiple fields such as video enhancement and content restoration.

[0065] The present invention provides an efficient and high-quality video resolution enhancement solution by innovatively combining a pre-trained video generation model and specialized super-resolution technology, which is expected to bring important breakthroughs in the field of video processing and enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 is a schematic diagram of a video resolution enhancement system based on a pre-trained video generation model provided by an embodiment of the present invention;

[0068] Figure 2 is a schematic diagram of the structure of a pre-trained video generation model provided in an embodiment of the present invention;

[0069] Figure 3 is a schematic diagram of the structural design of a super-resolution module provided by an embodiment of the present invention;

[0070] Figure 4 is a schematic diagram of the structure of a super-resolution module provided by an embodiment of the present invention;

[0071] Figure 5 is a schematic diagram of the structural design of the fusion module provided in an embodiment of the present invention;

[0072] Figure 6 is a schematic diagram of the structure of a fusion module provided in an embodiment of the present invention;

[0073] Figure 7 is a schematic diagram of a preset training strategy provided by an embodiment of the present invention;

[0074] Figure 8 It is a schematic diagram of training a model by combining a training module provided by an embodiment of the present invention with a preset training strategy;

[0075] Fig. 9 is a schematic diagram of a method for improving video resolution based on a pre-trained video generation model provided according to an embodiment of the present invention;

[0076] Fig.10 is a structural schematic diagram of a video resolution enhancement device based on a pre-trained video generation model provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The present invention can be explained in detail through the following embodiments. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "front", "back", "left", "right", etc. indicating directions or positional relationships, they only correspond to the drawings of the present application for the convenience of describing the present invention, and do not indicate or imply that the device or element referred to must have a specific direction.

[0078] Embodiment 1

[0079] refer to Figure 1 According to this embodiment, a video resolution enhancement system based on a pre-trained video generation model is disclosed, including the following modules:

[0080] A pre-trained video generation model is used to extract high-level features from low-resolution videos and their related text descriptions; the pre-trained video generation model adopts the Diffusion Transformer architecture, uses the PixArt-a open source high-quality text-to-image generation model, and is expanded into a video generation model by adding a temporal attention layer; the high-level features include: video content, video structure, and motion pattern; the related text description includes: video title, introduction, tags, or content description;

[0081] The pre-trained video generation model is one of the core components of the system. The model is pre-trained on massive video data and has powerful video understanding and generation capabilities. It can extract high-level semantic features from low-resolution videos and capture complex spatiotemporal relationships and content information. In this system, the pre-trained video generation model is mainly responsible for the initial understanding of video content and provides rich feature representation for subsequent processing.

[0082] The system inputs low-resolution videos into a pre-trained video generation model, which is usually pre-trained on large-scale video datasets and has strong video understanding capabilities. The processing process includes:

[0083] a) Video frame extraction: The system first decomposes the video into a series of frames;

[0084] b) Feature encoding: Each frame passes through the model’s encoder network to generate a high-dimensional feature representation;

[0085] c) Temporal modeling: The model considers the temporal relationship between frames and captures the dynamic characteristics of the video;

[0086] d) Semantic understanding: Combined with text description, the model performs a deeper semantic understanding.

[0087] The output high-level features include information such as video content, video structure, motion patterns, etc., laying the foundation for subsequent processing.

[0088] The pre-trained video generation model structure is as follows Figure 2 As shown, the model adopts the Diffusion Transformer (DiT) architecture, uses PixArt-a open source high-quality text-to-image generation model, and extends it to a video generation model by adding a temporal attention layer.

[0089] PixArt-a is a Transformer-based text-to-image generation model developed by researchers from Huawei Noah's Ark Lab, Dalian University of Technology, and the University of Hong Kong. This model is able to generate high-quality images up to 4K resolution directly from text prompts.

[0090] The feature extractor is used to extract additional visual features from the low-resolution video; the additional visual features include: multi-scale features, edge details, texture information, and motion information features.

[0091] Although the pre-trained video generation model can provide rich high-level features, in order to better reconstruct details, the system also uses a specialized feature extractor to process low-resolution videos. The feature extractor is a specially designed neural network module that extracts additional visual features from low-resolution videos. These visual features may include multi-scale features, texture information, edge details, motion information, etc. The feature extractor is designed to supplement the output of the pre-trained video generation model and provide more fine-grained visual information, which is critical for generating high-quality high-resolution videos.

[0092] The feature extractor includes the following units:

[0093] a) Multi-scale feature extraction unit: Use convolution kernels of different sizes to extract multi-scale features. A multi-layer convolution-pooling structure similar to VGGNet can be used to extract features at different depths.

[0094] b) Edge detail extraction unit: Apply specific edge detection algorithms to capture the edge and contour information features in the image. Apply specific edge detection algorithms, such as Sobel, Canny, etc., to extract edge information in the image. Special convolution kernels can be used to implement edge detection.

[0095] c) Texture information extraction unit: Use texture analysis technology to extract texture patterns in the video. Use texture analysis techniques such as LBP, GLCM, etc. to extract texture features in video frames. Convolutional layers and fully connected layers can also be used to model texture features.

[0096] Local Binary Pattern (LBP) and Gray Level Co-occurrence Matrix (GLCM) are methods used in the traditional computer vision field to extract image texture features and will not be described in detail here.

[0097] d) Motion information extraction unit: Analyze the changes between adjacent frames, extract motion information, use optical flow estimation algorithm or convolutional neural network to predict the motion between adjacent frames, and output features containing motion information.

[0098] These visual features complement the output of the pre-trained video generation model and provide more fine-grained visual information.

[0099] A super-resolution module that generates preliminary high-resolution videos from high-level features and additional visual features;

[0100] The super-resolution module is one of the core components of this system, responsible for mapping low-resolution features to high-resolution space. Its design incorporates the latest deep learning techniques, including residual learning, dense connections, and attention mechanisms.

[0101] refer to Figure 3 ,This embodiment provides a schematic diagram of a structural design of a super-resolution module, which allows the module to gradually extract and process features, and then generate high-quality high-resolution video frames through upsampling and detail reconstruction. The residual block helps to learn subtle feature differences, the upsampling layer is responsible for increasing the spatial resolution, and the attention layer helps the model focus on important image areas.

[0102] Specifically, the super-resolution module is a specially designed deep neural network for mapping low-resolution features to high-resolution space. It receives high-level features from the output of the pre-trained video generation model and additional visual features from the feature extractor, and generates preliminary high-resolution video frames through a series of convolutional layers, upsampling operations, and non-linear activation functions. The design of the super-resolution module incorporates the latest super-resolution technology and can effectively reconstruct details and textures. This process includes:

[0103] a) Feature fusion: Fusion of high-level features and additional visual features;

[0104] b) Feature upsampling: Through a series of convolution and upsampling operations, the spatial resolution of the feature map is gradually increased;

[0105] c) Non-linear mapping: Use multi-layer non-linear activation functions to learn complex low-resolution to high-resolution mapping relationships;

[0106] d) Detail reconstruction: Reconstruct image details through specially designed network structures (such as residual blocks, dense connections, etc.).

[0107] The output is preliminary high-resolution video frames, which have significantly improved clarity and detail, and then the preliminary high-resolution video is synthesized from the video frames.

[0108] refer to Figure 4 This embodiment provides a specific structural implementation of a super-resolution module, including the following sub-modules:

[0109] 1) Input submodule: used to receive input features; the input features include high-level features output by the pre-trained video generation model and additional visual features output by the feature extractor;

[0110] 2) Feature extraction and processing submodule: used to extract high-level features and additional visual features through three consecutive convolutional layers, including three convolutional layers, where the first two convolutional layers are followed by ReLU activation functions to introduce nonlinearity, while the third convolutional layer is not followed by ReLu activation function;

[0111] 3) Upsampling and detail reconstruction submodule: used to gradually increase the spatial resolution of the feature map through convolution and upsampling operations, including two PixelShuffle upsampling layers and three convolution layers. The first PixelShuffle upsampling layer is used for upsampling, which can reconstruct high-frequency details without significantly increasing the computational complexity. It is followed by two convolution layers, in which the first convolution layer is followed by a ReLU activation function, while the second convolution layer is not followed by an activation function. Then, PixelShuffle upsampling is applied again to further improve the resolution. The last convolution layer is used for the final detail adjustment to generate a preliminary high-resolution video.

[0112] 4) Output submodule: used to output the preliminary high-resolution video frames after the above processing.

[0113] This structural diagram clearly shows the internal structure of the super-resolution module, including: multiple convolutional layers for feature extraction and processing; ReLU activation function for introducing nonlinearity; PixelShuffle upsampling layer for increasing spatial resolution; and the final convolutional layer for generating the final high-resolution output. This structural design allows the module to gradually extract and process features, and then generate high-quality preliminary high-resolution video frames through upsampling and detail reconstruction. Through this detailed structural diagram, we can better understand how the super-resolution module achieves the conversion process from low resolution to high resolution.

[0114] The fusion module is used to fuse the high-level features and the preliminary high-resolution video to generate the final high-resolution video.

[0115] The role of the fusion module is to integrate the information from the pre-trained video generation model and the output of the super-resolution module to generate the final high-resolution output. Its core is an innovative multi-head attention mechanism that can adaptively learn the relationship between different features.

[0116] refer to Figure 5 This embodiment provides a schematic diagram of a structural design of a fusion module, which allows the module to effectively integrate features from different sources while maintaining high-resolution details. The multi-head attention mechanism enables the module to learn complex feature interactions, while the residual connection ensures the preservation of the original high-frequency information.

[0117] Specifically, the fusion module is a key component of the system, and its goal is to organically combine all the information from the previous modules to generate the final high-quality output, i.e., the final high-resolution video. The processing process includes:

[0118] a) Attention mechanism: Use a multi-head attention mechanism to learn the relationship and importance between different features.

[0119] b) Adaptive fusion: Adaptively adjust the fusion strategy according to the characteristics of different regions.

[0120] c) Temporal consistency processing: Consider the relationship between adjacent frames to ensure the temporal coherence of the video.

[0121] d) Detail enhancement: Apply detail enhancement technology to further improve the clarity and realism of the image.

[0122] refer to Figure 6 This embodiment provides a specific structural implementation of the fusion module, including the following submodules:

[0123] An input submodule, for receiving the feature output of the pre-trained video generation model and the output of the super-resolution module;

[0124] The feature processing submodule is used to fuse the feature output of the pre-trained video generation model and the output of the super-resolution module; it includes a feature mapping layer, a multi-head attention mechanism, and a feature fusion layer. The feature mapping layer maps the output of the input submodule to the same feature space, the multi-head attention mechanism captures the relationship between the features output by the feature mapping layer, and the feature fusion layer integrates the output of the multi-head attention mechanism.

[0125] A detail enhancement submodule, which is used to enhance the detail information of the image; it includes at least two convolutional layers and a ReLU activation function and a residual connection, wherein the convolutional layer receives the output of the feature processing submodule and performs a convolution operation, the ReLU activation function performs a nonlinear transformation on the output of the convolutional layer, and the residual connection directly adds the output of the super-resolution module to the output of the detail enhancement part;

[0126] The output submodule is used to generate the final high-resolution video frame through an output convolutional layer.

[0127] This structure diagram clearly shows the workflow of the fusion module:

[0128] 1) First, high-level features and preliminary high-resolution videos are processed through feature mapping layers;

[0129] 2) Use multi-head attention mechanism to capture the complex relationship between two features;

[0130] 3) Integrate the output of the attention mechanism through the feature fusion layer;

[0131] 4) Use multiple convolutional layers and activation functions for further feature processing and detail enhancement;

[0132] 5) Introducing residual connections to directly join the preliminary high-resolution video to retain more detailed information;

[0133] 6) Finally, the final high-resolution video frame is generated through the output convolution layer.

[0134] The video resolution enhancement system based on the pre-trained video generation model in this embodiment further includes:

[0135] A preprocessing module that checks whether the input low-resolution video can be read correctly and verifies the validity of its associated text description;

[0136] The preprocessing module is responsible for preliminary processing of the input low-resolution video and related text descriptions to ensure that the data format and quality meet the system requirements. The system first receives the low-resolution video provided by the user. This can be a video file in various formats, such as MP4, AVI, etc. At the same time, the system can also receive text descriptions related to the video content. These descriptions can be the title, introduction, tags, or more detailed content descriptions of the video. The text information serves as an additional input to help enhance the system's understanding of the video content.

[0137] In this module, the system will perform a preliminary check on the input to ensure that the video file can be read correctly and verify the validity of the text description. If any problems are found, the system will provide corresponding error prompts to guide the user to correct them.

[0138] The system also includes two auxiliary modules: a training module and an inference optimization module. The training module is used for training and optimizing the models in the pre-trained video generation model, the super-resolution module and the fusion module. The inference optimization module is used to improve the efficiency of model inference in the pre-trained video generation model, the super-resolution module and the fusion module.

[0139] A training module, used to train the model using a preset training strategy; the preset training strategy includes phased training, progressive resolution increase, and dynamic batch size adjustment;

[0140] refer to Figure 7 The training module of this embodiment adopts an innovative preset training strategy, including:

[0141] 1) Phased training: First, freeze the pre-trained video generation model and only train the super-resolution module and fusion module; then gradually unfreeze the layers of the pre-trained video generation model and perform global fine-tuning;

[0142] 2) Progressive resolution increase: training starts from a low resolution and gradually increases to the target resolution, which helps the model gradually learn multi-scale features;

[0143] 3) Dynamic batch size adjustment: Dynamically adjust the batch size according to the current training resolution to optimize memory usage and training efficiency.

[0144] This training strategy allows the model to effectively learn at different stages and scales while maintaining training efficiency through dynamic adjustments.

[0145] refer to Figure 8 This embodiment provides a schematic diagram of the entire training process, including: detailed data preparation steps to ensure the quality of training data; model initialization process to prepare for training; a complete training cycle, including forward propagation, loss calculation, back propagation and parameter update; regular verification and model preservation strategy to ensure optimal performance; setting an early stopping mechanism to avoid overfitting; and finally evaluation on the test set to ensure the generalization ability of the model.

[0146] Specifically, this embodiment provides detailed steps for the training module to train the model in combination with a preset training strategy:

[0147] 1) Data preparation: prepare low-resolution video datasets, high-resolution video datasets and related text descriptions, preprocess the datasets, including normalization, cropping, rotation and other operations, and then split the datasets into training sets and validation sets;

[0148] 2) Model initialization: initializing model parameters and setting hyperparameters, including learning rate and batch size;

[0149] 3) Training cycle, including: a. Loading batch data; b. Performing forward propagation and calculating model output; c. Calculating losses, including mean square error loss, perceptual loss, time consistency loss, etc.; d. Performing back propagation and calculating gradients; e. Updating model parameters to reduce losses; f. Determining whether the preset validation interval is reached. If it is within the validation interval, the model performance is evaluated on the validation set. If the model performance is improved, the current model is saved as the best model. If the performance is not improved, check whether the early stopping condition is met. If not, return to step a, continue loading batch data, and enter the next round of training cycle;

[0150] The early stopping condition is usually set to prevent the model from overfitting; the early stopping condition in this embodiment includes: the number of iterations reaches a preset maximum number of iterations or the performance on the validation set does not improve within the preset number of iterations;

[0151] The specific values ​​of the maximum number of iterations and the preset number of iterations are selected according to the accuracy requirements of the actual training. This is the existing technology and will not be repeated here.

[0152] 4) Final evaluation: After the early stopping condition is met, the training ends, the final evaluation is performed on the test set, and the final model is output.

[0153] The loss calculation uses a multi-component loss function to fully optimize the model performance. The loss function includes:

[0154] Mean square error loss: ensures overall pixel-level accuracy;

[0155] Perceptual loss: Use the pre-trained VGG network to extract features and compare the difference between the generated results and the target in the feature space to improve the visual quality;

[0156] Temporal consistency loss: ensures the temporal coherence of the video by comparing the optical flow between adjacent frames;

[0157] Pre-trained video generation model consistency loss: Ensure that the generated high-resolution video is consistent with the semantic understanding of the pre-trained video generation model.

[0158] This multi-component loss function design ensures that the generated results meet high standards in multiple aspects such as pixel accuracy, visual quality, temporal coherence, and semantic consistency.

[0159] The inference optimization module is used to improve the processing efficiency of the module through video segmentation processing, feature caching mechanism and dynamic resolution adjustment.

[0160] In order to improve the processing efficiency in practical applications, the inference optimization module of this embodiment adopts the following inference optimization technology:

[0161] 1) Video segmentation processing: For long videos, split them into segments of appropriate size for processing, and then merge the results. This can reduce memory usage and improve parallel processing capabilities;

[0162] 2) Feature caching mechanism: Establish a feature caching mechanism for processing between adjacent frames to avoid repeated calculations and improve inference efficiency;

[0163] 3) Dynamic resolution adjustment: Dynamically adjust the processing resolution according to the characteristics of the input video and user needs to achieve a balance between quality and speed;

[0164] Through this series of carefully designed steps, the system is able to generate a final output video that retains the semantic content of the original video while having high-resolution details. The entire process is end-to-end, and each step works closely together to achieve high-quality video resolution enhancement.

[0165] These key technical details together constitute the core of the present invention, which enables it to achieve efficient video resolution improvement while ensuring high-quality output. Through these innovative designs, the system can adapt to various complex video scenes and provide users with high-quality video enhancement experience.

[0166] It is worth noting that in the above-mentioned embodiment of the video resolution enhancement system based on the pre-trained video generation model, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0167] Embodiment 2

[0168] refer to Fig. 9 This embodiment discloses a method for improving video resolution based on a pre-trained video generation model, which is used in the video resolution improvement system based on a pre-trained video generation model in the above embodiment, and includes the following steps:

[0169] S1, obtaining the low-resolution video to be processed and its related text description and preprocessing it through a preprocessing module;

[0170] The system first receives a low-resolution video provided by the user. This can be a video file in various formats, such as MP4, AVI, etc. At the same time, the system can also receive text descriptions related to the video content. These related text descriptions can be the title, introduction, tags, or more detailed content descriptions of the video. Text information serves as additional input to help enhance the system's understanding of the video content.

[0171] In this step, the system will perform a preliminary check on the input through the preprocessing module to ensure that the video file can be read correctly and verify the validity of the text description. If any problems are found, the system will provide corresponding error prompts to guide the user to correct them.

[0172] S2, processing the pre-processed low-resolution video and its related text description through a pre-trained video generation model to obtain high-level features;

[0173] The system inputs low-resolution videos into a pre-trained video generation model, which is usually pre-trained on large-scale video datasets and has strong video understanding capabilities. The processing process includes:

[0174] a) Video frame extraction: The system first decomposes the video into a series of frames;

[0175] b) Feature encoding: Each frame passes through the model’s encoder network to generate a high-dimensional feature representation;

[0176] c) Temporal modeling: The model considers the temporal relationship between frames and captures the dynamic characteristics of the video;

[0177] d) Semantic understanding: Combined with text description, the model performs a deeper semantic understanding.

[0178] The output high-level features include information such as video content, video structure, motion patterns, etc., laying the foundation for subsequent processing.

[0179] S3, extracts additional visual features from low-resolution videos through feature extractor;

[0180] The additional visual features include: multi-scale features, edge details, texture information, and motion information features;

[0181] Step S3 specifically includes:

[0182] S31, extract multi-scale features using convolution kernels of different sizes through the multi-scale feature extraction unit, and use a multi-layer convolution-pooling structure similar to VGGNet to extract multi-scale features at different depths;

[0183] S32, using a specific edge detection algorithm to capture edge and contour information features in the image through an edge detail extraction unit; the specific edge detection algorithm includes Sobel and Canny;

[0184] S33, extracting texture features in the video by using a texture analysis technique through a texture information extraction unit or modeling texture features by using a convolutional layer and a fully connected layer; the texture analysis technique includes LBP and GLCM;

[0185] S34. Predict the motion between adjacent frames using an optical flow estimation algorithm or a convolutional neural network through a motion information extraction unit, and output the contained motion information features.

[0186] S4, inputting high-level features and additional visual features into the super-resolution module to obtain a preliminary high-resolution video;

[0187] Step S4 specifically includes:

[0188] S41, receiving high-level features and additional visual features through a super-resolution input submodule;

[0189] S42, extracting high-level features and additional visual features using three consecutive convolutional layers through a feature extraction and processing submodule, including three convolutional layers, wherein the first two convolutional layers are followed by a ReLU activation function to introduce nonlinearity, while the third convolutional layer is not followed by a ReLu activation function;

[0190] S43, using convolution and upsampling operations through upsampling and detail reconstruction submodules to gradually increase the spatial resolution of the feature map, including two PixelShuffle upsampling layers and three convolution layers, wherein upsampling is first performed through the first PixelShuffle upsampling layer, which can reconstruct high-frequency details without significantly increasing the computational complexity, followed by two convolution layers, wherein the first convolution layer is followed by a ReLU activation function, while the second convolution layer is not followed by an activation function, and then PixelShuffle upsampling is applied again to further improve the resolution, and the last convolution layer is used for final detail adjustment to generate a preliminary high-resolution video;

[0191] S44, outputting the preliminary high-resolution video frame after the above processing through the super-resolution output submodule.

[0192] S5. Input the high-level features and preliminary super-resolution images into the fusion module to obtain the final high-resolution video.

[0193] Step S5 specifically includes:

[0194] S51, receiving high-level features and preliminary high-resolution video frames through a fusion input submodule;

[0195] S52, fusing high-level features and preliminary high-resolution video frames through a feature processing submodule; comprising a feature mapping layer, a multi-head attention mechanism, and a feature fusion layer, wherein the feature mapping layer maps the output of the input submodule to the same feature space, the multi-head attention mechanism captures the relationship between the features output by the feature mapping layer, and the feature fusion layer integrates the output of the multi-head attention mechanism;

[0196] S53, enhancing detail information of the image through a detail enhancement submodule; comprising at least two convolutional layers and a ReLU activation function and a residual connection, wherein the convolutional layer receives the output of the feature processing submodule and performs a convolution operation, the ReLU activation function performs a nonlinear transformation on the output of the convolutional layer, and the residual connection directly adds the preliminary high-resolution video frame to the output of the detail enhancement part;

[0197] S54, generate the final high-resolution video frame using an output convolutional layer through the fusion output submodule.

[0198] The method of this embodiment utilizes the powerful capabilities of a large-scale pre-trained video generation model, combined with a specially designed super-resolution module, to achieve high-quality video resolution enhancement. The core steps of this method include: first receiving a low-resolution video input, extracting high-level features through a pre-trained video generation model, which contain the semantic information and temporal relationships of the video. Then, a special feature extractor is used to extract additional visual features from the low-resolution video to capture more detailed information. Next, these extracted features are input into a specially designed super-resolution module, which generates a preliminary high-resolution video through a multi-layer convolutional neural network and an upsampling layer. Finally, through an innovative fusion module, the output of the pre-trained model and the output of the super-resolution module are intelligently merged to generate the final high-resolution video output.

[0199] The method of this embodiment also includes a series of innovative technical details, such as the special design of the super-resolution module, the multi-head attention mechanism used in the fusion module, and the loss function design that comprehensively considers multiple factors. In addition, the present invention also proposes a unique training strategy, including phased training, progressive resolution increase, and dynamic batch size adjustment techniques to optimize model performance and training efficiency.

[0200] In the inference stage, the present invention uses optimization techniques such as video segmentation processing and feature caching mechanism to improve the efficiency of processing long videos. This method can be applied to a variety of practical scenarios, such as restoration and enhancement of old videos, quality improvement of video streaming services, image enhancement of video surveillance systems, and resolution improvement of medical images.

[0201] The main purpose of this embodiment is to provide an innovative video resolution enhancement method and system, which fully utilizes the powerful capabilities of the pre-trained video generation model and combines it with a specially designed super-resolution technology to generate high-quality, temporally coherent high-resolution video.

[0202] The method of this embodiment can not only effectively improve the spatial resolution of the video, but also maintain the temporal consistency and semantic integrity of the video. By integrating the advanced feature extraction capabilities of the pre-trained model and the customized super-resolution module, the method can handle various complex scenes and dynamic content, overcoming the limitations of traditional methods in dealing with fast motion and complex textures.

[0203] In addition, the present invention also includes a complete system implementation, covering the entire process from input processing to final output, and provides innovative training strategies and inference optimization techniques to improve processing efficiency and output quality. This method and system is expected to bring important application value in multiple fields such as video enhancement and content restoration.

[0204] This embodiment provides an efficient and high-quality video resolution enhancement solution by innovatively combining a pre-trained video generation model and specialized super-resolution technology, which is expected to bring important breakthroughs in the field of video processing and enhancement.

[0205] Embodiment 3

[0206] refer to Fig.10 , Fig.10 : is a structural diagram of the video resolution enhancement device based on the pre-trained video generation model of this embodiment. The video resolution enhancement device 20 based on the pre-trained video generation model of this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above method embodiment are implemented. Alternatively, when the processor 21 executes the computer program, the functions of each module / unit in the above device embodiments are implemented.

[0207] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the video resolution enhancement device 20 based on the pre-trained video generation model. For example, the computer program may be divided into the modules in Embodiment 2. For the specific functions of each module, please refer to the working process of the device described in the above embodiment, which will not be repeated here.

[0208] The video resolution enhancement device 20 based on the pre-trained video generation model may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the video resolution enhancement device 20 based on the pre-trained video generation model, and does not constitute a limitation on the video resolution enhancement device 20 based on the pre-trained video generation model, and may include more or fewer components than shown in the figure, or combine certain components, or different components, for example, the video resolution enhancement device 20 based on the pre-trained video generation model may also include input and output devices, network access devices, buses, and the like.

[0209] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the video resolution enhancement device 20 based on the pre-trained video generation model, and uses various interfaces and lines to connect various parts of the video resolution enhancement device 20 based on the pre-trained video generation model.

[0210] The memory 22 can be used to store the computer program and / or module. The processor 21 realizes various functions of the video resolution enhancement device 20 based on the pre-trained video generation model by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart MediaCard, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0211] Wherein, if the module / unit integrated in the video resolution enhancement device 20 based on the pre-trained video generation model is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 21, the steps of the above-mentioned method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunication signals.

[0212] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0213] The parts of the present invention that are not described in detail are prior art. It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention; therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and it is intended that all changes that fall within the meaning and scope of equivalent elements are included in the present invention.

Claims

1. A video resolution enhancement system based on a pre-trained video generation model, characterized in that: include: Pre-trained video generation models for extracting high-level features from low-resolution videos and their associated text descriptions; The pre-trained video generation model adopts the Diffusion Transformer architecture, uses PixArt-a open source high-quality text-to-image generation model, and expands it into a video generation model by adding a temporal attention layer; The high-level features include: video content, video structure, and motion mode; the related text description includes: video title, introduction, tags, or content description; A feature extractor, used to extract additional visual features from the low-resolution video; the additional visual features include: multi-scale features, edge details, texture information, and motion information features; A super-resolution module that generates preliminary high-resolution videos from high-level features and additional visual features; The super-resolution module includes the following submodules: Super-resolution input submodule: used to receive high-level features and additional visual features; Feature extraction and processing submodule: used to extract high-level features and additional visual features through three consecutive convolutional layers, including three convolutional layers, where the first two convolutional layers are followed by ReLU activation functions to introduce nonlinearity, while the third convolutional layer is not followed by ReLu activation function; Upsampling and detail reconstruction submodule: used to gradually increase the spatial resolution of the feature map through convolution and upsampling operations, including two PixelShuffle upsampling layers and three convolution layers. The first PixelShuffle upsampling layer is used for upsampling, which can reconstruct high-frequency details without significantly increasing the computational complexity. It is followed by two convolution layers, in which the first convolution layer is followed by a ReLU activation function, while the second convolution layer is not followed by an activation function. Then, PixelShuffle upsampling is applied again to further improve the resolution. The last convolution layer is used for final detail adjustment to generate a preliminary high-resolution video. Super-resolution output submodule: used to output preliminary high-resolution video frames after the above processing; The fusion module is used to fuse the high-level features and the preliminary high-resolution video to generate the final high-resolution video.

2. The video resolution enhancement system according to claim 1, characterized in that: The video resolution enhancement system further comprises: A preprocessing module that checks whether the input low-resolution video can be read correctly and verifies the validity of its associated text description; A training module, used to train the pre-trained video generation model, the super-resolution module, and the models in the fusion module through a preset training strategy; The inference optimization module is used to improve the efficiency of model inference in the pre-trained video generation model, super-resolution module and fusion module through video segmentation processing, feature caching mechanism and dynamic resolution adjustment.

3. The video resolution enhancement system according to claim 1, characterized in that: The feature extractor comprises the following units: Multi-scale feature extraction unit: used to extract multi-scale features using convolution kernels of different sizes, and uses VGGNet's multi-layer convolution-pooling structure to extract multi-scale features at different depths; Edge detail extraction unit: used to capture edge and contour information features in the image using a specific edge detection algorithm; the specific edge detection algorithm includes Sobel and Canny; Texture information extraction unit: used to extract texture features in the video using texture analysis technology or to model texture features by using convolutional layers and fully connected layers; The texture analysis techniques include LBP and GLCM; Motion information extraction unit: used to predict the motion between adjacent frames using an optical flow estimation algorithm or a convolutional neural network, and output the contained motion information features.

4. The video resolution enhancement system according to claim 1, characterized in that: The fusion module includes the following submodules: Fusion input submodule: used to receive high-level features and preliminary high-resolution video frames; Feature processing submodule: used to fuse high-level features and preliminary high-resolution video frames; Including feature mapping layer, multi-head attention mechanism and feature fusion layer, The feature mapping layer maps the output of the input submodule to the same feature space, the multi-head attention mechanism captures the relationship between the features output by the feature mapping layer, and the feature fusion layer integrates the output of the multi-head attention mechanism; Detail enhancement submodule: used to enhance the detail information of the image; including at least two convolutional layers and ReLU activation functions and residual connections, wherein the convolutional layer receives the output of the feature processing submodule and performs a convolution operation, the ReLU activation function performs a nonlinear transformation on the output of the convolutional layer, and the residual connection directly adds the preliminary high-resolution video frame to the output of the detail enhancement part; Fusion output submodule: used to generate the final high-resolution video frame through an output convolutional layer.

5. The video resolution enhancement system according to claim 2, characterized in that: The preset training strategy includes phased training, progressive resolution increase, and dynamic batch size adjustment; Phased training: First, freeze the pre-trained video generation model and only train the super-resolution module and fusion module; then gradually unfreeze the layers of the pre-trained video generation model and perform global fine-tuning; Progressive resolution increase: Start training from a low resolution, gradually increase to the target resolution, and gradually learn multi-scale features; Dynamic batch size adjustment: Dynamically adjust the batch size according to the current training resolution to optimize memory usage and training efficiency.

6. The video resolution enhancement system according to claim 5, characterized in that: The specific steps of the training module training the pre-trained video generation model, the super-resolution module and the model in the fusion module by using a preset training strategy include: 1) Data preparation: prepare low-resolution video datasets, high-resolution video datasets and related text descriptions, preprocess the datasets, including normalization, cropping, and rotation operations, and then split the datasets into training sets and validation sets; 2) Model initialization: Initialize model parameters and set hyperparameters, including learning rate and batch size; 3) Training cycle, including: a. Loading batch data; b. Performing forward propagation to calculate model output; c. Loss calculation; d. Performing back propagation to calculate gradient; e. Updating model parameters to reduce loss; f. Determining whether the preset validation interval has been reached. If it is within the validation interval, the model performance is evaluated on the validation set. If the model performance has improved, the current model is saved as the best model. If the performance has not improved, check whether the early stopping condition is met; if the early stopping condition is not met, return to step a, continue loading batch data, and enter the next round of training cycle; 4) Final evaluation: After the early stopping condition is met, the training ends, the final evaluation is performed on the test set, and the final model is output.

7. The video resolution enhancement system according to claim 6, characterized in that: The loss calculation includes: Mean square error loss: ensures overall pixel-level accuracy; Perceptual loss: Use the pre-trained VGG network to extract features and compare the difference between the generated results and the target in the feature space to improve the visual quality; Temporal consistency loss: ensures the temporal coherence of the video by comparing the optical flow between adjacent frames; Pre-trained video generation model consistency loss: Ensure that the generated high-resolution video is consistent with the semantic understanding of the pre-trained video generation model.

8. The video resolution enhancement system according to claim 2, characterized in that: The video segmentation processing, feature caching mechanism and dynamic resolution adjustment in the inference optimization module are specifically as follows: Video segmentation processing: split long videos into segments of appropriate size for processing, and then merge the processing results; Feature caching mechanism: Establish a feature caching mechanism for processing between adjacent frames to avoid repeated calculations; Dynamic resolution adjustment: Dynamically adjust the processing resolution based on the characteristics of the input video and user needs.

9. The video resolution enhancement system according to claim 2, characterized in that: When the preprocessing module detects that the low-resolution video cannot be correctly read or the validity of the relevant text description cannot be verified, a corresponding error prompt will be provided.

10. A method for improving video resolution based on a pre-trained video generation model, a system for improving video resolution based on a pre-trained video generation model according to any one of claims 1 to 9, characterized in that: The method comprises: S1, obtaining the low-resolution video to be processed and its related text description and preprocessing it through a preprocessing module; S2, processing the pre-processed low-resolution video and its related text description through a pre-trained video generation model to obtain high-level features; S3, extracts additional visual features from low-resolution videos through feature extractor; S4, inputting high-level features and additional visual features into the super-resolution module to obtain a preliminary high-resolution video; S5. Input the high-level features and preliminary super-resolution images into the fusion module to obtain the final high-resolution video.

Images