Multi-modal fusion real-time video super-resolution reconstruction method and system

Abstract

The invention relates to the technical field of computer vision and deep learning, in particular to a multi-modal fusion real-time video super-resolution reconstruction method and system, and the method comprises the following steps: obtaining a low-resolution video sequence; a CLIP model is adopted to extract visual features and text features, bimodal feature fusion is carried out, and guidance features are generated; performing feature alignment on the guidance features and the low-resolution video through a multi-modal fusion module; a lightweight residual module is adopted to extract high-quality features; fusing multi-frame features through inter-frame information flow propagation, and performing motion compensation by adopting adaptive Kalman filtering; the high-quality features and the information after feature fusion are combined to reconstruct a high-definition image, visual and text features are extracted by introducing a CLIP model, fusion of multi-modal information is achieved, the semantic understanding ability is enhanced, and the quality and detail representation of super-resolution reconstruction are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a multi-modal fusion real-time video super-resolution reconstruction method and system thereof. Background Art

[0002] With the rapid development of high-definition display technology, the resolution requirements for video content are increasing. However, due to transmission bandwidth limitations, storage capacity limitations, and physical constraints of acquisition equipment, a large amount of video content is still stored and transmitted at a lower resolution. Video super-resolution technology aims to reconstruct high-resolution video from low-resolution video to improve visual quality and detail performance.

[0003] Traditional video super-resolution methods mainly rely on interpolation algorithms or sample-based learning methods, which have significant limitations in processing complex textures and detail recovery. In recent years, deep learning methods have made significant progress in the field of video super-resolution, but existing methods still face the following problems: First, most methods rely only on visual information for super-resolution reconstruction, ignoring the auxiliary role of other modal information such as text; second, it is still a challenge to achieve real-time processing while ensuring super-resolution quality, especially for high-resolution videos such as 4K; finally, the problem of inter-frame consistency has not been fully solved in existing methods, resulting in jitter and incoherence in the reconstructed video in the temporal dimension.

[0004] In addition, existing deep learning methods usually have high computational complexity and are difficult to implement in real time on ordinary hardware, especially for high-resolution videos. Although some lightweight network designs have been proposed to reduce computational complexity, they often lead to a decrease in reconstruction quality. Therefore, how to achieve computationally efficient real-time processing while ensuring high-quality super-resolution reconstruction remains a key issue that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal fusion real-time video super-resolution reconstruction method and system thereof. The method adopts a lightweight three-stream structure based on adaptive Kalman filter prediction, combines visual and text features extracted by CLIP model, and realizes high-quality, real-time video super-resolution reconstruction through multimodal fusion, lightweight residual network and adaptive motion compensation and other technologies.

[0006] The present invention proposes a multi-modal fusion real-time video super-resolution reconstruction method, which includes the following steps: Acquire a low-resolution video sequence; The CLIP model is used to extract visual features and text features, and bimodal features are fused to generate guidance features; Performing feature alignment on the guidance feature and the low-resolution video through a multimodal fusion module; Use lightweight residual module to extract high-quality features; The multi-frame features are fused through inter-frame information flow propagation, and the adaptive Kalman filter is used for motion compensation; The high-quality features are combined with the information obtained by feature fusion to reconstruct a high-definition image.

[0007] Preferably, the step of extracting visual features and text features by the CLIP model specifically includes: Input the low-resolution video sequence into the ViT encoder and the ResNet network to extract visual features; Input the context description in the low-resolution video frame into the Text Encoder to extract text features; Calculate the similarity scores between visual features and text features through a cross-modal similarity encoder; The visual features and the text features are combined through a gated fusion module to generate guidance features.

[0008] Preferably, the multimodal fusion module dynamically integrates the guidance feature and the low-resolution video feature through a gating mechanism, and the specific steps are: Inputting the guidance features and the low-resolution video features into two parallel convolutional layers and ReLU layers; The outputs of the two parallel branches are added together and then input into another convolutional layer and ReLU layer; The output result is multiplied by the guidance gate in the guidance module to control the degree of fusion and generate alignment features.

[0009] Preferably, the residual path of the lightweight residual module adopts a channel separation convolution layer to reduce the computational complexity, and the specific steps are as follows: Convert video frames from Convert to 4 channels Tensor of ; Convert the tensor to a shape of A tensor of Indicates the number of channels; the number of channels is reduced to 2 through the convolution layer , and make two copies; Add three tensors of the same shape and pass them through the ReLU activation function; Change the tensor group shape from Transformed to , which is rearranged through the convolutional layer as

[0010] Expand the tensor channel by two times to get the number of channels to 2 The feature representation of .

[0011] Preferably, the specific steps of performing motion compensation by the adaptive Kalman filter include: Based on the deep features extracted by the feature learning network, a 3×3 convolutional neural network is used to calculate feature associations; The motion estimation of the low-resolution video sequence is performed through the adaptive model parameters to obtain the motion estimation value at time t+1; Sending the motion estimation value to a fusion module to obtain a Kalman motion estimation value; The motion compensation module is used to correct the low-resolution blocks after super-resolution and improve the consistency between frames.

[0012] Preferably, the adaptive Kalman filter includes a multi-branch multi-layer perceptron and a fusion module, and its Kalman motion estimation value is obtained by combining feature maps of different branches; the feature association module and the fusion module use layer normalization to process the output of each layer of the network.

[0013] Preferably, the lightweight residual module includes multiple densely connected residual blocks, each residual block includes layer normalization, a convolution layer and a Leaky-ReLU activation layer; a deformable convolution is provided in the residual block to extract local motion information of the video frame; dense connections are added between adjacent residual blocks to connect the input of each residual block with the output of all previous residual blocks.

[0014] Preferably, the method is performed through discriminator adversarial training, wherein the discriminator loss function consists of an adversarial loss function, a multi-scale loss function, a cycle consistency loss function, a guided loss function and a total variation regularization loss function; the multi-scale loss function is used to evaluate the reconstruction quality at different scales; and the cycle consistency loss function is used to ensure feature consistency.

[0015] Preferably, the discriminator is initialized using a pre-trained model extracted from a VGG network; the perceptual difference loss is calculated by extracting multi-layer features of the input image; and the feature learning network uses features output from the last convolutional layer of the ImageNet pre-trained model.

[0016] A real-time video super-resolution reconstruction system for performing multi-modal fusion of the method comprises: The feature extraction module is used to extract the visual features and text features of the low-resolution video using the CLIP model, and perform bimodal feature fusion to generate guidance features; A feature alignment module, used for aligning the guiding feature with the low-resolution video through a multimodal fusion module; Feature optimization module, used to extract high-quality features using a lightweight residual module; Motion compensation module, used to fuse multi-frame features through inter-frame information flow propagation and use adaptive Kalman filtering for motion compensation; An image reconstruction module, used to reconstruct a high-definition image by combining the high-quality features with information after feature fusion; The network training module is used to train and optimize the system through a generative adversarial network.

[0017] The present invention has the following beneficial effects: 1. By introducing the CLIP model to extract visual and text features, the fusion of multimodal information is achieved, the semantic understanding ability is enhanced, and the quality and detail expression of super-resolution reconstruction are improved. Compared with the method of single visual feature, the PSNR and SSIM indicators are improved by about 2dB and 0.05 respectively.

[0018] 2. The lightweight three-stream structure and channel separation convolution technology are adopted to significantly reduce the computational complexity, reducing the amount of calculation by more than 70% and the number of parameters by more than 50%, achieving real-time processing capabilities of 4K video with a frame rate of up to 30fps.

[0019] 3. The innovative introduction of adaptive Kalman filtering for motion compensation, combined with deep learning and classic signal processing technology, effectively solves the problem of inter-frame consistency, reduces jitter and artifacts in the time dimension, and improves video smoothness.

[0020] 4. A multi-component loss function system was designed to guide network learning from multiple angles, including adversarial loss, multi-scale loss, cycle consistency loss, etc., which comprehensively improved the visual quality of video super-resolution.

[0021] 5. The overall solution has high adaptability and can dynamically adjust processing parameters for different scenarios. It performs well in various application scenarios such as medical imaging, surveillance video, and live broadcast content. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is an overall flow chart of the multi-modal fusion real-time video super-resolution reconstruction method of the present invention.

[0023] Figure 2 It is a schematic diagram of the structure of the CLIP model in the present invention for extracting visual features and text features.

[0024] Figure 3 It is a structural diagram of the multimodal fusion module in the present invention.

[0025] Figure 4 It is a schematic diagram of the channel separation convolution layer structure of the lightweight residual module in the present invention.

[0026] Figure 5It is a schematic diagram of the process of performing motion compensation using adaptive Kalman filtering in the present invention.

[0027] Figure 6 It is a schematic diagram of the dense connection structure of the lightweight residual module in the present invention.

[0028] Figure 7 It is a structural block diagram of the multi-modal fusion real-time video super-resolution reconstruction system of the present invention.

[0029] Figure 8 It is a detailed structural diagram of the cross-modal attention mechanism of the CLIP model in the present invention.

[0030] Fig. 9 It is a calculation flow chart of the adaptive Kalman filter state update and prediction in the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Reference Figure 1 The present invention provides a multi-modal fusion real-time video super-resolution reconstruction method, which adopts a lightweight three-stream structure based on adaptive Kalman filter prediction and includes the following steps: Step 1: Obtain a low-resolution video sequence. Preferably, this step can directly obtain a low-resolution video through a video acquisition device, or extract an existing low-resolution video sequence from a video database. In this embodiment, the resolution of the low-resolution video sequence can be 720p (1280×720 pixels) or lower.

[0033] The lightweight three-stream structure based on adaptive Kalman filter prediction in the present invention refers to three main feature paths in the system that cooperate with each other but process in parallel. The first path is the visual feature stream, which contains two complementary branches: the ViT encoder branch is responsible for capturing the global semantic features and long-distance dependencies of the video frame, while the ResNet network branch focuses on extracting detailed features such as local textures and edges. The two branches work together, taking low-resolution video frames as input and outputting a fused rich visual feature representation.

[0034] The second path is the text feature flow, which is composed of the Text Encoder of the CLIP model. It specializes in processing text descriptions related to video content and extracting deep semantic features from them. This process takes video-related text descriptions as input and outputs high-dimensional text feature representations to provide semantic guidance for video super-resolution.

[0035] The third path is the temporal consistency flow, which integrates the adaptive Kalman filter and motion compensation modules to specifically process the temporal relationship and motion information between frames to ensure the consistency of the super-resolution results in the temporal dimension. This process receives the feature representation of consecutive video frames and outputs temporally consistent features after motion compensation.

[0036] The collaboration of these three paths is extremely sophisticated: the visual feature stream and the text feature stream first process the input data independently, and then perform deep feature fusion through the cross-modal similarity encoder and the gated fusion module to generate guidance features rich in semantic information. These guidance features are precisely integrated with the original low-resolution video features in the feature alignment module. At the same time, the temporal consistency stream receives the information after feature alignment, combines the historical frame information, and optimizes the consistency in the time dimension through the adaptive Kalman filter algorithm. Finally, the processing results of the three paths are unified and fused in the image reconstruction module to generate a super-resolution video with high spatial details and temporal smoothness.

[0037] This three-stream architecture design enables the system to simultaneously utilize spatial information, semantic information, and temporal information to achieve multi-dimensional, multi-modal comprehensive feature extraction and fusion. It is the core technical framework of the present invention for achieving high-quality real-time video super-resolution, which significantly improves processing efficiency while ensuring video quality.

[0038] Step 2: Use the CLIP (Contrastive Language-Image Pre-training) model to extract visual features and text features, and perform bimodal feature fusion to generate guidance features.

[0039] The text information acquisition in the present invention adopts a variety of flexible methods. First, the system can directly use the metadata information attached to the video file, such as the video title, description tags, etc. as text input. These metadata are usually provided by the creator when the video is generated or uploaded, and can accurately reflect the semantic information of the video content. Secondly, in specific application scenarios, pre-annotated scene description texts can be used. These descriptions are divided by timeline and accurately correspond to the video content. In addition, the system also supports the automatic generation of text descriptions through pre-trained scene description models, realizing real-time applications without human intervention.

[0040] In order to solve the problem of time alignment between text and video frames, the present invention designs a multi-level matching strategy. For text descriptions with timestamps (such as subtitles or scene annotations), the system accurately matches the text with the video frames within the corresponding time range according to the timestamp. When processing continuous video streams, a sliding time window (such as 5 seconds) mechanism is adopted to assign the same text description to all frames in the window, and the window updates the description as the video advances. The system can also intelligently detect the scene switching points or key frames of the video, and only update the text description at these key points. Ordinary frames between two key frames share the same text description, which improves processing efficiency.

[0041] In order to meet the real-time processing requirements, the present invention also implements a text caching mechanism and predictive text generation technology to prepare in advance the next text description that may be needed, ensuring that text features can be integrated with video frame features in a timely manner to minimize processing delays.

[0042] Reference Figure 2 and Figure 8 This step specifically includes: inputting the low-resolution video sequence into the ViT (VisionTransformer) encoder and the ResNet network to extract visual features; inputting the context description in the low-resolution video frame into the Text Encoder to extract text features; calculating the similarity score between the visual features and the text features through the cross-modal similarity encoder; and generating guidance features by combining the visual features and the text features through the gated fusion module.

[0043] In this embodiment, the dimension of the visual feature is ,in is the image batch size, is the image pixel size, is the number of characteristic channels. Preferably, It can be set to 256 to balance the feature expression ability and computational efficiency. The dimension of text features is , which matches the number of channels of visual features after dimensionality reduction.

[0044] The tensor dimension transformation in the lightweight residual module is one of the technical essences of the present invention, and its detailed dimension change process is further explained. The initial video frame input dimension is the batch size B multiplied by the height H, width W and RGB three channels, that is, the [B, H, W, 3] format. In the channel expansion stage, the system expands the number of channels from 3 to 4c (c=64) through a carefully designed convolution operation, generating a feature tensor of [B, H, W, 4c] dimensions.

[0045] The next reshape reorganization phase is a key step. The system rearranges the above tensor into the shape of [B, H, W, 4c, Choi], where Choi=4 is the channel selection factor. This transformation is achieved through the reshape operation, which does not change the total amount of data, but only adjusts the organizational structure to prepare for the subsequent channel separation convolution. In the channel dimensionality reduction phase, the number of channels is effectively reduced from 4c×Choi to 2c through innovative grouped convolution operations, and two identical copies are generated to form a feature tensor with three dimensions of [B, H, W, 2c].

[0046] The feature fusion stage adds the three tensors element-wise and processes them through the ReLU activation function, keeping the dimensions unchanged. Then, in the downsampling and shape transformation stage, the system performs spatial downsampling on the activated feature map, transforming the dimensions from [B, H, W, 2c] to [B, H / 2, W / 2, 2c], while completing the spatial resolution transformation. The channel rearrangement stage rearranges the features into the form of [B, H, W, c] through precise convolution operations, restoring the original spatial resolution but reducing the number of channels to c. The final channel expansion step increases the number of channels from c to 2c, producing the final output tensor [B, H, W, 2c].

[0047] This precise dimensionality transformation and channel operation design greatly reduces the computational complexity while maintaining sufficiently strong feature expression capabilities. It is the core technology for achieving lightweight and efficient processing.

[0048] The present invention innovatively designs a CLIP model cross-modal feature extraction and fusion algorithm, the detailed process is as follows: 1. Visual feature extraction: For the input low-resolution video frame , first extract visual features through the ViT encoder and ResNet dual-path network: , in, Represents the visual encoder combination. For the ResNet path, a five-layer structure with 3×3 convolution kernels is used, and batch normalization and ReLU activation functions are added between each layer; for the ViT path, the image is divided into Patch, where is the optimal segmentation size. After linear projection, each patch is input into the Transformer encoder. The Transformer contains 12 self-attention layers, each of which contains 8 attention heads.

[0049] 2. Text feature extraction: text description of video content , extract semantic features through Text Encoder , in, represents a text encoder, is the text feature, is the text feature dimension. TextEncoder is based on the Transformer architecture and contains 12 self-attention layers.

[0050] 3. Cross-modal attention calculation: In order to effectively integrate visual and text features, the present invention designs a cross-modal attention mechanism, such as Figure 8 As shown. First, calculate the similarity score between visual features and text features: , in, is the feature dimension (usually set to , is the transposed matrix of text features, To extract visual features, and are the height and width of the feature map, respectively. is the number of channels, is the transposed matrix of text features, and softmax is the normalization function to ensure that the similarity value range is between [0,1]. Next, construct a cross-modal attention map : , in, and is a learnable parameter matrix for feature mapping, is the Sigmoid activation function. Attention map The value range of is [0,1], which indicates the degree of response of each spatial position in the visual feature to the text feature.

[0051] 4. Gated feature fusion: based on the calculated attention map , fuses visual and textual features through a gating mechanism: , in, is the guidance feature after fusion, and is a learnable projection matrix used for alignment and fusion of feature channels, Text features After spatial expansion, the feature map Represents element-wise multiplication (Hadamard product).

[0052] Preferably, and Both Convolutional layer is used for alignment and fusion of feature channels. When it is close to 1, the model relies more on visual features; when When it is close to 0, the model relies more on text features. Experiments show that in areas with complex textures, The values ​​are usually high, while in areas containing clear semantic information, The value is lower.

[0053] Step 3: Perform feature alignment between the guidance feature and the low-resolution video through a multimodal fusion module.

[0054] Reference Figure 3 The multimodal fusion module dynamically integrates the guidance features and low-resolution video features through a gating mechanism. The specific steps are as follows: input the guidance features and low-resolution video features into two parallel convolutional layers and ReLU layers; add the outputs of the two parallel branches and input them into another convolutional layer and ReLU layer; multiply the output result with the guidance gate in the guidance module to control the degree of fusion and generate alignment features.

[0055] The mathematical expression of the gating mechanism is: , in, and denote guidance features and low-resolution video features respectively, is the attention weight matrix, Represents element-by-element multiplication. This mechanism allows the system to dynamically adjust the importance of different modal features according to the content. Preferably, when the video content is complex, the system tends to give higher weight to visual features; when the video content is simple but contains rich semantic information, the system increases the weight of text features. The calculation is done by the following steps: , in, represents the feature connection operation, represents a 3×3 convolutional layer, is the Sigmoid activation function. In this way, the guidance gate can adaptively learn the importance weights between the guidance features and the low-resolution video features.

[0056] Step 4: Use a lightweight residual module to extract high-quality features.

[0057] Reference Figure 4 , the residual path of the lightweight residual module uses a channel separation convolution layer to reduce the computational complexity. The specific steps are as follows: the video frame is transformed from a size of Convert to 4 channels ; Convert the tensor to a shape of A tensor of Indicates the number of channels; the number of channels is reduced to 2 through the convolution layer , and make two copies; add three tensors of the same shape and pass them through the ReLU activation function; change the shape of the tensor group from Transformed to , which is rearranged through the convolutional layer as ; Expand the tensor channel by two times to get the number of channels to 2 The feature representation of .

[0058] Preferably, Set to 64 and Choi to 4. This configuration can achieve the best computational efficiency on most hardware platforms. The channel separation convolution design reduces the computational complexity of the module from the traditional convolution to Reduce to , achieving a significant improvement in computational efficiency. The detailed calculation process of channel separation convolution can be expressed as: , in, is the input feature, which is divided into Group( is the optimal value), For the Group features, For the 3×3 convolution operation of group features, This group convolution strategy greatly reduces the number of parameters and calculations while maintaining the feature expression capability.

[0059] Step 5: Fusion of multi-frame features through inter-frame information flow propagation, and use of adaptive Kalman filtering for motion compensation.

[0060] Reference Figure 5 and Fig. 9 The specific steps of motion compensation using adaptive Kalman filtering include: using a 3×3 convolutional neural network to calculate feature associations based on deep features extracted by a feature learning network; performing motion estimation on low-resolution video sequences through adaptive model parameters to obtain motion estimation values ​​at time t+1; sending the motion estimation values ​​to a fusion module to obtain Kalman motion estimation values; and correcting the low-resolution blocks after super-resolution through a motion compensation module to improve inter-frame consistency.

[0061] The present invention innovatively develops a deep adaptive Kalman filter algorithm, which combines traditional Kalman filtering with deep learning to achieve accurate motion prediction. The detailed algorithm is as follows: 1. System state definition: In the video super-resolution task, define the state vector Contains pixel position and motion speed information: , in, express The pixel position at the moment, Indicated in and 2. State prediction stage: based on the state of the previous moment Predict the current state : , in, is the state transfer matrix, is the control input matrix, is the control vector, is process noise, with covariance In the present invention, It is obtained by learning instead of the fixed matrix in traditional Kalman filtering: , in, It is a small network composed of three 3×3 convolutional layers. The input is the state transfer matrix and state vector at the previous moment, and the output is the state transfer matrix at the current moment. Represents the state transition matrix at the previous moment t-1. This adaptive design enables the model to dynamically adjust the state prediction strategy according to the content.

[0062] 3. Forecast covariance update: predict the covariance matrix of the state Updated via: , in, is the state covariance matrix of the previous moment, is the process noise covariance matrix. In the present invention, It is not a fixed value, but is learned through the network: , in, For the noise estimation network, the softplus activation function ensures that the covariance matrix is ​​positive definite.

[0063] 4. State update phase: Observations based on deep features , calculate the Kalman gain And update the status: , , , in, is the observation matrix, is the observation noise covariance matrix, is the identity matrix, represents the updated state covariance matrix, reflecting the uncertainty of state estimation. In the present invention, the observation value Extracted from deep features through feature association module: , in, represents the feature association module, which is implemented by a 3×3 convolutional neural network. for The characteristics of the moment, is the parameter of the feature association module. The observation noise covariance matrix Also obtained in an adaptive way: , in, Estimation network for observation noise.

[0064] 5. Motion compensation: based on the updated state , calculate the motion offset, and implement pixel-level motion compensation through bilinear interpolation: , in, From the state vector Extracted from and represents the pixel coordinates in the target image, Represents the input image.

[0065] Experiments show that compared with traditional Kalman filtering, the adaptive Kalman filtering of the present invention shows significant advantages in processing complex motion patterns, and the motion prediction accuracy is improved by about 35%, especially when processing scenes such as non-rigid object motion and camera shake.

[0066] The present invention adopts an innovative composite strategy of super-resolution first and then compensation fine-tuning. Specifically, the system first performs preliminary super-resolution processing on the input low-resolution video frame sequence, and generates preliminary super-resolution results through high-quality features extracted by the lightweight residual module. Although these results have been significantly enhanced in the spatial dimension, there may be inconsistencies in the temporal dimension.

[0067] Subsequently, the system calculates the feature correlation between adjacent frames based on the deep feature extraction network to obtain the motion estimation value. It is worth noting that this step uses the features of the original low-resolution video frame rather than the super-resolution features. This design considers the balance between computational efficiency and accuracy. Next, the motion estimation value is sent to the Kalman filter fusion module and integrated with the historical motion information to generate a more accurate Kalman motion estimation value.

[0068] In the final critical link, the motion compensation module uses the Kalman motion estimation value to accurately fine-tune and time-align the initial super-resolution results to correct the incoherence in the time dimension. The low-resolution blocks after super-resolution here actually refer to the initial super-resolution image blocks. Motion compensation is used to keep adjacent frames consistent in time and eliminate jitter and flicker.

[0069] This strategy of super-resolution first and compensation later has many advantages over the traditional method of compensation first and super-resolution later: it allows each frame to first obtain the best spatial super-resolution reconstruction quality, and then specifically solve the temporal consistency problem; it effectively avoids the accuracy limitations of motion compensation in the low-resolution domain; it better preserves high-frequency detail information; and it achieves the best balance between spatial quality and temporal smoothness.

[0070] Step 6: Reconstruct a high-definition image by combining the high-quality features with the information after feature fusion. This step combines the high-quality features extracted in step 4 with the information after feature fusion in step 5 by constructing a generator, and reconstructs the high-definition image through upsampling and linear interpolation networks. In this embodiment, upsampling uses sub-pixel convolution technology. Compared with traditional bicubic interpolation, sub-pixel convolution can learn more complex upsampling patterns and produce higher quality reconstruction results.

[0071] Preferably, the upsampling ratio may be 2x, 3x or 4x, which is selected according to actual application requirements.

[0072] The calculation process of sub-pixel convolution is as follows: , in, is the fused feature, Conv is the convolution operation, It is the pixel shuffle operation. is the generated super-resolution image. The Pixel Shuffle operation is defined as: , in, is the upsampling ratio, represents the rounding down operation, mod represents the modulus operation, Represents the channel index in the output feature map.

[0073] In addition, the present invention also includes the method of training the discriminator adversarially. The discriminator uses an adversarial generative network to identify the authenticity of the generated super-resolution video and the true super-resolution video. The loss function of the adversarial generative network is: , in, To combat the loss function, is the multi-scale loss function, is the cycle consistency loss function, is the guide loss function, is the total variation regularization loss function, and is the weight coefficient of the corresponding loss function.

[0074] Preferably, These weight values ​​were determined through extensive experiments and can achieve a good balance between various loss functions.

[0075] Adversarial Loss Function The calculation formula is: , in, represents the discriminator, represents a generator, represents a real high-resolution image, Indicates low-resolution input represents the generated super-resolution image, Represents the real high-resolution image The expectation of the distribution, Represents a low-resolution input image Expectation of the distribution.

[0076] Multi-scale loss function The reconstruction quality is comprehensively evaluated by calculating the difference between the generated image and the real image at different scales: , in, Indicates downsampling the image times the operation, is the number of scales considered (usually set to 3), For the This multi-scale design enables the model to focus on both global structure and local details. Represents the generated super-resolution image.

[0077] The calculation formula of the perceptual feature loss function is: , in, Indicates the extraction of Layer features, is the number of feature layers extracted in the loss function, To generate the super-resolution image, For real ultra-high resolution images, represents the weight coefficient of the s-th layer feature. Preferably, Set to 5, from shallow to deep features are taken into account. The discriminator is initialized with a pre-trained model extracted from the VGG network, and the perceptual difference loss is calculated by extracting multi-layer features of the input image. The feature learning network uses the features output from the last convolutional layer of the ImageNet pre-trained model. This pre-training strategy enables the model to converge faster and obtain better generalization ability.

[0078] Reference Figure 6 The lightweight residual module of the present invention includes a plurality of densely connected residual blocks, each residual block includes layer normalization, a convolution layer and a Leaky-ReLU activation layer; a deformable convolution is provided in the residual block for extracting local motion information of the video frame; dense connections are added between adjacent residual blocks to connect the input of each residual block with the output of all previous residual blocks.

[0079] The mathematical definition of the recursive residual network is: , in, represents the input features, represents a recursive convolutional layer, The residual connection operation of the number of layers of the identity module is defined as: , in, represents the low-resolution video frame features, represents the fused features, represents the residual connection operation, represents the characteristics of the previous moment t-1, Represents a recursive convolutional layer.

[0080] Preferably, three layers of recursive residual networks are stacked for progressive feature extraction, with the number of convolution kernels in the first recursive convolution layer being 160, the second layer being 256, and the third layer being 256. This configuration achieves a good balance between model capacity and computational efficiency. The Leaky-ReLU activation function is defined as , in, is a leakage parameter, preferably set to 0.2. Compared with standard ReLU, Leaky-ReLU avoids the dead ReLU problem and enhances the expressiveness of the model. Represents the input value, Indicates the condition if a value is entered.

[0081] The deformable convolution operation is defined as: , in, is the input feature map, is the current location, For the predefined offsets of sampling points, is the learned offset, is the corresponding weight, is the number of sampling points. In this embodiment, Set to 9, corresponding to a 3×3 convolution kernel.

[0082] Dense connections are implemented as follows: , in, For the The output of the layer, Before The connections of the layer outputs, For the The nonlinear transformation function of the layer. This dense connection mechanism promotes feature reuse, improves gradient flow, and accelerates model training.

[0083] Reference Figure 7 The present invention also provides a multi-modal fusion real-time video super-resolution reconstruction system, comprising: Feature extraction module 1, used to extract visual features and text features of low-resolution videos using the CLIP model, and perform bimodal feature fusion to generate guidance features; A feature alignment module 2, used for aligning the guiding feature with the low-resolution video through a multimodal fusion module; Feature optimization module 3, used to extract high-quality features using a lightweight residual module; The motion compensation module 4 is used to fuse the features of multiple frames through inter-frame information flow propagation and perform motion compensation using an adaptive Kalman filter; An image reconstruction module 5 is used to reconstruct a high-definition image by combining the high-quality features with information after feature fusion; The network training module 6 is used to train and optimize the system through a generative adversarial network.

[0084] Preferably, the feature extraction module 1 comprises a visual feature extraction unit 11 and a text feature extraction unit 12. The visual feature extraction unit 11 uses a ViT encoder and a ResNet network to process low-resolution video frames, and the text feature extraction unit 12 uses a Text Encoder of a CLIP model to process text description information.

[0085] The feature alignment module 2 includes a parallel convolution unit 21 and a fusion control unit 22. The parallel convolution unit 21 performs convolution processing on the guidance feature and the low-resolution video feature, and the fusion control unit 22 controls the degree of fusion through the guidance gate.

[0086] The feature optimization module 3 includes a channel transformation unit 31 and a residual processing unit 32. The channel transformation unit 31 is responsible for the conversion of the tensor shape and the adjustment of the number of channels, and the residual processing unit 32 realizes the residual connection and enhancement of the features.

[0087] The motion compensation module 4 includes a feature association unit 41 and a Kalman filter unit 42. The feature association unit 41 extracts inter-frame association information, and the Kalman filter unit 42 performs motion estimation and prediction.

[0088] The image reconstruction module 5 includes a feature fusion unit 51 and an upsampling unit 52. The feature fusion unit 51 combines high-quality features and fused information, and the upsampling unit 52 achieves high-resolution reconstruction through sub-pixel convolution.

[0089] The network training module 6 includes a generator training unit 61 and a discriminator training unit 62. The generator training unit 61 optimizes the reconstruction network parameters, and the discriminator training unit 62 provides adversarial training signals.

[0090] The system of the present invention realizes modular design, and each functional module cooperates closely to form a complete processing flow. The system has high scalability and adaptability, and the parameters and configuration of each module can be adjusted according to actual application requirements.

[0091] In actual application, the system can dynamically adjust the processing strategy according to different video contents. For example, for scenes with a lot of details (such as landscape videos), the system will rely more on visual features; for scenes with clear semantic information (such as character dialogues), the system will increase the weight of text features. Experiments show that this dynamic adjustment strategy enables the system to perform well on different types of video content.

[0092] In addition, the system supports deployment on a variety of hardware platforms, including desktop GPUs, mobile GPUs, and dedicated chips. On NVIDIA RTX2080Ti, the system can process 1080p video at 30fps; on NVIDIA RTX3090, the system can process 4K video at 25fps. Through optimization technologies such as model quantization and pruning, the system can also achieve real-time processing of 720p video on mid-range mobile devices.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-modal fusion real-time video super-resolution reconstruction method, characterized in that: The following steps are involved: Acquire a low-resolution video sequence; The CLIP model is used to extract visual features and text features, and bimodal features are fused to generate guidance features; Performing feature alignment on the guidance feature and the low-resolution video through a multimodal fusion module; Use lightweight residual module to extract high-quality features; The multi-frame features are fused through inter-frame information flow propagation, and the adaptive Kalman filter is used for motion compensation; The high-quality features are combined with the information obtained by feature fusion to reconstruct a high-definition image.

2. The multi-modal fusion real-time video super-resolution reconstruction method according to claim 1, characterized in that: The steps of extracting visual features and text features by the CLIP model specifically include: Input the low-resolution video sequence into the ViT encoder and the ResNet network to extract visual features; Input the context description in the low-resolution video frame into the Text Encoder to extract text features; Calculate the similarity scores between visual features and text features through a cross-modal similarity encoder; The visual features and the text features are combined through a gated fusion module to generate guidance features.

3. The multi-modal fusion real-time video super-resolution reconstruction method according to claim 1, characterized in that: The multimodal fusion module dynamically integrates the guidance features and the low-resolution video features through a gating mechanism, and the specific steps are as follows: Inputting the guidance features and the low-resolution video features into two parallel convolutional layers and ReLU layers; The outputs of the two parallel branches are added together and then input into another convolutional layer and ReLU layer; The output result is multiplied by the guidance gate in the guidance module to control the degree of fusion and generate alignment features.

4. The multi-modal fusion real-time video super-resolution reconstruction method according to claim 1, characterized in that: The residual path of the lightweight residual module adopts a channel separation convolution layer to reduce the computational complexity. The specific steps are as follows: Convert video frames from Convert to 4 channels Tensor of ; Convert the tensor to a shape of A tensor of Indicates the number of channels; the number of channels is reduced to 2 through the convolution layer , and make two copies; Add three tensors of the same shape and pass them through the ReLU activation function; Change the tensor group shape from Transformed to , which is rearranged through the convolutional layer as ; Expand the tensor channel by two times to get the number of channels to 2 The feature representation of .

5. The multi-modal fusion real-time video super-resolution reconstruction method according to claim 1, characterized in that: The specific steps of the adaptive Kalman filter for motion compensation include: Based on the deep features extracted by the feature learning network, a 3×3 convolutional neural network is used to calculate feature associations; The motion estimation of the low-resolution video sequence is performed through the adaptive model parameters to obtain the motion estimation value at time t+1; Sending the motion estimation value to a fusion module to obtain a Kalman motion estimation value; The motion compensation module is used to correct the low-resolution blocks after super-resolution and improve the consistency between frames.

6. The multi-modal fusion real-time video super-resolution reconstruction method according to claim 5, characterized in that: The adaptive Kalman filter includes a multi-branch multi-layer perceptron and a fusion module, and its Kalman motion estimation value is obtained by combining feature maps of different branches; The feature association module and the fusion module use layer normalization to process the output of each layer of the network.

7. The multi-modal fusion real-time video super-resolution reconstruction method according to claim 1, characterized in that: The lightweight residual module includes multiple densely connected residual blocks, each residual block includes layer normalization, convolution layer and Leaky-ReLU activation layer; deformable convolution is set in the residual block to extract local motion information of video frames; dense connections are added between adjacent residual blocks to connect the input of each residual block with the output of all previous residual blocks.

8. The multi-modal fusion real-time video super-resolution reconstruction method according to claim 1, characterized in that: The method is trained through discriminator adversarial training, wherein the discriminator loss function consists of an adversarial loss function, a multi-scale loss function, a cycle consistency loss function, a guided loss function and a total variation regularization loss function; the multi-scale loss function is used to evaluate the reconstruction quality at different scales; and the cycle consistency loss function is used to ensure feature consistency.

9. The multi-modal fusion real-time video super-resolution reconstruction method according to claim 8, characterized in that: The discriminator is initialized using a pre-trained model extracted from a VGG network; the perceptual difference loss is calculated by extracting multi-layer features of an input image; and the feature learning network uses features output from the last convolutional layer of an ImageNet pre-trained model.

10. A multi-modal fusion real-time video super-resolution reconstruction system that implements the method according to any one of claims 1 to 9, characterized in that: include: The feature extraction module is used to extract the visual features and text features of the low-resolution video using the CLIP model, and perform bimodal feature fusion to generate guidance features; A feature alignment module, used for aligning the guiding feature with the low-resolution video through a multimodal fusion module; Feature optimization module, used to extract high-quality features using a lightweight residual module; Motion compensation module, used to fuse multi-frame features through inter-frame information flow propagation and use adaptive Kalman filtering for motion compensation; An image reconstruction module, used to reconstruct a high-definition image by combining the high-quality features with information after feature fusion; The network training module is used to train and optimize the system through a generative adversarial network.

Images