Video reconstruction method and related apparatus, electronic device, and storage medium

Abstract

The application discloses a video reconstruction method and related device, electronic equipment and storage medium, including: based on video frame sequence image set in video data, first feature extraction is carried out, and target feature map is obtained;Based on the correlation between each sub-feature map obtained by dividing the target feature map in the spatial dimension, a plurality of feature sets are constructed;Wherein, the feature set contains at least one sub-feature map;Based on each feature set, denoising prediction is carried out respectively, and the denoising features corresponding to each feature set are obtained;Based on the first fusion feature of each denoising feature, decoding is carried out, and the target video sequence image set after super-resolution reconstruction of the video frame sequence image set is obtained;Wherein, the resolution of the target video sequence image set is higher than that of the video frame sequence image set.The above scheme can improve the super-resolution reconstruction efficiency of video data.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video reconstruction method and related apparatus, electronic equipment and storage medium. Background Technology

[0002] With the development of video processing technology, video super-resolution technology is receiving increasing attention for its ability to recover high-resolution video from low-resolution video, thereby improving visual perception.

[0003] In existing technologies, video super-resolution reconstruction methods mainly rely on large-scale convolutional neural networks (CNNs) or spatiotemporal transformation models. These methods generally suffer from problems such as large redundancy of model parameters and high computational complexity, resulting in long reconstruction times and low efficiency, making it difficult to meet the needs of applications with high real-time requirements. In view of this, how to improve the efficiency of super-resolution reconstruction of video data has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a video reconstruction method and related apparatus, electronic equipment, and storage medium that can improve the efficiency of super-resolution reconstruction of video data.

[0005] To address the aforementioned technical problems, a first aspect of this application provides a video reconstruction method, comprising: extracting a first feature based on a video frame sequence image set in video data to obtain a target feature map; wherein the video frame sequence image set is a set of continuous image frames to be reconstructed in the video data; constructing several feature sets based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map; wherein each feature set contains at least one sub-feature map; performing denoising prediction based on each feature set to obtain denoised features corresponding to each feature set; and decoding based on a first fusion feature of each denoised feature to obtain a target video sequence image set after super-resolution reconstruction of the video frame sequence image set; wherein the resolution of the target video sequence image set is higher than that of the video frame sequence image set.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a video reconstruction apparatus, comprising: a feature extraction module, a set construction module, a denoising prediction module, and a feature decoding module. The feature extraction module is used to perform first feature extraction based on a video frame sequence image set in video data to obtain a target feature map; wherein, the video frame sequence image set is a set of continuous image frames to be reconstructed in the video data; the set construction module is used to construct several feature sets based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map; wherein, each feature set contains at least one sub-feature map; the denoising prediction module is used to perform denoising prediction based on each feature set to obtain denoised features corresponding to each feature set; the feature decoding module is used to decode based on the first fused features of each denoised feature to obtain a target video sequence image set after super-resolution reconstruction of the video frame sequence image set; wherein, the resolution of the target video sequence image set is higher than that of the video frame sequence image set.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the video reconstruction method of the first aspect described above.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the video reconstruction method of the first aspect described above.

[0009] The above scheme extracts the first feature from the video frame sequence image set to be reconstructed in the video data to obtain the target feature map. Based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, several feature sets are constructed, each containing at least one sub-feature map. Denoising prediction is performed on each feature set to obtain the corresponding denoised features. Decoding is then performed based on the first fusion feature of each denoised feature to obtain the target video sequence image set with higher resolution after super-resolution reconstruction. Therefore, constructing several feature sets based on the correlation between the sub-feature maps helps to ensure that sub-feature maps belonging to the same feature set have strong mutual influence, while sub-feature maps belonging to different feature sets have weak mutual influence. Therefore, using the feature set as the basic unit for denoising prediction and performing denoising prediction on each feature set separately can skip redundant interactions between sub-feature maps not belonging to the same feature set, reduce redundant calculations introduced when directly processing the target feature map uniformly, and thus improve processing efficiency. Therefore, it can improve the efficiency of super-resolution reconstruction of video data. Attached Figure Description

[0010] Figure 1This is a flowchart illustrating an embodiment of the video reconstruction method of this application; Figure 2 This is a schematic diagram of an embodiment of video data processing in the video reconstruction method of this application; Figure 3 This is a schematic diagram of the framework of an embodiment of knowledge distillation in the video reconstruction method of this application; Figure 4 This is a schematic diagram of the framework of an embodiment of the video reconstruction device of this application; Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0011] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0012] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0013] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0014] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0015] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the video reconstruction method of this application. It should be noted that the process operations in this embodiment can be performed by an electronic device with computing capabilities or related equipment containing electronic devices; no specific structure or type is limited herein. Specifically, this embodiment may include the following steps: Step S11: Perform first feature extraction based on the video frame sequence image set in the video data to obtain the target feature map.

[0016] In this embodiment of the disclosure, the data type of the video data is streaming video or non-streaming video. Specifically, streaming video may include real-time captured video streams, such as videos captured in real time by a camera device, while non-streaming video may include pre-stored video files.

[0017] As mentioned earlier, when performing video super-resolution reconstruction on video data, the right to use the video data can be obtained in advance by issuing an authorization request. The authorization request can specify the scope of use, time limit and other authorization conditions of the video data to meet compliance requirements.

[0018] It should be noted that the original resolution of video data may vary in different implementation scenarios. This application does not limit the range of original resolution of video data applicable to the following embodiments. Specifically, video data of different resolutions within the applicable resolution range can be processed using the video reconstruction method of the following embodiments.

[0019] In one implementation scenario, the video frame sequence image set is a set of continuous image frames to be reconstructed from the video data. Specifically, the video data consists of continuous temporal image frames, and the video frame sequence image set to be reconstructed is a set of consecutive image frames selected sequentially from the video data based on temporal order. For example, the preset number of frames is 4 frames, that is, the images of the first to fifth frames of the video data are used as a set of video frame sequence images to be reconstructed, the sixth to ninth frames are used as a set of video frame sequence images to be reconstructed, and so on.

[0020] It should be noted that the preset quantity can be set according to actual needs, and is not limited in this application.

[0021] In one implementation scenario, the first feature extraction is used to compress the set of video frame sequence images to be reconstructed into a latent space, specifically, the latent space represents a low-dimensional, abstract vector space.

[0022] In a specific implementation scenario, the first feature extraction includes spatial compression and temporal compression. Specifically, the video frame sequence image set to be reconstructed can be transformed from the original high-dimensional data into a low-dimensional target feature map by means of 4x temporal compression and 8x spatial compression.

[0023] In a specific implementation scenario, the first feature extraction is based on an encoder. Specifically, the encoder is a variational autoencoder (VAE). The variational autoencoder maps the input high-dimensional video frame sequence image set to a low-dimensional latent space through its encoding part, thereby obtaining the target feature map.

[0024] Step S12: Based on the correlation between the sub-feature maps obtained by dividing the target feature map in spatial dimensions, several feature sets are constructed.

[0025] In this embodiment of the disclosure, the feature set includes at least one sub-feature map. Specifically, sub-feature maps belonging to the same feature set have a strong mutual influence relationship, while sub-feature maps belonging to different feature sets have a weak mutual influence relationship.

[0026] In one implementation scenario, the target feature map is divided into sub-feature maps in the spatial dimension. Specifically, the division can be carried out by a preset division strategy, such as uniformly dividing the target feature map into multiple sub-feature maps in the spatial dimension, or performing non-uniform division according to the feature distribution in different regions of the target feature map in the spatial dimension.

[0027] In a specific implementation scenario, the size of the target feature map is represented by both the spatial dimension and the channel dimension. For example, the size of the target feature map is (C×H×W), where C represents the channel dimension, H×W represents the spatial dimension, H represents the height direction, and W represents the width direction. In the process of dividing the target feature map into sub-feature maps, the channel dimension C remains unchanged, and only the spatial dimension H×W is divided.

[0028] In a specific implementation scenario, the target feature map is divided into a first number of partitions in the height direction and a second number of partitions in the width direction. Based on these first and second partition numbers, the target feature map is divided in the height and width directions, respectively, to obtain several sub-feature maps. The number of channels in the sub-feature maps is the same as the number of channels in the target feature map. Specifically, as mentioned earlier, if the target feature map has dimensions (C×H×W), the first partition number is h, and the second partition number is w, then the target feature map can be divided into h×w sub-feature maps, each with dimensions C×(H / h)×(W / w).

[0029] In one implementation scenario, before constructing several feature sets based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, feature dimensionality reduction is performed on each sub-feature map to obtain dimensionality-reduced sub-feature maps. The correlation between the dimensionality-reduced sub-feature maps is then calculated, and several dimensionality-reduced sub-feature maps whose correlation meets preset association conditions are selected and grouped into the same feature set. This scheme reduces the dimensionality of the sub-feature maps through feature dimensionality reduction, thereby reducing computational load and improving computational efficiency. Furthermore, grouping dimensionality-reduced sub-feature maps that meet preset association conditions into the same feature set helps to ensure strong mutual influence among the sub-feature maps within the same set, laying the foundation for subsequent denoising and prediction of each feature set.

[0030] In a specific implementation scenario, feature dimensionality reduction can be achieved based on algorithms such as average pooling and principal component analysis. The correlation between the sub-feature maps after dimensionality reduction is calculated, and the magnitude of the correlation represents the degree of association between two sub-feature maps. For example, when object A, object B, and scene background appear in a video frame sequence image set, there is a strong correlation between several sub-feature maps of object A; if object A and object B have some interaction, there is also a strong correlation between the sub-feature maps related to the interaction between the two; while the correlation between sub-feature maps of object A that are not related to the interaction and the sub-feature maps of object B that are not related to the interaction is weak; similarly, the correlation between the sub-feature maps of the scene background and the sub-feature maps of object A is also weak.

[0031] In a specific implementation scenario, the correlation can be calculated by calculating cosine similarity. For example, the range of cosine similarity is [-1, 1]. The closer the value is to 1, the stronger the correlation between the two sub-feature maps. The closer the value is to -1, the weaker the correlation between the two sub-feature maps. The closer the value is to 0, the less correlated the two sub-feature maps are.

[0032] In another specific implementation scenario, after obtaining the sub-feature maps after feature dimensionality reduction, the correlation between the sub-feature maps after dimensionality reduction can be obtained by calculating block-level attention maps. Specifically, the sub-feature maps are used as the smallest unit for attention calculation to obtain the correlation between the sub-feature maps.

[0033] In a specific implementation scenario, the preset association condition is that the correlation between sub-feature maps is not less than a preset threshold. When the cosine similarity is used as the correlation between sub-feature maps, the preset association condition can be that the cosine similarity is greater than the preset threshold. When the cosine similarity between the sub-feature maps after dimensionality reduction of two features is greater than the preset threshold, the two sub-feature maps are assigned to the same feature set.

[0034] In a specific implementation scenario, after constructing several feature sets, each feature set can be further optimized. For example, abnormal sub-feature maps with low correlation to other sub-feature maps can be removed to improve the correlation and consistency among sub-feature maps within the feature set. In a specific implementation scenario, for each feature set, the average correlation between each sub-feature map within the feature set and other sub-feature maps is calculated. Sub-feature maps with an average correlation lower than a preset optimization threshold are removed from the feature set, thus obtaining the optimized feature set.

[0035] In another specific implementation scenario, feature sets can be constructed using clustering algorithms, dividing sub-feature maps into different sets based on their correlation. For example, the K-means clustering algorithm can be used to assign sub-feature maps to the feature set represented by the nearest cluster center by calculating the distance or similarity between them. Alternatively, hierarchical clustering algorithms can be used to progressively merge sub-feature maps based on their correlation, forming feature sets at different levels. These clustering algorithms can be selected and adjusted according to actual needs and data characteristics.

[0036] In one implementation scenario, feature sets are divided based on the correlation between sub-feature maps. Therefore, sub-feature maps belonging to the same feature set have a strong mutual influence relationship, while sub-feature maps belonging to different feature sets have a weaker mutual influence relationship. In this case, since the correlation of detailed features between sub-feature maps of different feature sets is low, even without performing feature interaction, it will not lead to the loss or distortion of key reconstruction information, nor will it affect the image clarity, detail integrity, and inter-frame smoothness of the final super-resolution video. Therefore, whether or not feature interaction is performed between them has no substantial impact on super-resolution reconstruction. Thus, using feature sets as the basic unit for denoising prediction and performing denoising prediction on each feature set separately can improve the processing efficiency of video reconstruction while ensuring the quality of video super-resolution reconstruction as much as possible.

[0037] In one implementation scenario, considering the low resolution of the video data to be super-reconstructed, and the potential for image structure errors in low-resolution video data (e.g., some edges or texture regions may appear discontinuous or blurry due to information loss), such structural errors can affect the accuracy of subsequent super-reconstruction, for example, reconstructing a blurry kitten as a clear puppy. Therefore, after performing the first feature extraction based on the set of video frame sequences to be reconstructed from the video data to obtain the target feature map, and before constructing several feature sets based on the correlation between the sub-feature maps obtained from the target feature map, a preset intensity noise is injected into the target feature map to obtain an updated target feature map. The preset intensity noise is used to weaken the feature structure of the target feature map at a preset intensity. It should be noted that the preset intensity noise injected into the target feature map is different from the pure noise added during the diffusion process. Several feature sets are constructed based on the correlation between the sub-feature maps obtained from the updated target feature map. This scheme, by injecting preset intensity noise, can make the feature distribution in the target feature map more uniform, reduce the interference caused by some features being too prominent in the correlation calculation, and thus improve the quality of the constructed feature sets.

[0038] In a specific implementation scenario, the noise of the preset intensity injected into the target feature map is in the range of 0.3 to 0.6. While weakening the feature structure of the target feature map, it can avoid excessive destruction of the feature information of the target feature map, thereby ensuring the accuracy of the subsequent construction of the feature set based on the sub-feature map.

[0039] Step S13: Perform denoising prediction based on each feature set to obtain the denoised features corresponding to each feature set.

[0040] In one implementation scenario, the feature set is used as the execution unit for denoising prediction. Noisy feature sets at different noise scales are taken as input, and the noise scale embedding information and the correlation information between the captured sub-feature maps are combined. The denoising network learns and predicts the noise distribution or specific noise value superimposed in the feature set. Then, according to the mathematical formula of the diffusion reverse process, the predicted noise is stripped from the noisy feature set to obtain the denoised features corresponding to each feature set.

[0041] In one implementation scenario, forward diffusion is performed on the sub-feature maps in the feature set to obtain noisy features. Attention is then calculated based on the noisy features to obtain enhanced features. Backward diffusion is then performed based on the enhanced features to predict the denoised features corresponding to the feature set, thereby improving the accuracy and quality of the denoised features.

[0042] In a specific implementation scenario, attention calculation based on noisy features can be achieved through a spatiotemporal attention mechanism. The spatiotemporal attention mechanism can simultaneously capture temporal and spatial information in a video frame sequence image set. By assigning different attention weights to the noisy features in the temporal and spatial dimensions, it can highlight key features and suppress irrelevant features, thereby enhancing the ability to generate details and thus obtaining enhanced features.

[0043] In one implementation scenario, denoising prediction includes at least a diffusion mechanism that integrates the attention calculation process. Denoising prediction is performed based on each feature set. Before obtaining the denoising features corresponding to each feature set, historical attention key-value pairs generated in the historical denoising prediction are obtained based on the historical denoising predictions already completed by several historical video frame sequence image sets preceding the current video frame sequence image set to be reconstructed. For example, historical attention key-value pairs generated during the denoising prediction process of several historical video frame sequence image sets preceding the current video frame sequence image set to be reconstructed can be obtained. Specifically, historical attention key-value pairs generated during the denoising prediction process of 8 historical video frame sequence image sets preceding the current video frame sequence image set to be reconstructed can be obtained. The specific number of historical frames is not limited in this application. The historical attention key-value pairs are used as reference data and attention calculation is performed with the feature set to obtain enhanced features. Reverse diffusion is performed based on the enhanced features to predict the denoising features corresponding to the feature set. That is, the current video frame sequence image set to be reconstructed can achieve denoising prediction of the video frame sequence image set to be reconstructed by relying only on the obtained historical attention key-value pairs and the attention key-value pairs in the current denoising prediction process. The above scheme applies historical attention key-value pairs to the denoising prediction of the current video frame sequence image set to be reconstructed, so as to provide richer contextual information, which helps to improve the accuracy and stability of the current denoising prediction, reduce the amount of data required for denoising prediction, and improve the real-time performance and efficiency of video reconstruction.

[0044] Step S14: Decode based on the first fusion feature of each denoising feature to obtain the target video sequence image set after super-resolution reconstruction of the video frame sequence image set.

[0045] In this embodiment of the disclosure, the resolution of the target video sequence image set is higher than that of the video frame sequence image set. For example, the video frame sequence image set is an LR (Low-Resolution) image set relative to the target video sequence image set. The frame image pixel size is small and lacks detailed information. This is usually caused by factors such as insufficient performance of the acquisition device, limited transmission bandwidth, multiple compression encoding, or degradation of old recordings. It manifests as problems such as blurry images, jagged edges, loss of texture, and inter-frame motion misalignment. The target video sequence image set is an HR (High-Resolution) image set relative to the video frame sequence image set. The frame image pixel size is larger, richer in detail, and clearer.

[0046] In one implementation scenario, constructing several feature sets based on the correlation between sub-feature maps helps to ensure strong mutual influence between sub-feature maps belonging to the same feature set, while exhibiting weaker mutual influence between sub-feature maps belonging to different feature sets. Therefore, performing denoising prediction on each feature set separately can skip redundant interactions between sub-feature maps not belonging to the same feature set. While ensuring the quality of video super-resolution reconstruction as much as possible, it reduces redundant computation introduced by directly processing the target feature map uniformly, thereby improving processing efficiency. Thus, it can improve the efficiency of super-resolution reconstruction of video data.

[0047] In a specific implementation scenario, a first fused feature is obtained based on each denoised feature. Specifically, the denoised features can be fused through methods such as concatenation, addition, or weighted summation. For example, concatenation can be used to concatenate the denoised features in a certain order along the dimensions to form a new feature vector containing all denoised feature information; or addition can be used to add the elements of each denoised feature at corresponding positions to obtain the first fused feature; alternatively, different weights can be assigned to each denoised feature according to their importance, and then a weighted summation can be performed to obtain the first fused feature.

[0048] In a specific implementation scenario, decoding the first fused feature can be achieved based on a decoder, such as a variational self-decoder, and the video content represented by the first fused feature can be gradually recovered based on 3D convolution.

[0049] It should be noted that the video super-resolution reconstruction process in this application is performed in a streaming manner. That is, when processing video data, the video reconstruction is performed sequentially on each video frame sequence image set obtained based on time sequence in the video data. For example, after extracting the target feature map of the current video frame sequence image set to be reconstructed, the target feature map of the next video frame sequence image set to be reconstructed is extracted.

[0050] In one implementation scenario, before decoding the video frame sequence image set after super-resolution reconstruction using the fused features based on various denoising features to obtain the target video sequence image set, a second feature extraction is performed based on the video frame sequence image set to obtain video features. These video features are then fused with the first fused feature to obtain a second fused feature. Decoding is then performed based on this second fused feature to obtain the target video sequence image set. This scheme leverages the structural prior of the video features in the video frame sequence image set during the decoding process, minimizing ambiguity issues and improving decoding efficiency.

[0051] In a specific implementation scenario, the video features and the fused features have the same feature dimensions. The first feature extraction includes spatial compression and temporal compression, and the second feature extraction includes pixel rearrangement and causal convolution. Specifically, the steps of the second feature extraction include first upsampling the video frame sequence image set to a first sub-feature that matches the resolution of the first fused feature through pixel rearrangement, and then using two layers of 3D causal convolution to extract the spatiotemporal features of the first sub-feature to obtain the video features.

[0052] In a specific implementation scenario, the steps of fusing video features and the first fusion feature to obtain the second fusion feature include concatenating the video features and the first fusion feature in the channel dimension, then introducing a 1×1 convolution and attention fusion module, adaptively assigning weights to the structural information of the video features and the detailed information of the first fusion feature, and then weighted fusing to obtain the second fusion feature.

[0053] In one implementation scenario, the target video sequence image set is obtained by processing the video frame sequence image set by a video reconstruction model. The video reconstruction model is the target student model obtained by knowledge distillation of the target teacher model. The video reconstruction model includes a target encoder, a target diffusion attention block, and a target decoder. The target encoder is used to extract the first feature based on the video frame sequence image set to be reconstructed in the video data to obtain the target feature map. The target diffusion attention block is used to construct several feature sets based on the correlation between the sub-feature maps obtained by dividing the target feature map in the spatial dimension. Denoising prediction is performed based on each feature set to obtain the denoised features corresponding to each feature set. The target decoder is used to decode based on the first fusion feature of each denoised feature to obtain the target video sequence image set after super-resolution reconstruction of the video frame sequence image set. The above scheme transfers knowledge from the complex target teacher model to the more compact target student model through knowledge distillation. This allows the target student model to maintain high reconstruction quality while having faster inference speed and lower computational resource consumption, making it more suitable for real-time and resource-constrained requirements in practical application scenarios. Furthermore, the partitioning of feature sets in the student model reduces the amount of data computation, supporting the lightweight model structure of the target student model to achieve video reconstruction while ensuring the quality of video reconstruction as much as possible.

[0054] In a specific implementation scenario, the knowledge distillation steps include: acquiring a sample dataset, which contains at least a number of sample videos and sample images of a resolution threshold or higher to avoid the problem of insufficient sample video data of a resolution threshold or higher; treating sample images as relatively special single-frame sample videos to increase the data volume of the sample dataset and improve the quality of subsequent target teacher model training; performing degradation processing based on the sample dataset to obtain target training data, the degradation processing including at least one of optical flow-guided motion blur and model-guided noise injection; training the initial teacher model based on the target training data to obtain the target teacher model, the model structure of the initial teacher model including a sequentially connected teacher encoder, a multi-step diffusion attention block, and a teacher decoder; using the first mapping relationship between the input features and output features of the multi-step diffusion attention block in the target teacher model as the first training objective; using the second mapping relationship between the input features and output video of the teacher decoder in the target teacher model as the second training objective; distilling the model knowledge of the target teacher model into the target student model to obtain a video reconstruction model, the target diffusion attention block in the target student model being a single-step diffusion attention block.

[0055] Please see Figure 2 , Figure 2 This is a schematic diagram of an embodiment of video data processing in the video reconstruction method of this application. In one implementation scenario, the video reconstruction model further includes a frame grouping module and a noise injection module, such as... Figure 2 As shown, after acquiring video data, the video data is divided into a preset number of frames by the frame grouping module to obtain a set of video frame sequence images to be reconstructed. The set of video frame sequence images to be reconstructed is then matched with a temporal compression strategy and input to the target encoder. The target encoder performs temporal and spatial compression on the set of video frame sequence images to be reconstructed according to the temporal compression strategy to obtain a target feature map. The noise injection module injects noise of a preset intensity into the target feature map. Specifically, the noise intensity ranges from 0.3 to 0.6. The updated target feature map is output as the input to the target diffusion attention block. In addition, the set of video frame sequence images to be reconstructed obtained by the frame grouping module is also directly output as the input data of the target decoder.

[0056] In a specific implementation scenario, the dataset is degraded to simulate the features of real low-resolution videos. Specifically, optical flow-guided motion blur and model-guided noise injection can be used to treat the sample data and the degraded target training data as a training pair.

[0057] In a specific implementation scenario, 3D convolution and spatiotemporal attention modeling are used to process sample images to achieve joint training of image and video data.

[0058] In a specific implementation scenario, the initial teacher model's structure includes a sequentially connected teacher encoder, a multi-step diffusion attention block, and a teacher decoder. Specifically, the teacher encoder is used to compress the video to be reconstructed into the latent space, which can be achieved through 4x temporal and 8x spatial compression. The multi-step diffusion attention block adopts full-temporal and spatiotemporal dense attention to capture long-distance dependencies across frames, while introducing textual conditions, such as fixed prompts "Cinematic, High Contrast, hyper-detailed," to enhance the ability to generate details. The teacher decoder gradually recovers denoising features based on 3D convolution to obtain high-resolution video.

[0059] In a specific implementation scenario, the training objective of the initial teacher model can be to use Flow Matching (FM) loss to optimize the denoising process of the multi-step diffusion attention block, and to achieve the convergence of the initial teacher model based on the optimizer to obtain the target teacher model.

[0060] In a specific implementation scenario, the target decoder in the target student model has the same network structure as the teacher decoder in the target teacher model, or the network structure of the target decoder in the target student model has the same network structure as the lightweight teacher decoder. The network structure of the target diffusion attention block in the target student model is a lightweight multi-step diffusion attention block with a sparse causal mechanism. Specifically, after obtaining the target feature map output by the target decoder, the target diffusion attention block does not directly perform denoising prediction on the target feature map. Instead, it constructs several feature sets based on the correlation between the sub-feature maps obtained from the target feature map. Denoising prediction is then performed on each feature set to obtain the denoised features corresponding to each feature set. For example, the target feature map is divided into non-overlapping blocks of size (2, 8, 8). Average pooling is performed on each block to calculate the block-level coarse attention map. The top-k (k=1) most relevant block pairs are selected. During the denoising prediction process, full attention calculation is performed on the sub-feature maps selected from the feature sets to reduce the computational load. Furthermore, the target diffusion attention block uses a sliding window of a preset size to obtain historical attention key-value pairs from the super-resolution reconstruction process that has already been completed in the image set of the video frame sequence to be reconstructed, thereby improving the data processing efficiency of video reconstruction.

[0061] In a specific implementation scenario, before video reconstruction is performed on the video frame sequence image set to be reconstructed, the video reconstruction model masks several historical video frames in the video data that exceed the preset number of historical frames preceding the video frame sequence image set to be reconstructed. Specifically, this can be achieved by introducing a causal mask into the target diffusion attention block. Under the effect of the causal mask, when the target diffusion attention block processes the current video frame sequence image set to be reconstructed, it cannot obtain relevant information of several historical video frames that exceed the preset number of historical frames. It can only utilize the historical attention key-value pairs within the preset number of historical frames and the feature information of the current frame itself, thus avoiding the increase in computational complexity and the decrease in model performance caused by introducing too much irrelevant historical information.

[0062] In a specific implementation scenario, the target decoder of the target student model is a lightweight teacher decoder. The target decoder can use the video features of the image set of the video frame sequence to be reconstructed and the first fused features as decoding input. The decoding process utilizes the structural prior of the video features to avoid the ambiguity problem of the lightweight decoder.

[0063] In a specific implementation scenario, the distribution matching distillation (DMD) loss is adopted to enable the target student model to learn the denoising distribution of the target teacher model: the clean latent features output by the teacher model are constrained by KL divergence with the latent features predicted by the student model in one step, while retaining the FM loss to optimize the single-step denoising process.

[0064] In a specific implementation scenario, MSE loss (pixel-level reconstruction), LPIPS loss (perceptual quality), and distillation loss (aligning with the teacher model output) are combined as the training objectives of the target student model.

[0065] In a specific implementation scenario, the target diffusion attention block in the video reconstruction model is deployed on the first processor, the target decoder is deployed on the second processor, and there is a communication connection between the first processor and the second processor.

[0066] In a specific implementation scenario, in order to adapt to the difference in video resolution between the training and inference processes, the video reconstruction model can introduce a local attention window constraint to limit the attention range during inference to a 1152×1152 window, aligning it with the position encoding range during training and avoiding duplicate textures or blurring.

[0067] In a specific implementation scenario, the video reconstruction model uses the relevant features obtained from the processing of the first frame of video data as the basic reference for the entire subsequent video data processing process. However, during the streaming reconstruction of video data, the basic reference latent features may be contaminated. Therefore, after a certain number of frames, the potentially contaminated basic reference latent features are reset to their initial state.

[0068] Please see Figure 3 , Figure 3This is a schematic diagram of the framework of an embodiment of the knowledge distillation method in the video reconstruction method of this application. Specifically, knowledge distillation includes a first stage of teacher model training and a second stage of student model distillation. In the first stage of teacher model training, a sample dataset is acquired, which includes sample videos with a resolution of 120K and sample images with a resolution of 180K. Degradation processing is performed on the sample dataset to obtain target training data, and the sample images are modeled as target training data. The degradation processing includes at least one of optical flow-guided motion blur and model-guided noise injection. The training data pair constructed from the sample dataset and target training data is processed based on the initial teacher model with sequentially connected teacher encoder, multi-step diffused attention block and teacher decoder, and then processed through Flow. Matching loss optimization employs an optimizer learning rate of 5e-5 and a weight decay of 0.01 to achieve 20K iterations, yielding the target teacher model. In the second stage, student model distillation, the target student model's target diffusion attention block divides the target feature map output by the target encoder into several sub-feature maps. For example, the target feature map is divided into non-overlapping blocks of size (2, 8, 8) to obtain sub-feature maps. Average pooling is performed on each sub-feature map, and block-level coarse attention maps are calculated to select the top-k (k=1) most relevant block pairs to construct a feature set. A causal mask is introduced, combined with a KV-cache sliding window mechanism, for example, with a window size of 8 frames. The target decoder is a lightweight conditional decoder that fuses video features and a first fusion feature to obtain a second fusion feature. Decoding is then performed based on the second fusion feature to obtain a target video sequence image set. The MSE loss (pixel-level reconstruction), LPIPS loss (perceptual quality), and distillation loss (aligning with the teacher model output) are used as training objectives. The first mapping relationship between the input and output features of the multi-step diffusion attention block in the target teacher model is used as the training objective. 5K iterations are achieved through DMD loss and FM loss to learn the single-step denoising corresponding to the multi-step denoising of the multi-step diffusion attention block in the target teacher model, and single-step distillation optimization is achieved. Finally, the video reconstruction model after training convergence is obtained.

[0069] The above scheme extracts the first feature from the video frame sequence image set to be reconstructed in the video data to obtain the target feature map. Based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, several feature sets are constructed, each containing at least one sub-feature map. Denoising prediction is performed on each feature set to obtain the corresponding denoised features. Decoding is then performed based on the first fusion feature of each denoised feature to obtain the target video sequence image set with higher resolution after super-resolution reconstruction. Therefore, constructing several feature sets based on the correlation between the sub-feature maps helps to ensure that sub-feature maps belonging to the same feature set have strong mutual influence, while sub-feature maps belonging to different feature sets have weak mutual influence. Therefore, using the feature set as the basic unit for denoising prediction and performing denoising prediction on each feature set separately can skip redundant interactions between sub-feature maps not belonging to the same feature set, reduce redundant calculations introduced when directly processing the target feature map uniformly, and thus improve processing efficiency. Therefore, it can improve the efficiency of super-resolution reconstruction of video data.

[0070] Please see Figure 4 , Figure 4 This is a schematic diagram of the framework of an embodiment of the video reconstruction apparatus of this application. The video reconstruction apparatus 40 includes: a feature extraction module 41, a set construction module 42, a denoising prediction module 43, and a feature decoding module 44. The feature extraction module 41 is used to perform a first feature extraction based on a set of video frame sequence images in the video data to obtain a target feature map; wherein, the set of video frame sequence images is a set of continuous image frames to be reconstructed in the video data; the set construction module 42 is used to construct several feature sets based on the correlation between the sub-feature maps obtained by dividing the target feature map in the spatial dimension; wherein, the feature set contains at least one sub-feature map; the denoising prediction module 43 is used to perform denoising prediction based on each feature set to obtain denoised features corresponding to each feature set; the feature decoding module 44 is used to decode based on the first fusion feature of each denoised feature to obtain a target video sequence image set after super-resolution reconstruction of the video frame sequence image set; wherein, the resolution of the target video sequence image set is higher than that of the video frame sequence image set.

[0071] In the above scheme, the video reconstruction device 40 performs first feature extraction based on the video frame sequence image set to be reconstructed in the video data to obtain a target feature map. Based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, several feature sets are constructed, and each feature set contains at least one sub-feature map. Denoising prediction is performed based on each feature set to obtain the denoised features corresponding to each feature set. Decoding is performed based on the first fusion feature of each denoised feature to obtain a target video sequence image set with higher resolution after super-resolution reconstruction of the video frame sequence image set. Therefore, constructing several feature sets based on the correlation between the sub-feature maps helps to ensure that sub-feature maps belonging to the same feature set have a strong mutual influence relationship, while sub-feature maps belonging to different feature sets have a weak mutual influence relationship. Therefore, using the feature set as the basic unit for denoising prediction and performing denoising prediction on each feature set separately can skip redundant interactions between sub-feature maps not belonging to the same feature set, reduce redundant calculations introduced when directly processing the target feature map uniformly, and thus improve processing efficiency. Therefore, it can improve the efficiency of super-resolution reconstruction of video data.

[0072] In some disclosed embodiments, before constructing several feature sets based on the correlation between the various sub-feature maps obtained by dividing the target feature map in spatial dimensions, the video reconstruction device 40 further includes a feature dimensionality reduction module (not shown) for performing feature dimensionality reduction based on each sub-feature map to obtain sub-feature maps after feature dimensionality reduction; the set construction module 42 further includes a correlation calculation module (not shown) for calculating the correlation between the sub-feature maps after feature dimensionality reduction; the set construction module 42 further includes a correlation filtering module (not shown) for filtering several sub-feature maps after feature dimensionality reduction whose correlation satisfies preset association conditions and classifying them into the same feature set.

[0073] In some disclosed embodiments, the denoising prediction module 43 further includes a forward diffusion module (not shown) for forward diffusion of the sub-feature maps in the feature set to obtain noisy features; the denoising prediction module 43 further includes a feature enhancement module (not shown) for attention calculation based on the noisy features to obtain enhanced features; the denoising prediction module 43 further includes a reverse diffusion module (not shown) for reverse diffusion based on the enhanced features to predict the denoised features corresponding to the feature set.

[0074] In some disclosed embodiments, the denoising prediction includes at least a diffusion mechanism that integrates the attention calculation process. Before performing denoising prediction based on each feature set to obtain the denoised features corresponding to each feature set, the video reconstruction device 40 further includes a historical key-value pair acquisition module (not shown), which is used to acquire historical attention key-value pairs generated in the historical denoising prediction based on the historical denoising predictions already completed for several historical video frame sequence image sets before the current video frame sequence image set to be reconstructed. The denoising prediction module 43 further includes an attention calculation module (not shown), which is used to use the historical attention key-value pairs as reference data to perform attention calculation with the feature set to obtain enhanced features. The denoising prediction module 43 further includes a denoising prediction module (not shown), which is used to perform reverse diffusion based on the enhanced features to predict the denoised features corresponding to the feature set.

[0075] In some disclosed embodiments, before decoding the fused features based on each denoising feature to obtain the target video sequence image set after super-resolution reconstruction of the video frame sequence image set, the video reconstruction device 40 further includes a second feature extraction module (not shown), used to perform second feature extraction based on the video frame sequence image set to obtain video features; wherein, the first feature extraction includes spatial compression and temporal compression, the second feature extraction includes pixel rearrangement and causal convolution, and the video features and the fused features have the same feature dimension; the feature decoding module 44 further includes a feature fusion module (not shown), used to fuse based on the video features and the first fused features to obtain a second fused feature; the feature decoding module 44 further includes a fusion decoding module (not shown), used to decode based on the second fused feature to obtain the target video sequence image set.

[0076] In some disclosed embodiments, after performing a first feature extraction based on a set of video frame sequence images in the video data to obtain a target feature map, and before constructing several feature sets based on the correlation between the sub-feature maps obtained by dividing the target feature map in the spatial dimension, the video reconstruction device 40 further includes a noise injection module (not shown) for injecting noise of a preset intensity into the target feature map to obtain an updated target feature map; wherein, the preset intensity noise is used to weaken the feature structure of the target feature map at a preset intensity; the set construction module 42 further includes a construction sub-module (not shown) for constructing several feature sets based on the correlation between the sub-feature maps obtained by dividing the updated target feature map.

[0077] In some disclosed embodiments, the set construction module 42 further includes a partition number acquisition module (not shown), used to acquire a first partition number of the target feature map in the height direction and a second partition number of the target feature map in the width direction; the set construction module 42 further includes a sub-feature map partitioning module (not shown), used to partition the target feature map in the height direction and the width direction based on the first partition number and the second partition number respectively, to obtain several sub-feature maps; wherein, the number of channels in the channel dimension of the sub-feature map is consistent with the number of channels in the channel dimension of the target feature map.

[0078] In some disclosed embodiments, the target video sequence image set is obtained by processing the video frame sequence image set by a video reconstruction model, and the video reconstruction model is the target student model obtained by knowledge distillation of the target teacher model. The video reconstruction model includes a target encoder, a target diffusion attention block, and a target decoder. The target encoder is used to extract the first feature based on the video frame sequence image set in the video data to obtain a target feature map. The target diffusion attention block is used to construct several feature sets based on the correlation between the sub-feature maps obtained by dividing the target feature map in the spatial dimension, and to perform denoising prediction based on each feature set to obtain the denoised features corresponding to each feature set. The target decoder is used to decode based on the first fusion feature of each denoised feature to obtain the target video sequence image set after super-resolution reconstruction of the video frame sequence image set.

[0079] In some disclosed embodiments, the video reconstruction apparatus 40 further includes a sample data acquisition module (not shown) for acquiring a sample dataset; wherein the sample dataset contains at least a plurality of sample videos and sample images not less than a resolution threshold; the video reconstruction apparatus 40 further includes a sample data degradation module (not shown) for performing degradation processing based on the sample dataset to obtain target training data; wherein the degradation processing includes at least one of optical flow-guided motion blur and model-guided noise injection; the video reconstruction apparatus 40 further includes a teacher model training module (not shown) for training an initial teacher model based on the target training data to obtain a target teacher model; wherein the model structure of the initial teacher model includes a sequentially connected teacher encoder, a multi-step diffusion attention block, and a teacher decoder; the video reconstruction apparatus 40 further includes a student model training module (not shown) for distilling the model knowledge of the target teacher model into a target student model with a first mapping relationship between the input features and output features of the multi-step diffusion attention block in the target teacher model as a first training objective and a second mapping relationship between the input features and output video of the teacher decoder in the target teacher model as a second training objective to obtain a video reconstruction model; wherein the target diffusion attention block in the target student model is a single-step diffusion attention block.

[0080] In some disclosed embodiments, before video reconstruction is performed on the video frame sequence image set to be reconstructed, the video reconstruction model masks several historical video frames in the video data that exceed a preset number of historical frames before the video frame sequence image set to be reconstructed; and / or, the target diffusion attention block uses a sliding window of a preset size to obtain historical attention key-value pairs from the super-resolution reconstruction process that has already been completed before the video frame sequence image set to be reconstructed; and / or, the target diffusion attention block is deployed on a first processor, the target decoder is deployed on a second processor, and there is a communication connection between the first processor and the second processor.

[0081] Please see Figure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 50 includes at least a memory 51 and a processor 52 coupled to each other. The memory 51 stores at least program instructions, and the processor 52 is used to execute the program instructions to implement the steps in any of the above-described video reconstruction method embodiments. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here. Exemplarily, the electronic device 50 may include, but is not limited to, smartphones, tablet computers, etc., and the specific type of the electronic device 50 is not limited here.

[0082] Specifically, processor 52 controls itself and memory 51 to implement the steps in any of the video reconstruction method embodiments described above. Processor 52 can also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 52 can be implemented using integrated circuit chips.

[0083] In the above scheme, electronic device 50 performs first feature extraction based on the video frame sequence image set to be reconstructed from the video data to obtain a target feature map. Based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, several feature sets are constructed, each containing at least one sub-feature map. Denoising prediction is performed based on each feature set to obtain denoised features corresponding to each feature set. Decoding is performed based on the first fusion feature of each denoised feature to obtain a target video sequence image set with higher resolution after super-resolution reconstruction of the video frame sequence image set. Therefore, constructing several feature sets based on the correlation between sub-feature maps helps to ensure strong mutual influence between sub-feature maps belonging to the same feature set, while weaker mutual influence between sub-feature maps belonging to different feature sets. Therefore, using feature sets as the basic unit for denoising prediction and performing denoising prediction on each feature set separately can skip redundant interactions between sub-feature maps not belonging to the same feature set, reducing redundant calculations introduced when directly processing the target feature map uniformly, thereby improving processing efficiency. Thus, it can improve the efficiency of super-resolution reconstruction of video data.

[0084] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 61 that can be executed by a processor. The program instructions 61 are used to implement the steps in any of the above-described video reconstruction method embodiments.

[0085] In the above scheme, the computer-readable storage medium 60 performs first feature extraction based on the video frame sequence image set to be reconstructed from the video data to obtain a target feature map. Based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, several feature sets are constructed, each containing at least one sub-feature map. Denoising prediction is performed based on each feature set to obtain denoising features corresponding to each feature set. Decoding is performed based on the first fusion feature of each denoising feature to obtain a target video sequence image set with higher resolution after super-resolution reconstruction of the video frame sequence image set. Therefore, constructing several feature sets based on the correlation between sub-feature maps helps to ensure strong mutual influence between sub-feature maps belonging to the same feature set, while weaker mutual influence between sub-feature maps belonging to different feature sets. Therefore, using feature sets as the basic unit for denoising prediction and performing denoising prediction on each feature set separately can skip redundant interactions between sub-feature maps not belonging to the same feature set, reducing redundant calculations introduced when directly processing the target feature map uniformly, thereby improving processing efficiency. Thus, it can improve the efficiency of super-resolution reconstruction of video data.

[0086] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0087] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0088] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0089] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0090] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0091] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A video reconstruction method, characterized in that, include: The first feature is extracted based on the video frame sequence image set in the video data to obtain the target feature map; wherein, the video frame sequence image set is the set of continuous image frames to be reconstructed in the video data; Based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, several feature sets are constructed; wherein, the feature set contains at least one of the sub-feature maps; Denoising prediction is performed based on each of the aforementioned feature sets to obtain the denoised features corresponding to each of the aforementioned feature sets; Decoding is performed based on the first fusion feature of each of the denoising features to obtain the target video sequence image set after super-resolution reconstruction of the video frame sequence image set; wherein the resolution of the target video sequence image set is higher than that of the video frame sequence image set.

2. The method according to claim 1, characterized in that, Before constructing several feature sets based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, the process includes: Based on each of the sub-feature maps, feature dimensionality reduction is performed to obtain the sub-feature maps after feature dimensionality reduction; Based on the correlation between the sub-feature maps obtained by dividing the target feature map, several feature sets are constructed, including: Calculate the correlation between the sub-feature maps after dimensionality reduction of each of the aforementioned features; Sub-feature maps after dimensionality reduction of several features whose relevance meets the preset association conditions are selected and divided into the same feature set.

3. The method according to claim 1, characterized in that, The step of performing denoising prediction based on each of the aforementioned feature sets to obtain denoised features corresponding to each feature set includes: Forward diffusion is performed on the sub-feature maps in the feature set to obtain noisy features; Attention is calculated based on the noise-adding features to obtain the enhancement features; Based on the enhanced features, reverse diffusion is performed to predict the denoised features corresponding to the feature set.

4. The method according to claim 1, characterized in that, The denoising prediction includes at least a diffusion mechanism that integrates the attention calculation process. Before performing denoising prediction based on each of the feature sets to obtain the denoised features corresponding to each feature set, the method further includes: Based on the historical denoising predictions already completed for several historical video frame sequence image sets preceding the current video frame sequence image set to be reconstructed, obtain the historical attention key-value pairs generated in the historical denoising predictions; The step of performing denoising prediction based on each of the aforementioned feature sets to obtain denoised features corresponding to each feature set includes: Using the historical attention key-value pairs as reference data, attention calculation is performed with the feature set to obtain enhanced features; Based on the enhanced features, reverse diffusion is performed to predict the denoised features corresponding to the feature set.

5. The method according to claim 1, characterized in that, Before decoding the fused features based on each of the denoising features to obtain the target video sequence image set after super-resolution reconstruction of the video frame sequence image set, the method further includes: Based on the video frame sequence image set, a second feature extraction is performed to obtain video features; wherein, the first feature extraction includes spatial compression and temporal compression, the second feature extraction includes pixel rearrangement and causal convolution, and the video features have the same feature dimension as the fused features; The decoding based on the fused features of each of the denoising features yields the target video sequence image set after super-resolution reconstruction of the video frame sequence image set, including: The second fusion feature is obtained by fusing the video features and the first fusion feature. The target video sequence image set is obtained by decoding based on the second fusion feature.

6. The method according to claim 1, characterized in that, After performing the first feature extraction on the video frame sequence image set in the video data to obtain the target feature map, and before constructing several feature sets based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, the method further includes: A preset intensity noise is injected into the target feature map to obtain an updated target feature map; wherein, the preset intensity noise is used to weaken the feature structure of the target feature map at a preset intensity. Based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map, several feature sets are constructed, including: Based on the correlation between the sub-feature maps obtained by dividing the updated target feature map in spatial dimensions, several feature sets are constructed.

7. The method according to claim 1, characterized in that, The steps for partitioning the target feature map in the spatial dimension include: Obtain the first number of divisions of the target feature map in the height direction, and obtain the second number of divisions of the target feature map in the width direction; The target feature map is divided into several sub-feature maps based on the first division number and the second division number in the height and width directions, respectively; wherein the number of channels in the channel dimension of the sub-feature map is the same as the number of channels in the channel dimension of the target feature map.

8. The method according to claim 1, characterized in that, The target video sequence image set is obtained by processing the video frame sequence image set by a video reconstruction model. The video reconstruction model is a target student model obtained by knowledge distillation of the target teacher model. The video reconstruction model includes a target encoder, a target diffusion attention block, and a target decoder. The target encoder is used to extract a first feature based on the video frame sequence image set in the video data to obtain a target feature map. The target diffusion attention block is used to construct several feature sets based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map. Denoising prediction is performed based on each of the feature sets to obtain the denoised features corresponding to each feature set. The target decoder is used to decode based on the first fusion feature of each of the denoised features to obtain the target video sequence image set after super-resolution reconstruction of the video frame sequence image set.

9. The method according to claim 8, characterized in that, The steps of knowledge distillation include: Obtain a sample dataset; wherein the sample dataset contains a number of sample videos and sample images of a resolution not less than a certain threshold; Degradation processing is performed on the sample dataset to obtain target training data; wherein, the degradation processing includes at least one of optical flow-guided motion blur and model-guided noise injection; The initial teacher model is trained based on the target training data to obtain the target teacher model; wherein, the model structure of the initial teacher model includes a sequentially connected teacher encoder, a multi-step diffusing attention block, and a teacher decoder; The first training objective is to use the first mapping relationship between the input features and output features of the multi-step diffusion attention block in the target teacher model as the first training objective, and the second training objective is to use the second mapping relationship between the input features and output video of the teacher decoder in the target teacher model as the second training objective. The model knowledge of the target teacher model is distilled into the target student model to obtain the video reconstruction model; wherein, the target diffusion attention block in the target student model is a single-step diffusion attention block.

10. The method according to claim 8, characterized in that, Before performing video reconstruction on the video frame sequence image set, the video reconstruction model masks several historical video frame sequence image sets in the video data that exceed the range of a preset number of historical frames preceding the video frame sequence image set. And / or, the target diffusion attention block uses a sliding window of a preset size to obtain historical attention key-value pairs from the super-resolution reconstruction process that has been completed before the video frame sequence image set in the video data; And / or, the target diffusion attention block is deployed on the first processor, the target decoder is deployed on the second processor, and there is a communication connection between the first processor and the second processor.

11. A video super-resolution reconstruction device, characterized in that, include: The feature extraction module is used to perform first feature extraction based on the video frame sequence image set in the video data to obtain a target feature map; wherein, the video frame sequence image set is a set of continuous image frames to be reconstructed in the video data; A set construction module is used to construct several feature sets based on the correlation between the sub-feature maps obtained by spatially partitioning the target feature map; wherein the feature set contains at least one of the sub-feature maps; The denoising prediction module is used to perform denoising prediction based on each of the feature sets to obtain the denoised features corresponding to each feature set. The feature decoding module is used to decode based on the first fused feature of each of the denoising features to obtain the target video sequence image set after super-resolution reconstruction of the video frame sequence image set; wherein the resolution of the target video sequence image set is higher than that of the video frame sequence image set.

12. An electronic device, characterized in that, The method includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the video reconstruction method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the video reconstruction method according to any one of claims 1 to 10.

Images