Video super-resolution model training method and device, video super-resolution processing method and device

Abstract

The present disclosure provides a video super-resolution model training method and device, and relates to the technical fields of computer vision, deep learning and the like. The specific implementation scheme is as follows: obtaining a video set including two or more video frames; obtaining a pre-established video super-resolution network, the video super-resolution network including a sliding window alignment subnetwork, a cyclic alignment subnetwork and a refinement alignment subnetwork, wherein the sliding window alignment subnetwork is used for feature extraction and fusion of multiple continuous video frames, the cyclic alignment subnetwork is used for bidirectional propagation of the features output by the sliding window alignment subnetwork to generate reconstructed features and alignment parameters, and the refinement alignment subnetwork is used for re-alignment and bidirectional propagation of the reconstructed features based on the alignment parameters to obtain super-resolved video frames; and performing the following training steps: inputting continuous video frames selected from the video set into the video super-resolution network, calculating a loss value of the video super-resolution network, and obtaining a video super-resolution model based on the loss value of the video super-resolution network. The implementation manner improves the effect of video super-resolution of the model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to the fields of computer vision, deep learning, etc., and in particular to a video super-resolution model training method and apparatus, a video super-resolution processing method and apparatus, electronic equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Video super-resolution (VSR) is a task that aims to recover a high-resolution counterpart from a given low-resolution video. In recent years, with the explosive growth of internet video data, VSR technology has attracted significant attention from researchers. Like image super-resolution, VSR is also an ill-posed problem. However, unlike image super-resolution, VSR requires not only attention to the corresponding low-resolution frames but also the utilization of information from consecutive frames in the video sequence. Some early VSR methods followed the ideas of image super-resolution, failing to effectively utilize the information from consecutive video frames, resulting in unsatisfactory results. Summary of the Invention

[0003] This disclosure provides a video super-resolution model training method and apparatus, a video super-resolution processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to the first aspect, a video super-resolution model training method is provided. This method includes: acquiring a video set containing two or more video frames; acquiring a pre-established video super-resolution network, which includes a sliding window alignment subnet, a cyclic alignment subnet, and a thinning alignment subnet. The sliding window alignment subnet is used for feature extraction and fusion of multiple consecutive video frames; the cyclic alignment subnet is used for bidirectional propagation of the features output by the sliding window alignment subnet to generate reconstructed features and alignment parameters; and the thinning alignment subnet performs re-alignment and bidirectional propagation of the reconstructed features based on the alignment parameters to obtain super-resolution video frames. The following training steps are performed: inputting consecutive video frames selected from the video set into the video super-resolution network and calculating the loss value of the video super-resolution network; and obtaining the video super-resolution model based on the loss value of the video super-resolution network.

[0005] According to the second aspect, a video super-resolution processing method is provided, the method comprising: acquiring multiple video frames to be processed; inputting the video frames to be processed into a video super-resolution model generated by the method described in any implementation of the first aspect; and outputting the video super-resolution processing result of the video frames to be processed.

[0006] According to a third aspect, a video super-resolution model training device is provided, comprising: a video acquisition unit configured to acquire a video set including two or more video frames; a network acquisition unit configured to acquire a pre-established video super-resolution network, the video super-resolution network including: a sliding window alignment subnet, a cyclic alignment subnet, and a thinning alignment subnet, wherein the sliding window alignment subnet is used for feature extraction and fusion of multiple consecutive video frames, the cyclic alignment subnet is used for bidirectional propagation of the features output by the sliding window alignment subnet to generate reconstructed features and alignment parameters, and the thinning alignment subnet is used for re-aligning and bidirectional propagation of the reconstructed features based on the alignment parameters to obtain super-resolution video frames; a selection unit configured to input consecutive video frames selected from the video set into the video super-resolution network; a calculation unit configured to calculate the loss value of the video super-resolution network; and an acquisition unit configured to obtain a video super-resolution model based on the loss value of the video super-resolution network.

[0007] According to the fourth aspect, a video super-resolution processing apparatus is also provided, the apparatus comprising: an acquisition unit configured to acquire a plurality of video frames to be processed; and an input unit configured to input the video frames to be processed into a video super-resolution model generated by the apparatus as described in any implementation of the third aspect, to obtain a video super-resolution processing result of the video frames to be processed.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first or second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first or second aspect.

[0010] According to a seventh aspect, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in either the first or second aspect.

[0011] The video super-resolution model training method and apparatus provided in this disclosure first acquire a video set including two or more video frames; second, acquire a pre-established video super-resolution network, which includes a sliding window alignment subnet, a cyclic alignment subnet, and a thinning alignment subnet. The sliding window alignment subnet is used for feature extraction and fusion of multiple consecutive video frames, the cyclic alignment subnet is used for bidirectional propagation of the features output by the sliding window alignment subnet to generate reconstructed features and alignment parameters, and the thinning alignment subnet performs re-alignment and bidirectional propagation of the reconstructed features based on the alignment parameters to obtain super-resolution video frames; third, input consecutive video frames selected from the video set into the video super-resolution network; fourth, calculate the loss value of the video super-resolution network; and finally, obtain the video super-resolution model based on the loss value of the video super-resolution network. This disclosure, by training a video super-resolution network combining a sliding window network and a bidirectional cyclic network, enables the obtained video super-resolution model to possess the advantages of both networks; and by using a video super-resolution network to perform multi-stage super-resolution and thinning of the input video frames, the detail preservation effect of the video frame sequence is ensured, improving the model's video super-resolution performance.

[0012] The video super-resolution processing method and apparatus provided in the embodiments of this disclosure acquire multiple videos to be processed; input the videos to be processed into a video super-resolution model generated by the video super-resolution model training method of this embodiment, and obtain the video super-resolution processing result of the videos to be processed. Therefore, by employing a video super-resolution network including multiple stages of super-resolution, a reliable video super-resolution processing can be performed on the videos to be processed, ensuring the effectiveness of the super-resolution processing.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This is a flowchart of an embodiment of the video super-resolution model training method disclosed herein;

[0016] Figure 2 This is a schematic diagram of a video super-resolution network in one embodiment of this disclosure;

[0017] Figure 3 This is a flowchart of one embodiment of the video super-resolution processing method disclosed herein;

[0018] Figure 4 This is a schematic diagram of a structure of an embodiment of the video super-resolution model training device according to the present disclosure;

[0019] Figure 5 This is a schematic diagram of a structure of an embodiment of the video super-resolution processing apparatus according to the present disclosure;

[0020] Figure 6 This is a block diagram of an electronic device used to implement the video super-resolution model training method or video super-resolution processing method of the embodiments of this disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] For video frames within the same video segment, the content source may be the same, but there may be motion offsets, scene offsets, or lighting offsets. In this case, how to align the video frames and find a common reference point is the problem that video frame alignment needs to solve.

[0023] Video alignment methods extract motion information to align adjacent frames in a video with a target frame. These methods primarily include motion estimation compensation and deformable convolution. Motion estimation compensation extracts inter-frame motion information (e.g., using optical flow) and performs inter-frame warping operations based on this information to align the frames. Deformable convolution uses deformable convolution to shift the input features through a convolution operation. The kernel of deformable convolution is obtained by adding an offset to a traditional convolution kernel, as seen in the EDVR (Video Restoration with Enhanced Deformable Convolutional Networks) model.

[0024] Some early methods, inspired by image super-resolution, simply applied the architecture to video super-resolution, but failed to fully utilize the temporal information between adjacent frames. To address this issue, recent methods have designed many complex modules, such as bidirectional recurrent neural networks (BRNNs). The basic idea of ​​BRNNs is to propose two recurrent neural networks for each training sequence, one forward and one backward. The forward sub-network takes forward video frames as input, and the backward sub-network takes backward video frames as input. The recurrent neural network incorporates an optical flow alignment network (SpyNet) for alignment during propagation. A significant advantage of recurrent neural networks is their ability to utilize relevant information from preceding and following frames during the mapping between input and output sequences. The improvement of BRNNs lies in assuming that the current output (the output of frame t) is related not only to the preceding frames but also to the following frames. For example, predicting a frame in a video requires considering both preceding and following frames. A BRNN consists of two recurrent neural networks stacked together, and the output is determined by the states of the hidden layers of these two recurrent neural networks.

[0025] Based on whether video frames are explicitly aligned, video super-resolution methods can be divided into two main categories: unaligned methods and aligned methods. While unaligned methods have simpler network structures, they typically perform poorly in recovering large-scale motion videos. Aligned methods usually have complex alignment modules with a large number of parameters. Therefore, designing a model with a small number of parameters, including an alignment module, and capable of achieving good recovery results is an urgent and challenging problem. Based on this, this disclosure proposes a video super-resolution model training method. Figure 1 A flowchart 100 is shown as an embodiment of a video super-resolution model training method according to the present disclosure, the video super-resolution model training method comprising the following steps:

[0026] Step 101: Obtain a video set that includes two or more video frames.

[0027] In this embodiment, the execution entity running the video super-resolution model training method can obtain the video set in various ways. For example, the execution entity can obtain the video set stored in the database server through a wired or wireless connection. Alternatively, a user can obtain the video set collected by the terminal by communicating with the terminal.

[0028] Here, the video set may include at least two or more video frames, which may consist of at least one video sequence. For the samples needed to train the video super-resolution network, consecutive video frames may be selected from the video set. The selected consecutive video frames may be a video sequence. By inputting the video sequence into the video super-resolution network, a high-resolution video sequence output by the video super-resolution network can be obtained.

[0029] The collection, storage, use, processing, transmission, provision, and disclosure of video frames and video sequences involved in this technical solution are performed after authorization and comply with relevant laws and regulations.

[0030] Step 102: Obtain the pre-established video super-resolution network.

[0031] The video super-resolution network includes a sliding window alignment subnet, a cyclic alignment subnet, and a fine-grained alignment subnet, such as... Figure 2 As shown, in the video super-resolution network, the sliding window alignment subnet 1, the cyclic alignment subnet 2, and the thinning alignment subnet 3 are connected sequentially. The sliding window alignment subnet 1 receives multiple consecutive input video frames (video sequences), which are processed sequentially by the cyclic alignment subnet 2 and the thinning alignment subnet 3 to obtain the super-resolution video frames output by the thinning alignment subnet 3. Figure 2 The video frames output by the refined alignment subnet 3 are also multiple consecutive video frames (video sequences). Compared to the multiple consecutive video frames input to the sliding window alignment subnet 1, the output video frames have a higher video resolution, achieving the effect of recovering high-resolution video from low-resolution video.

[0032] In this embodiment, the sliding window alignment subnet uses a sliding window method to estimate multiple consecutive video frames to obtain high-quality reference frames that are close to the labeled high-resolution video. For example, in the sliding window alignment subnet, 2N+1 (N≥1) consecutive video frames are used as input. In this input, the middle frame is represented as the reference frame, and the others are represented as adjacent frames. The sliding window alignment subnet includes an alignment module (PCD, Pyramid, Cascading and Deformable convolution) and a fusion module (TSA, Temporal and Spatial Attention). Each adjacent frame is aligned with the reference frame at the feature level by the alignment module, and the fusion module fuses the image information of different frames to obtain the fused image.

[0033] Optionally, in the sliding window alignment subnet, a pre-blurring module can be used before the alignment module to preprocess the blurred input video frames and improve the alignment accuracy.

[0034] In some optional implementations of this embodiment, the sliding window alignment subnet includes: a feature extraction module and a local fusion module; the feature extraction module is used to extract features from selected continuous video frames; the local fusion module is used to align the target frame features and adjacent frame features adjacent to the target frame features in the features of the selected continuous video frames to obtain at least one neighbor alignment feature; the local fusion module is also used to connect at least one neighbor alignment feature and the target frame feature in sequence, and then obtain the fused features through residual blocks.

[0035] In this optional implementation, the main function of the feature extraction module is to map the selected continuous video frames to a high-dimensional feature space to obtain the features of the high-dimensional video frames. The feature extraction module can be an encoder; for example, the feature extractor consists of two convolutional layers.

[0036] In this optional implementation, the neighbor alignment feature is used to characterize the features of adjacent frames that are aligned with the target frame features. The neighbor alignment feature and the target frame features can be connected together using the concat function.

[0037] Specifically, such as Figure 2 In the sliding window alignment subnet 1, there are two modules: a feature extraction module and a local fusion module (LFM). The feature extraction module is used to extract features from the selected consecutive video frames. The purpose of the local fusion module (LFM) is to first fuse the target frame features (e.g., the middle frame in the selected consecutive video frames) with the information of its adjacent frames (all video frames on both sides of the middle frame) from the features of the selected consecutive video frames. Then, the fused features are passed together with the features of the selected consecutive video frames to the next stage of the cyclic alignment subnet 2.

[0038] In this embodiment, the input to the sliding alignment subnet is any consecutive video frames (let's say N). After feature extraction by the feature extraction module, the features of N consecutive video frames are obtained. These features of N consecutive video frames are then sequentially input into the local fusion module (LFM). Taking the i-th (i>1) input among the features of N consecutive video frames as an example, when inputting the i-th feature, the i+1 and i-1 features are input simultaneously. After alignment and fusion in the local fusion module (LFM), the fused i-th feature is obtained.

[0039] In this embodiment, the Local Fusion Module (LFM) first aligns the features of adjacent frames with the features of the target frame, and then fuses the aligned adjacent features using the concat function and multiple residual blocks. This feature alignment can employ flow-guided deformation convolution. The fused features can be represented as shown in equation (1):

[0040] (1)

[0041] In equation (1), the target frame features These are intermediate frames, obtained by extracting features from selected consecutive video frames using a feature extraction module. The `concat` function combines features from adjacent frames with the target frame's features, and then passes them through a residual block. (A series of convolutional modules) are used to fuse features to obtain the fused features.

[0042] In this optional implementation, local feature fusion of the local fusion module is performed before feature propagation of video frames. This can enhance cross-frame feature fusion during feature propagation and improve the fusion effect of the sliding window aligned subnet in fusing video frame features.

[0043] In this embodiment, both the cyclic alignment subnet and the thinning alignment subnet can be bidirectional recurrent neural networks. Using bidirectional recurrent neural networks can make full use of the temporal information between video sequences, thereby improving the super-resolution effect.

[0044] like Figure 2 As shown, the cyclic alignment subnet 2 can be a traditional bidirectional recurrent neural network. The cyclic alignment subnet 2 includes a feature bidirectional propagation module and a reconstruction module. The feature bidirectional propagation module is used to bidirectionally propagate and align the features output by the sliding window alignment subnet 1 (which may include selected video features and fused features output by the local fusion module) to obtain reconstructed features and alignment parameters. The reconstruction module is used to reconstruct the reconstructed features output by the feature bidirectional propagation module to obtain a reconstructed graph. Figure 2 In the process, the reconstruction graph generated by the reconstruction module of the cyclically aligned subnet 2 can be used to calculate the branch loss of the cyclically aligned subnet 2.

[0045] In this embodiment, alignment parameters refer to the parameters used by the bidirectional recurrent neural network during inter-frame alignment processes such as optical flow and deformation convolution, such as the offset between two video frames output by the residual block during alignment. Compared to traditional bidirectional recurrent neural networks, the refined alignment subnet can fully utilize the alignment parameters from previous alignment operations of the recurrent alignment subnet, and perform further alignment based on the alignment parameters of the recurrent alignment subnet, thereby optimizing the alignment parameters of the recurrent alignment subnet and obtaining better alignment results.

[0046] Bidirectional recurrent neural networks generally include residual networks (ResNet). The residual network of a bidirectional recurrent neural network has multiple residual blocks. Each residual block can use different principles to fuse features. For example, the offset (the deviation between the reference frame and the adjacent frame) generated by the alignment module can be fused to obtain the fused offset; or the offset and weight values ​​(the weight values ​​corresponding to different offsets) generated by the alignment module can be fused to obtain the fused offset and fused weight values.

[0047] Unlike image super-resolution, video super-resolution typically requires aligning adjacent frames with the current frame to better integrate information from both. The role of alignment is particularly significant in large-scale motion video super-resolution tasks. In the use of bidirectional recurrent networks, there are often multiple identical alignment operations. To fully utilize the results of previous alignment operations, a refinement alignment module is employed in the refinement alignment subnet to leverage previously aligned parameters and achieve better alignment results.

[0048] In some optional implementations of this embodiment, such as Figure 2 As shown, the above-mentioned refined alignment subnet 3 may include: a feature bidirectional propagation module, a refined alignment module, and a reconstruction module; the refined alignment module pre-aligns the reconstructed features based on the alignment parameters to obtain pre-aligned features; based on the pre-aligned features, it generates residuals of the alignment parameters; based on the alignment parameters and the residuals of the alignment parameters, it generates new parameters; the feature bidirectional propagation module performs bidirectional propagation and alignment on the reconstructed features based on the new parameters to generate reconstructed features; the reconstruction module is used to reconstruct the reconstructed features to obtain the super-resolution video frames corresponding to the reconstructed features.

[0049] In this optional implementation, the refinement alignment module uses the offset and weight generated by the cyclic alignment subnet 2 for pre-alignment, and the pre-alignment feature is represented as shown in equation (2):

[0050] (2)

[0051] in This is the first step in refining the alignment of subnets. The input feature (also the first input feature of the cyclically aligned subnet output) (one reconstructed feature) and It is the first in the cyclically aligned subnet. The feature is directed towards the first The offset and weight values ​​for feature alignment. It is a deformable convolution. Then, pre-aligned features and To generate offset and weight residuals As shown in equation (3):

[0052] (3)

[0053] The last two pairs of offsets and weights are used for final alignment to obtain the reconstructed features. As shown in equation (4):

[0054] (4)

[0055] In this optional implementation, the feature bidirectional propagation module is used to perform bidirectional propagation and alignment of the reconstructed features using a bidirectional recurrent neural network to generate reconstructed features.

[0056] In this optional implementation, the reconstruction module is used to restore the features of the video frame to obtain the restored video frame.

[0057] Optionally, the refined alignment subnet may also include a pixel shuffle module, which can select pixels from the video frames output by the reconstruction module to obtain better super-resolution video frames.

[0058] This optional implementation provides a refined alignment subnet that utilizes the alignment parameters of the cyclic alignment subnet to achieve better alignment results. Specifically, the refined alignment module first pre-aligns the input features of the refined alignment subnet using the offsets and weights generated during feature alignment in the cyclic alignment subnet. Then, it uses the pre-aligned features to generate residuals of offsets and weights. Finally, these two pairs of offsets and weights are used for the final feature alignment, thereby improving the alignment performance of the video super-resolution network and ensuring the processing performance of the video super-resolution model on video sequences.

[0059] Step 103: Select consecutive video frames from the video set and input them into the video super-resolution network.

[0060] The selected consecutive video frames are video sequences, which have multiple consecutive video frames.

[0061] In this embodiment, the executing entity can select multiple consecutive video frames from the video set obtained in step 101, and execute the training steps from steps 103 to 105 to complete one iteration of training for the video super-resolution network. The selection method and number of video frames from the video set are not limited in this application, nor are the number of iterations for training the video super-resolution network limited. For example, in one iteration, multiple consecutive video frames can be randomly selected, and these selected consecutive video frames can be labeled with ground truth values. The loss value of the video super-resolution network is calculated using the ground truth values ​​of the selected consecutive video frames, and the parameters of the video super-resolution network are adjusted accordingly.

[0062] In this embodiment, the selected consecutive video frames are simultaneously input into the video super-resolution network in chronological order. The video super-resolution network performs super-resolution processing on the selected consecutive video frames to obtain a super-resolution result, which consists of multiple high-resolution video frames, thus improving the resolution compared to the selected consecutive video frames.

[0063] Step 104: Calculate the loss value of the video super-resolution network.

[0064] In this embodiment, during each iteration of training of the video super-resolution network, multiple consecutive video frames are selected from the video set and input into the video super-resolution network. The loss value of the video super-resolution network is calculated based on the loss function pre-set for the video super-resolution network.

[0065] In this embodiment, the loss function of the video super-resolution network can be the mean squared error function, which is the expectation of the squared difference between the predicted (estimated) value and the true value of the video super-resolution network. During the iterative training of the video super-resolution network, the gradient descent algorithm can be used to minimize the loss function of the video super-resolution network, thereby iteratively optimizing the network parameters of the video super-resolution network.

[0066] The gradient is essentially a vector representing the directional derivative of a loss function at a given point, where the loss function reaches its maximum value along that direction. In other words, the loss function changes most rapidly and at its highest rate of change along that direction at that point. In deep learning, the main task of a neural network is to find the optimal network parameters (weights and biases) during learning; these optimal parameters are those that minimize the loss function.

[0067] To optimize each subnet in the video super-resolution network, a separate branch loss function can be set for the cyclic alignment subnet to calculate its branch loss. The loss value of the video super-resolution network is then used as the overall loss. In some optional implementations of this embodiment, calculating the loss value of the video super-resolution network includes: calculating the overall loss of the video super-resolution network; calculating the branch loss of the cyclic alignment subnet based on the pre-set branch loss function; and obtaining the loss value of the video super-resolution network based on the overall loss and the branch loss of the cyclic alignment subnet.

[0068] In this optional implementation, during each iteration of training of the video super-resolution network, the overall loss of the video super-resolution network is calculated based on the loss function pre-set for the video super-resolution network; the branch loss of the cyclic alignment subnet is calculated based on the branch loss function; the above-mentioned obtaining the loss value of the video super-resolution network based on the overall loss and the branch loss of the cyclic alignment subnet includes: adding the overall loss to the branch loss of the cyclic alignment subnet to obtain the loss value of the video super-resolution network.

[0069] In this embodiment, the loss function and branch loss function pre-set for the video super-resolution network can both adopt the mean squared error function.

[0070] Optionally, in each iteration of training the video super-resolution network, the loss value of the video super-resolution network based on the overall loss and the branch loss of the cyclic alignment subnet is obtained by: setting weight values ​​for the video super-resolution network and the cyclic alignment subnet respectively; multiplying the overall loss by the weight value of the video super-resolution network to obtain a first product value; multiplying the branch loss of the cyclic alignment subnet by the weight value of the cyclic alignment subnet to obtain a second product value; and adding the first product value and the second product value to obtain the loss value of the video super-resolution network.

[0071] In this optional implementation, by setting a loss function for the cyclic alignment subnet, the branch loss of the cyclic alignment subnet can be calculated separately, thereby enabling bidirectional propagation of features in the cyclic alignment subnet and adding a branch loss to the reconstruction result of the features generated by the cyclic alignment subnet. This makes the features of the cyclic alignment subnet closer to the real high-resolution feature space and improves the reliability of the super-resolution processing of the cyclic alignment subnet.

[0072] In some optional implementations of this embodiment, the above-mentioned calculation of the branch loss of the cyclic alignment subnet based on the pre-set branch loss function includes: obtaining the reconstruction graph of the cyclic alignment subnet, wherein the reconstruction graph is obtained by reconstructing the reconstruction features; and calculating the branch loss of the cyclic alignment subnet based on the reconstruction graph and the branch loss function.

[0073] In this embodiment, as Figure 2 The two reconstruction modules in the cyclic alignment subnet and the thinning alignment subnet shown have the same function and can both obtain reconstructed images. The principle of the reconstruction module is the conventional reconstruction principle. The implementation principle of the reconstruction module will not be elaborated here. To reconstruct features in the recurrent alignment subnet, the features are upsampled and convolved to obtain a reconstructed map. The branch loss function is then calculated using this feature. Specifically, the branch loss... The result is obtained by calculation using equation (5):

[0074] (5)

[0075] Equation (5) represents the branch loss function set for the cyclic alignment subnet, where, in Equation (5), This represents the true value of the video frame. For the reconstruction graph (features of the reconstruction module) (obtained after upsampling and convolution processing) This is the error value, which can be a fixed value preset for cyclic alignment of subnets.

[0076] In this optional implementation, the branch loss of the cyclic alignment subnet is calculated by reconstructing the graph output by the cyclic alignment subnet, ensuring the reliability of the loss value calculation for the cyclic alignment subnet and providing a reliable basis for the stable training of the cyclic alignment subnet.

[0077] Step 105: Based on the loss value of the video super-resolution network, obtain the video super-resolution model.

[0078] In this embodiment, the video super-resolution model is obtained by iteratively training the video super-resolution network and then tuning its parameters. The loss value of the video super-resolution network can be used to detect whether the video super-resolution network meets the training completion condition. After the video super-resolution network meets the training completion condition, the video super-resolution model is obtained.

[0079] In this embodiment, the training completion conditions include at least one of the following: the number of training iterations of the video super-resolution network reaches a predetermined iteration threshold, and the loss value of the video super-resolution network is less than a predetermined loss value threshold. The predetermined iteration threshold is an empirical value derived from the loss value of the video super-resolution network. For example, the predetermined iteration threshold for the video super-resolution network is 50,000 iterations. The predetermined loss value threshold for the video super-resolution network is 0.01.

[0080] Optionally, in this embodiment, in response to the video super-resolution network not meeting the training completion conditions, the relevant parameters in the video super-resolution network are adjusted to make the loss value of the video super-resolution network converge, and based on the adjusted video super-resolution network, the above training steps 103-105 are continued.

[0081] In this optional implementation, adjusting the relevant parameters of the video super-resolution network when the training completion conditions are not met can help the loss value of the video super-resolution network converge.

[0082] In this embodiment, if training is not complete, the loss value of the video super-resolution network can be converged by adjusting its parameters. Specifically, adjusting the relevant parameters in the video super-resolution network to converge its loss value includes: repeatedly adjusting the parameters or loss weights of any one of the subnetworks in the sliding window alignment subnetwork, the cyclic alignment subnetwork, and the refined alignment subnetwork by executing steps 103 to 105, so that the loss value of the video super-resolution network converges.

[0083] Optionally, during each iteration, the parameters of the sliding window alignment subnet, the cyclic alignment subnet, and the refinement of two or more subnets in the alignment subnet can be adjusted simultaneously to ensure that the loss value of the video super-resolution network gradually decreases until it stabilizes.

[0084] In this embodiment, the above-mentioned method of obtaining a video super-resolution model based on the loss value of the video super-resolution network includes: in response to the video super-resolution network meeting the training completion condition, using the video super-resolution network that meets the training completion condition as the video super-resolution model; and using the output of the refined alignment subnet as the output of the video super-resolution model.

[0085] In some optional implementations of this embodiment, obtaining the video super-resolution model based on the loss value of the video super-resolution network includes: in response to the video super-resolution network meeting the training completion condition, using the video super-resolution network as the video super-resolution model; adding the features fused by the sliding window alignment subnet to the super-resolution video frames output by the refined alignment subnet to obtain the output of the video super-resolution model.

[0086] In this optional implementation, the content of adding the features fused from the sliding window aligned subnet to the super-resolution video output from the refined aligned subnet is used as the output of the video super-resolution model, which can enrich the output of the video super-resolution model and improve its output effect.

[0087] This disclosure proposes a multi-stage video super-resolution network that combines the ideas of the sliding window method and the recurrent network method, using a multi-stage strategy for video super-resolution. Specifically, firstly, feature extraction and local feature fusion are performed on the input video frames in the sliding window alignment sub-network; then, the fused features are propagated in the recurrent alignment sub-network, and a branch loss is introduced to enhance feature alignment during the propagation process; finally, a refinement alignment module is introduced in the refinement alignment sub-network to reuse the alignment parameters generated by the recurrent alignment sub-network for feature refinement alignment and enhanced propagation, resulting in the super-resolution video frames.

[0088] The video super-resolution model obtained by the embodiments of this disclosure achieves a PSNR (Peak Signal-to-Noise Ratio) of 28.13 on the vid4 dataset (a commonly used dataset in the field of video super-resolution) with only 1.45M parameters. Among current lightweight video super-resolution methods, the video super-resolution model of this disclosure achieves the highest PSNR and SSIM (structural similarity index measurement) on four standard video super-resolution test datasets (REDS4 dataset, UDM10 dataset, Vimeo-90K-T dataset, and Vid4 dataset) with the fewest parameters.

[0089] The video super-resolution model training method provided in this disclosure firstly acquires a video set including two or more video frames; secondly, it acquires a pre-established video super-resolution network, which includes a sliding window alignment subnet, a cyclic alignment subnet, and a thinning alignment subnet. The sliding window alignment subnet is used for feature extraction and fusion of multiple consecutive video frames, the cyclic alignment subnet is used for bidirectional propagation of the features output by the sliding window alignment subnet to generate reconstructed features and alignment parameters, and the thinning alignment subnet realigns and propagates the reconstructed features based on the alignment parameters to obtain super-resolution video frames; thirdly, it inputs consecutive video frames selected from the video set into the video super-resolution network; fourthly, it calculates the loss value of the video super-resolution network; and finally, it obtains the video super-resolution model based on the loss value of the video super-resolution network. This disclosure, by training a video super-resolution network combining a sliding window network and a bidirectional cyclic network, enables the obtained video super-resolution model to possess the advantages of both networks. Furthermore, by using a video super-resolution network to perform multi-stage super-resolution and thinning of the input video frames, it ensures the preservation of details in the video frame sequence and improves the model's video super-resolution performance.

[0090] Furthermore, based on the video super-resolution model training method provided in the above embodiments, this disclosure also provides an embodiment of a video super-resolution processing method, which combines artificial intelligence fields such as computer vision and deep learning.

[0091] See Figure 3 The diagram illustrates a flow 300 of an embodiment of the video super-resolution processing method according to the present disclosure. The video super-resolution processing method provided in this embodiment includes the following steps:

[0092] Step 301: Obtain multiple video frames to be processed.

[0093] In this embodiment, the multiple video frames to be processed can be multiple consecutive video frames. Each video frame to be processed can be a video frame including information such as people, objects, and scenery. By processing the video frames to be processed through a video super-resolution model, the result of video super-resolution processing can be obtained. The execution subject of the video super-resolution processing method can obtain the video frames to be processed in various ways. For example, the execution subject can obtain the video frames to be processed stored in the database server through a wired or wireless connection. Alternatively, the execution subject can also receive video frames to be processed in real time from a terminal or other devices.

[0094] Step 302: Input the video frame to be processed into the video super-resolution model and output the video super-resolution processing result of the video frame to be processed.

[0095] In this embodiment, the executing entity can input the video frame to be processed obtained in step 301 into the video super-resolution model to obtain the video super-resolution processing result of the obtained video frame to be processed. It should be noted that the video super-resolution processing result is a video frame after high-resolution super-resolution processing of the obtained video frame to be processed. Based on the structure of the video super-resolution model, it can be seen that the video frame in the obtained video super-resolution processing result has a higher resolution than the obtained video frame to be processed.

[0096] In this embodiment, the video super-resolution model can be as described above. Figure 1 The training was performed using the method described in the embodiments; for details of the training process, please refer to [link / reference]. Figure 1 The relevant descriptions of the embodiments will not be repeated here.

[0097] In some optional implementations of this embodiment, the method further includes: upsampling the video frame to be processed to obtain a sampled video frame; and adding the video super-resolution processing result to the sampled video frame to obtain a processed video frame.

[0098] like Figure 2As shown, the video frame S to be processed is input into the video super-resolution model (formed by sliding window alignment subnet 1, cyclic alignment subnet 2, and thinning alignment subnet 3) and the upsampler 4 respectively. Upsampling is performed in the upsampler 4 to obtain the sampled video frame. The video super-resolution processing result is obtained by processing through the video super-resolution model. The sampled video frame is added to the video super-resolution processing result output by the video super-resolution model to obtain the processed video frame S'.

[0099] In this optional implementation, the video super-resolution processing result is added to the sampled video frame, which makes the video frame to be processed more fully effective and improves the reliability of the video super-resolution processing result.

[0100] The video super-resolution processing method provided in this disclosure acquires multiple videos to be processed; the videos to be processed are input into a video super-resolution model generated using the video super-resolution model training method of this embodiment to obtain the video super-resolution processing result of the videos to be processed. Therefore, by employing a video super-resolution network that includes multiple stages of super-resolution, a reliable video super-resolution processing method can be used to process the videos to be processed, ensuring the effectiveness of the super-resolution processing.

[0101] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a video super-resolution model training device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0102] like Figure 4 As shown, the video super-resolution model training device 400 provided in this embodiment includes: a video acquisition unit 401, a network acquisition unit 402, a selection unit 403, a calculation unit 404, and an acquisition unit 405. The video acquisition unit 401 can be configured to acquire a video set including two or more video frames. The network acquisition unit 402 can be configured to acquire a pre-established video super-resolution network, which includes: a sliding window alignment subnet, a cyclic alignment subnet, and a thinning alignment subnet. The sliding window alignment subnet is used for feature extraction and fusion of multiple consecutive video frames. The cyclic alignment subnet is used for bidirectional propagation of the features output by the sliding window alignment subnet to generate reconstructed features and alignment parameters. The thinning alignment subnet realigns and bidirectionally propagates the reconstructed features based on the alignment parameters to obtain the super-resolution video frames. The selection unit 403 can be configured to input consecutive video frames selected from the video set into the video super-resolution network. The calculation unit 404 can be configured to calculate the loss value of the video super-resolution network. The aforementioned unit 405 can be configured to obtain a video super-resolution model based on the loss value of the video super-resolution network.

[0103] In this embodiment, the specific processing and technical effects of the video acquisition unit 401, network acquisition unit 402, selection unit 403, calculation unit 404, and acquisition unit 405 in the video super-resolution model training device 400 can be found in the following references. Figure 1 The relevant descriptions of steps 101, 102, 103, 104, and 105 in the corresponding embodiments will not be repeated here.

[0104] In some optional implementations of this embodiment, the calculation unit 404 includes: an overall calculation module (not shown in the figure), a local calculation module (not shown in the figure), and a result module (not shown in the figure). The overall calculation module can be configured to calculate the overall loss of the video super-resolution network. The local calculation module can be configured to calculate the branch loss of the cyclic alignment subnet based on a pre-set branch loss function. The result module can be configured to obtain the loss value of the video super-resolution network based on the overall loss and the branch loss of the cyclic alignment subnet.

[0105] In some optional implementations of this embodiment, the aforementioned local calculation module includes: an acquisition submodule (not shown in the figure) and a calculation submodule (not shown in the figure). The acquisition submodule can be configured to acquire the reconstruction graph of the cyclically aligned subnet, the reconstruction graph being obtained by reconstructing the reconstruction features. The calculation submodule can be configured to calculate the branch loss of the cyclically aligned subnet based on the reconstruction graph and the branch loss function.

[0106] In some optional implementations of this embodiment, the sliding window alignment subnet includes a feature extraction module and a local fusion module. The feature extraction module extracts features from selected consecutive video frames; the local fusion module aligns the target frame features and adjacent frame features from the selected consecutive video frames to obtain at least one neighbor-aligned feature; the local fusion module further connects the at least one neighbor-aligned feature and the target frame feature sequentially, and then uses residual blocks to obtain the fused feature.

[0107] In some optional implementations of this embodiment, the refined alignment subnet includes: a feature bidirectional propagation module, a refined alignment module, and a reconstruction module. The refined alignment module pre-aligns the reconstructed features based on alignment parameters to obtain pre-aligned features; generates residuals of alignment parameters based on the pre-aligned features; and generates new parameters based on the alignment parameters and their residuals. The feature bidirectional propagation module performs bidirectional propagation and alignment of the reconstructed features based on the new parameters to generate reconstructed features. The reconstruction module reconstructs the reconstructed features to obtain super-resolution video frames corresponding to the reconstructed features.

[0108] In some optional implementations of this embodiment, the obtaining unit 405 includes: a shaping module (not shown in the figure) and an addition module (not shown in the figure). The shaping module can be configured to use the video super-resolution network as a video super-resolution model in response to the video super-resolution network meeting the training completion condition. The addition module can be configured to add the features fused from the sliding window aligned subnet to the super-resolution video frames output by the refined aligned subnet to obtain the output of the video super-resolution model.

[0109] The video super-resolution model training apparatus provided in the embodiments of this disclosure firstly involves a video acquisition unit 401 acquiring a video set including two or more video frames; secondly, a network acquisition unit 402 acquiring a pre-established video super-resolution network, which includes a sliding window alignment subnet, a cyclic alignment subnet, and a thinning alignment subnet. The sliding window alignment subnet is used for feature extraction and fusion of multiple consecutive video frames, the cyclic alignment subnet is used for bidirectional propagation of the features output by the sliding window alignment subnet to generate reconstructed features and alignment parameters, and the thinning alignment subnet performs realignment and bidirectional propagation of the reconstructed features based on the alignment parameters to obtain super-resolution video frames; thirdly, a selection unit 403 inputs consecutive video frames selected from the video set into the video super-resolution network; fourthly, a calculation unit 404 calculates the loss value of the video super-resolution network; and finally, an acquisition unit 405 obtains the video super-resolution model based on the loss value of the video super-resolution network. This disclosure enables the obtained video super-resolution model to possess the advantages of both networks by training a video super-resolution network that combines a sliding window network and a bidirectional recurrent network. Furthermore, by employing a video super-resolution network to perform multi-stage super-resolution and refinement on the input video frames, the detail preservation effect of the video frame sequence is ensured, thereby improving the model's performance in video super-resolution.

[0110] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a video super-resolution processing apparatus, which is similar to... Figure 3 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0111] like Figure 5 As shown, the video super-resolution processing apparatus 500 provided in this embodiment includes: an acquisition unit 501 and an input unit 502. The acquisition unit 501 can be configured to acquire multiple video frames to be processed. The input unit 502 can be configured to input the video frames to be processed as described above. Figure 3 The video super-resolution model generated by the device described in the embodiment outputs the video super-resolution processing result of the video frame to be processed.

[0112] In this embodiment, the specific processing of the acquisition unit 501 and the input unit 502 in the video super-resolution processing device 500, and the resulting technical effects, can be found in reference to [reference needed]. Figure 3 The relevant descriptions of steps 301 and 302 in the corresponding embodiments will not be repeated here.

[0113] In some optional implementations of this embodiment, the video super-resolution processing apparatus 500 further includes: a sampling unit (not shown in the figure) and an addition unit (not shown in the figure). The sampling unit can be configured to upsample the video frame to be processed to obtain a sampled video frame. The addition unit can be configured to add the video super-resolution processing result to the sampled video frame to obtain the processed video frame.

[0114] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0115] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0116] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0117] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0118] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as video super-resolution model training methods or video super-resolution processing methods. For example, in some embodiments, the video super-resolution model training method or video super-resolution processing method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the video super-resolution model training method or video super-resolution processing method described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured by any other suitable means (e.g., by means of firmware) to perform a video super-resolution model training method or a video super-resolution processing method.

[0120] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable video super-resolution model training device, video super-resolution processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0124] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0125] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0126] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0127] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a video super-resolution model, the method comprising: Get a video set that includes two or more video frames; A pre-established video super-resolution network is obtained, comprising: a sliding window alignment subnet, a cyclic alignment subnet, and a thinning alignment subnet. The sliding window alignment subnet is used to extract and fuse features from multiple consecutive video frames. The cyclic alignment subnet is used to perform bidirectional propagation and alignment of the features output by the sliding window alignment subnet to generate reconstructed features and alignment parameters. The thinning alignment subnet performs realignment and bidirectional propagation of the reconstructed features based on the alignment parameters to generate reconstructed features and reconstructs the reconstructed features to obtain the super-resolution video frames. Perform the following training steps: A series of video frames selected from the video set are input into the video super-resolution network, and the loss value of the video super-resolution network is calculated; based on the loss value of the video super-resolution network, a video super-resolution model is obtained. The sliding window aligned subnet includes: a feature extraction module and a local fusion module; The feature extraction module is used to extract features from selected consecutive video frames; The local fusion module is used to align the target frame features and adjacent frame features adjacent to the target frame features in the features of selected consecutive video frames to obtain at least one adjacent aligned feature. The local fusion module is further configured to connect the at least one neighbor-aligned feature and the target frame feature in sequence, and then obtain the fused feature through a residual block.

2. The method according to claim 1, wherein, The calculation of the loss value of the video super-resolution network includes: Calculate the overall loss of the video super-resolution network; The branch loss of the cyclic alignment subnet is calculated based on a pre-set branch loss function. The loss value of the video super-resolution network is obtained based on the overall loss and the branch loss of the cyclic alignment subnet.

3. The method according to claim 2, wherein, The calculation of the branch loss of the cyclically aligned subnet based on a pre-set branch loss function includes: Obtain the reconstruction map of the cyclic aligned subnet, which is obtained by reconstructing the reconstruction features; Based on the reconstructed graph and the branch loss function, the branch loss of the cyclic alignment subnet is calculated.

4. The method according to claim 1, wherein, The process of obtaining a video super-resolution model based on the loss value of the video super-resolution network includes: In response to the video super-resolution network meeting the training completion condition, the video super-resolution network is used as a video super-resolution model; The features fused from the sliding window aligned subnet are added to the super-resolution video frames output from the refined aligned subnet to obtain the output of the video super-resolution model.

5. A video super-resolution processing method, the method comprising: Acquire multiple video frames to be processed; The video frame to be processed is input into the video super-resolution model generated by the method described in any one of claims 1-4, and the video super-resolution processing result of the video frame to be processed is output.

6. The method according to claim 5, further comprising: The video frame to be processed is upsampled to obtain a sampled video frame; The video super-resolution processing result is added to the sampled video frame to obtain the processed video frame.

7. A video super-resolution model training device, the device comprising: The video acquisition unit is configured to acquire a video set that includes two or more video frames. The network acquisition unit is configured to acquire a pre-established video super-resolution network, which includes a sliding window alignment subnet, a cyclic alignment subnet, and a thinning alignment subnet. The sliding window alignment subnet is used to extract and fuse features from multiple consecutive video frames. The cyclic alignment subnet is used to perform bidirectional propagation and alignment of the features output by the sliding window alignment subnet to generate reconstructed features and alignment parameters. The thinning alignment subnet performs realignment and bidirectional propagation of the reconstructed features based on the alignment parameters to generate reconstructed features and reconstructs the reconstructed features to obtain the super-resolution video frames. The selection unit is configured to input consecutive video frames selected from the video set into the video super-resolution network; The computing unit is configured to calculate the loss value of the video super-resolution network; The obtained unit is configured based on the loss value of the video super-resolution network to obtain the video super-resolution model; The sliding window aligned subnet includes: a feature extraction module and a local fusion module; The feature extraction module is used to extract features from selected consecutive video frames; The local fusion module is used to align the target frame features and adjacent frame features adjacent to the target frame features in the features of selected consecutive video frames to obtain at least one adjacent aligned feature. The local fusion module is further configured to connect the at least one neighbor-aligned feature and the target frame feature in sequence, and then obtain the fused feature through a residual block.

8. The apparatus according to claim 7, wherein, The computing unit includes: The overall calculation module is configured to calculate the overall loss of the video super-resolution network; The local calculation module is configured to calculate the branch loss of the cyclically aligned subnet based on a pre-set branch loss function; The module is configured to obtain the loss value of the video super-resolution network based on the overall loss and the branch loss of the cyclic alignment subnet.

9. The apparatus according to claim 8, wherein, The local computing module includes: The acquisition submodule is configured to acquire a reconstruction graph of the cyclic alignment subnet, the reconstruction graph being obtained by reconstructing the reconstruction features; The calculation submodule is configured to calculate the branch loss of the cyclic alignment subnet based on the reconstructed graph and the branch loss function.

10. The apparatus according to claim 7, wherein, The obtaining unit includes: The shaping module is configured to use the video super-resolution network as a video super-resolution model in response to the video super-resolution network meeting the training completion condition. The addition module is configured to add the features fused from the sliding window aligned subnet to the super-resolution video frames output from the refined aligned subnet, thereby obtaining the output of the video super-resolution model.

11. A video super-resolution processing apparatus, the apparatus comprising: The acquisition unit is configured to acquire multiple video frames to be processed. The input unit is configured to input the video frame to be processed into a video super-resolution model generated by the apparatus as described in any one of claims 7-10, and output the video super-resolution processing result of the video frame to be processed.

12. The apparatus of claim 11, further comprising: The sampling unit is configured to upsample the video frame to be processed to obtain a sampled video frame; The addition unit is configured to add the video super-resolution processing result to the sampled video frame to obtain the processed video frame.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images