A video synchronization spatiotemporal super-resolution method based on inter-frame feature alignment

Abstract

The present invention provides a method and system for generating high-frame-rate and high-resolution videos based on synchronous spatiotemporal super-resolution. The method comprises: extracting frames of a video file one by one to form a video frame sequence; extracting features from each pair of adjacent frames to obtain feature vectors, and obtaining inter-frame motion representation vectors through an inter-frame motion alignment network; using RBPN to super-resolve the adjacent frames to obtain high-resolution frame vectors and low-resolution frame vectors corresponding to the adjacent frames; using a spatiotemporal super-resolution network to generate high-resolution frame vectors and low-resolution frame vectors corresponding to inserted frames; optimizing the high-resolution frame vectors corresponding to the adjacent frames and the inserted frames to obtain target high-resolution vectors corresponding to the adjacent frames and the inserted frames and generating three target video frames by upsampling; and re-encoding the generated target video frames into high-frame-rate and high-resolution videos.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and image processing, and more specifically, to a video synchronization spatiotemporal super-resolution method based on inter-frame feature alignment. Background Art

[0002] Video interpolation and video super-resolution are hot topics in the field of video enhancement in the current academic and industrial circles. Video interpolation technology can increase the frame rate of the video and make the video look smoother. Video interpolation technology mainly includes traditional linear interpolation represented by DVF and deep learning-based interpolation technology represented by DAIN. Video super-resolution technology can increase the resolution of the video. Through video super-resolution technology, 540P video can be upgraded to 4K quality video. Video super-resolution methods are mainly based on deep learning algorithms. Representative models include VSR, RBPN, Zooming Slow-Mo and other methods. Video interpolation and video super-resolution can be combined to form a spatiotemporal super-resolution task. Many studies have noticed the correlation between temporal super-resolution and spatial super-resolution, so that temporal super-resolution and spatial super-resolution can assist each other to achieve better super-resolution results.

[0003] In recent years, televisions that support 2K and even 4K have become increasingly popular. Some old movies and TV programs have poor display effects on these high-definition display devices, affecting the user's viewing experience. Most of the existing video spatiotemporal super-resolution technologies are based on the alignment of optical flow maps of adjacent frames, which depends on the accuracy of optical flow calculations. Inaccurate optical flow will cause a large number of artifacts. At the same time, the existing video spatiotemporal super-resolution methods first perform video interpolation and then perform video super-resolution. Compared with executing the two models independently, it only improves the final effect, but does not improve the overall computing performance. Therefore, completing the video spatiotemporal super-resolution task requires a lot of computing resources.

[0004] Chinese patent application "Video quality improvement method, device, electronic device and storage medium" (CN111654723A) discloses a video quality improvement method, device, electronic device and storage medium, which relate to the fields of video processing and deep learning. The specific implementation scheme is: extract adjacent first image frames and second image frames from the original video; extract first image features from the first image frame, and extract second image features from the second image frame; interpolate and generate target image features according to the feature continuity between the first image feature and the second image feature; upsample the first image feature, the target image feature and the second image feature, respectively, and restore the image to obtain the target images of each frame after interpolation; generate a target video according to the target images of each frame. However, in this method, video super-resolution is based on super-resolution of a single frame, which is less effective than the video super-resolution method based on the context of adjacent frames.

[0005] The Chinese patent application "Video spatiotemporal super-resolution enhancement method based on time slices" (CN112070677A) discloses a video spatiotemporal super-resolution enhancement method based on time slices, including: inputting the video to be enhanced and converting it into vertical time slices; sending the vertical time slices into the time slice super-resolution module to obtain the time slices after super-resolution, and then converting them into video A; the slice super-resolution module can capture the spatiotemporal correlation contained in the vertical time slices, so as to obtain a video consistent with the target video frame rate; sending video A into the feature transformation module to obtain video B consistent with the target video spatial resolution; sending video B into the optimization module to enhance video details and eliminate artifacts. This method explicitly models the time slices, better mines the spatiotemporal correlation information, captures the video information of long time series, and achieves the improvement of video frame rate and video spatial resolution.

[0006] Although the above two patents realize both temporal super-resolution and spatial super-resolution in one workflow, the two tasks are completed in series, and temporal super-resolution must be completed before spatial super-resolution. Compared with using two different super-resolution models for processing, the overall computing performance is not improved and a large amount of computing resources are consumed. Therefore, it is hoped to provide an improved video spatiotemporal super-resolution method that can avoid the artifact problem caused by optical flow alignment, and at the same time adjust the network structure to synchronize temporal super-resolution and spatial super-resolution, thereby reducing computing resource consumption. Summary of the invention

[0007] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0008] According to one aspect of the present invention, a video synchronization spatiotemporal super-resolution method is provided, the method comprising:

[0009] Extract frames from the video file one by one to form a video frame sequence;

[0010] For each pair of adjacent frames in the video frame sequence, the following operations are performed:

[0011] Extracting features from the adjacent frames to obtain feature vectors corresponding to the adjacent frames, and obtaining inter-frame motion representation vectors through an inter-frame motion alignment network;

[0012] Using a recurrent back projection network (RBPN) to super-resolution the adjacent frames to obtain a high-resolution frame vector and a low-resolution frame vector corresponding to the adjacent frames;

[0013] A spatiotemporal super-resolution network constructed using an upsampling module and a pyramid module is used to generate a high-resolution frame vector and a low-resolution frame vector corresponding to the inserted frame;

[0014] Optimizing the high-resolution frame vectors corresponding to the adjacent frames and the inserted frames to obtain target high-resolution vectors corresponding to the adjacent frames and the inserted frames and generating three target video frames by upsampling; and

[0015] Each target video frame generated after processing each pair of adjacent frames in the video frame sequence is re-encoded into a video with a high frame rate and high resolution.

[0016] According to an embodiment of the present invention, obtaining the inter-frame motion representation vector through the inter-frame motion alignment network further includes:

[0017] The feature vectors corresponding to the adjacent frames are concatenated and input into the convolution layer for preliminary alignment;

[0018] Input bottleneck convolutional layer to reduce the number of channels; and

[0019] The input residual blocks are aligned to obtain the aligned inter-frame motion representation vector.

[0020] According to a further embodiment of the present invention, the super-resolution of the adjacent frames using a cyclic back projection network (RBPN) further comprises:

[0021] The adjacent frames are spliced ​​and input into the convolution layer for fusion and dimension upgrading;

[0022] spliced ​​with the inter-frame motion representation vector and then input into the RBPN to obtain a high-resolution frame vector corresponding to the adjacent frame; and

[0023] The low-resolution frame vectors corresponding to the adjacent frames are obtained by downsampling.

[0024] According to a further embodiment of the present invention, the step of generating a high-resolution frame vector and a low-resolution frame vector corresponding to an inserted frame by using a spatiotemporal super-resolution network further comprises:

[0025] splicing the high-resolution frame vectors corresponding to the adjacent frames and the inter-frame motion representation vectors and inputting them into an upsampling module and a pyramid module composed of a residual block and a deconvolution module respectively;

[0026] The outputs of the two are fused and calculated to obtain a high-resolution frame vector corresponding to the inserted frame; and

[0027] A low-resolution frame vector corresponding to the inserted frame is obtained by downsampling.

[0028] According to a further embodiment of the present invention, optimizing the high-resolution frame vectors corresponding to the adjacent frames and the inserted frames further comprises:

[0029] The result obtained by concatenating the low-resolution frame vectors of the adjacent frame and the inserted frame and the inter-frame motion representation vector and performing upsampling is fused with the high-resolution frame vector of the adjacent frame to obtain a target high-resolution frame vector corresponding to the adjacent frame;

[0030] Obtaining a target low-resolution frame vector corresponding to the adjacent frames by downsampling; and

[0031] The target low-resolution frame vector corresponding to the adjacent frame is spliced ​​with the low-resolution frame vector of the inserted frame and the inter-frame motion representation vector, and the result of upsampling is fused with the high-resolution frame vector corresponding to the inserted frame to obtain the target high-resolution frame vector corresponding to the inserted frame.

[0032] According to another aspect of the present invention, a system for generating high frame rate and high resolution video based on synchronous spatiotemporal super-resolution is provided, the system comprising:

[0033] A video preprocessing module, wherein the video preprocessing module is configured to extract frames of the video file one by one to form a video frame sequence;

[0034] An inter-frame motion alignment module, wherein the inter-frame motion alignment module is configured to: extract features from each pair of adjacent frames in the video frame sequence to obtain feature vectors corresponding to the adjacent frames, and obtain an inter-frame motion representation vector through an inter-frame motion alignment network;

[0035] A space-time super-resolution module, wherein the space-time super-resolution module is configured to:

[0036] Using a recurrent back projection network (RBPN) to super-resolution the adjacent frames to obtain a high-resolution frame vector and a low-resolution frame vector corresponding to the adjacent frames;

[0037] A spatiotemporal super-resolution network constructed using an upsampling module and a pyramid module is used to generate a high-resolution frame vector and a low-resolution frame vector corresponding to the inserted frame;

[0038] an optimization and reconstruction module, the optimization and reconstruction module being configured to optimize the high-resolution frame vectors corresponding to the adjacent frames and the inserted frames to obtain target high-resolution vectors corresponding to the adjacent frames and the inserted frames and to generate three target video frames by upsampling; and

[0039] A video reconstruction module is configured to re-encode each target video frame generated after processing each pair of adjacent frames in the video frame sequence into a video with a high frame rate and high resolution.

[0040] According to one embodiment of the present invention, the inter-frame motion alignment module is further configured to:

[0041] The feature vectors corresponding to the adjacent frames are concatenated and input into the convolution layer for preliminary alignment;

[0042] Input bottleneck convolutional layer to reduce the number of channels; and

[0043] The input residual blocks are aligned to obtain the aligned inter-frame motion representation vector.

[0044] According to another embodiment of the present invention, the spatiotemporal super-resolution module is further configured to:

[0045] The adjacent frames are spliced ​​and input into the convolution layer for fusion and dimension upgrading;

[0046] spliced ​​with the inter-frame motion representation vector and then input into the RBPN to obtain a high-resolution frame vector corresponding to the adjacent frame; and

[0047] The low-resolution frame vectors corresponding to the adjacent frames are obtained by downsampling.

[0048] According to another embodiment of the present invention, the spatiotemporal super-resolution module is further configured to:

[0049] splicing the high-resolution frame vectors corresponding to the adjacent frames and the inter-frame motion representation vectors and inputting them into an upsampling module and a pyramid module composed of a residual block and a deconvolution module respectively;

[0050] The outputs of the two are fused and calculated to obtain a high-resolution frame vector corresponding to the inserted frame; and

[0051] A low-resolution frame vector corresponding to the inserted frame is obtained by downsampling.

[0052] According to another embodiment of the present invention, the optimization and reconstruction module is further configured to:

[0053] The result obtained by concatenating the low-resolution frame vectors of the adjacent frame and the inserted frame and the inter-frame motion representation vector and performing upsampling is fused with the high-resolution frame vector of the adjacent frame to obtain a target high-resolution frame vector corresponding to the adjacent frame;

[0054] Obtaining a target low-resolution frame vector corresponding to the adjacent frames by downsampling; and

[0055] The target low-resolution frame vector corresponding to the adjacent frame is spliced ​​with the low-resolution frame vector of the inserted frame and the inter-frame motion representation vector, and the result of upsampling is fused with the high-resolution frame vector corresponding to the inserted frame to obtain the target high-resolution frame vector corresponding to the inserted frame.

[0056] Compared with the solutions in the prior art, the video synchronization spatiotemporal super-resolution method and system provided by the present invention have at least the following advantages:

[0057] (1) Most video super-resolution methods are based on optical flow for alignment, and their performance depends on the accuracy of optical flow calculation. Inaccurate optical flow will cause a large number of artifacts. The present invention uses an inter-frame image feature alignment network, which can adaptively align at the feature level and effectively reduce artifacts; and

[0058] (2) The existing spatiotemporal super-resolution methods all perform video interpolation first and then perform video super-resolution. Compared with executing the two models independently, only the final effect is improved, but the overall computing performance is not improved. The present invention combines the inter-frame alignment required in both interpolation and super-resolution operations, and can perform spatiotemporal super-resolution simultaneously, thereby improving computing skills while ensuring video effects and saving computing resource consumption.

[0059] These and other features and advantages will become apparent by reading the following detailed description and by reference to the associated drawings.It is to be understood that the foregoing general description and the following detailed description are illustrative only and are not restrictive of the aspects of what is claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to understand the manner in which the above features of the present invention are used in detail, the above briefly summarized contents can be described in more detail with reference to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings only show some typical aspects of the present invention and should not be considered to limit its scope, because the description may allow for other equally effective aspects.

[0061] Figure 1 An example architecture diagram of a video synchronization spatiotemporal super-resolution system based on inter-frame feature alignment according to an embodiment of the present invention is shown.

[0062] Figure 2 A schematic structural diagram of an inter-frame motion alignment module according to an embodiment of the present invention is shown.

[0063] Figure 3 An example structural diagram of RBPN in a spatiotemporal super-resolution module according to an embodiment of the present invention is shown.

[0064] Figure 4 An example structural diagram of a spatiotemporal super-resolution network in a spatiotemporal super-resolution module according to an embodiment of the present invention is shown.

[0065] Figure 5 An example structural diagram of an optimization network in an optimization reconstruction module according to an embodiment of the present invention is shown.

[0066] Figure 6 The overall flow chart of a video synchronization spatiotemporal super-resolution method based on inter-frame feature alignment according to an embodiment of the present invention is shown.

[0067] Figure 7 A flowchart of a video synchronization spatiotemporal super-resolution method based on inter-frame feature alignment according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0068] The present invention will be described in detail below in conjunction with the accompanying drawings, and the features of the present invention will be further revealed in the following specific description.

[0069] Figure 1 FIG. 1 is an exemplary architecture diagram of a video synchronization spatiotemporal super-resolution system 100 based on inter-frame feature alignment according to an embodiment of the present invention. Figure 1 As shown in , the system 100 of the present invention includes: a video preprocessing module 101, an inter-frame motion alignment module 102, a spatiotemporal super-resolution module 103, an optimization and reconstruction module 104 and a video reconstruction module 105.

[0070] The video preprocessing module 101 may be configured to extract frames of the input video file one by one to form an initial video frame sequence. The inter-frame motion alignment module 102 may be configured to perform feature extraction on each pair of adjacent frames in the initial video frame sequence to obtain feature vectors corresponding to the two frames, and then obtain inter-frame motion representation vectors through an inter-frame motion alignment network. The spatiotemporal super-resolution module 103 may include a recurrent back-projection network (RBPN) and a spatiotemporal super-resolution network. The spatiotemporal super-resolution module 103 may be configured to use two RBPNs to perform super-resolution on two adjacent frames to obtain two high-resolution frame vectors corresponding to the two adjacent frames, respectively, and then obtain two low-resolution frame vectors corresponding to the two high-resolution frame vectors by downsampling. Further, the spatiotemporal super-resolution module 103 may also be configured to generate a high-resolution frame vector corresponding to the inserted frame using a spatiotemporal super-resolution network constructed using a residual block and a pyramid module, and then obtain a low-resolution frame vector corresponding to the high-resolution frame vector by downsampling. The optimization reconstruction module 104 can be configured to optimize the result vectors (i.e., the high-resolution frame vectors and low-resolution frame vectors of the adjacent frames and the inserted frames) generated by the spatiotemporal super-resolution module 103 to obtain target high-resolution vectors of two adjacent frames and the inserted frames, and then perform upsampling to generate the final three target video frames. The video reconstruction module 105 can be configured to re-encode each target frame generated after processing each pair of adjacent frames in the initial video frame sequence into a video to obtain a high frame rate and high resolution video.

[0071] Those skilled in the art will appreciate that the system of the present invention and its modules can be implemented in hardware or software, and the modules can be combined or combined in any suitable manner. In addition, the system of the present invention can use the REDS dataset (a high-quality video super-resolution dataset) to construct a loss function based on structural similarity (SSIM) for end-to-end training.

[0072] Figure 2 The schematic structure diagram of the inter-frame motion alignment module 102 according to an embodiment of the present invention is shown. The inter-frame motion alignment module 102 may include a feature extraction network and an inter-frame motion alignment network. The feature extraction network may be composed of a convolution layer and five residual blocks, and is used to extract the feature vector map corresponding to two adjacent frames of the input. The inter-frame motion alignment network may be composed of a convolution layer, a bottleneck layer and five residual blocks, and is used to splice the feature vectors of adjacent frames and input them into the convolution layer for preliminary alignment, and then input them into the bottleneck convolution layer to reduce the number of channels, and then input them into the residual block to complete the alignment, and obtain the aligned inter-frame motion representation vector. Figure 2 As shown, the inter-frame motion alignment module 102 aligns adjacent frames Input the feature extraction network to obtain the feature vector F corresponding to these two adjacent frames i 、F i+1 , and then the obtained eigenvector F i 、F i+1 Input the inter-frame motion alignment network to obtain the aligned inter-frame motion representation vector F i→i+1 .

[0073] Figure 3 An example structural diagram of a recurrent back projection network (RBPN) 300 in the spatiotemporal super-resolution module 103 according to an embodiment of the present invention is shown. The recurrent back projection network (RBPN) is a relatively mature network structure for super-resolution of continuous frame images, and the network structure is divided into three parts: initial feature extraction, multi-projection, and reconstruction. In the present invention, what is slightly different from RBPN is that the motion representation of adjacent frames is not a picture but a vector, so it is necessary to splice the adjacent frames and input them into three convolutional layers for fusion and dimensionality increase, and then splice (concat) with the inter-frame motion representation vector of the adjacent frames. The rest of the parts are consistent with RBPN. Figure 3 As shown in the figure, the initial feature extraction is divided into two parts, namely (1) the initial video frame Perform feature extraction to obtain the tensor L i ; and (2) the feature vector obtained by splicing adjacent frames and inputting three convolutional layers and the inter-frame motion representation vector F i→i+1 Perform feature extraction to obtain the tensor M i Then, the obtained tensor L i and the tensor M iInput MISR block and SISR block respectively, and obtain high-resolution feature vector through upsampling and What you get and Input the residual block to obtain the residual e i After that, the residual e i With SISR branch Add together to get the same frame as the original video The corresponding high-resolution frame vector H i , then the obtained high-resolution frame vector H i Downsample to obtain the corresponding low-resolution frame vector L i For use in space-time super-resolution networks.

[0074] Figure 4 An example structural diagram of the spatiotemporal super-resolution network 400 in the spatiotemporal super-resolution module 103 according to an embodiment of the present invention is shown. The spatiotemporal super-resolution module 103 may use two recurrent back projection networks (RBPNs) 300 to respectively super-resolve two original adjacent frames to obtain the corresponding high-resolution frame vectors and low-resolution frame vectors of the adjacent frames, and then use the spatiotemporal super-resolution network 400 to perform frame interpolation and super-resolution based on the original frames to obtain the high-resolution frame vectors and low-resolution frame vectors corresponding to the inserted frames. Specifically, the spatiotemporal super-resolution network 400 may concatenate the high-resolution frame vectors and inter-frame motion representation vectors corresponding to the adjacent frames and input them into the upsampling module and the pyramid module composed of the residual block and the deconvolution module, respectively, and the outputs of the two may be fused and calculated to obtain the high-resolution frame vector corresponding to the inserted frame, and the inserted frame may be downsampled to obtain the corresponding low-resolution frame vector. Figure 4 As shown, the spatiotemporal super-resolution network 400 will Figure 3 In the example shown, the high-resolution frame vector H corresponding to the adjacent frames is obtained i and H i+1 and the inter-frame motion representation vector F i→i+1 After splicing, the upsampling module and the pyramid module, which consist of 5 residual blocks and 5 deconvolution layers, are input respectively. The outputs of the two are fused to obtain the high-resolution frame vector H corresponding to the inserted frame. i+t’ Then, the spatiotemporal super-resolution network 400 can Figure 3 In the example shown, the low-resolution frame vector L corresponding to the adjacent frames is obtained i and L i+1 and the inter-frame motion representation vector F i→i+1 After splicing, the input is a downsampling module consisting of two residual blocks and one convolutional layer, and the high-resolution frame vector H corresponding to the obtained inserted frame is i+t’ Input is a downsampling module consisting of two residual blocks and one convolutional layer, and the outputs of the two are fused to obtain the low-resolution frame vector L corresponding to the inserted frame. i+t’ .

[0075] Figure 5 An example structural diagram of the optimization network 500 in the optimization reconstruction module 104 according to an embodiment of the present invention is shown. The optimization reconstruction module 104 is mainly used to further refine the output result of the previous module (i.e., the spatiotemporal super-resolution module 103). Specifically, for the original adjacent frames, the low-resolution frame vectors of the corresponding frames and the generated inserted frames and the inter-frame motion representation vectors can be spliced ​​and input into the network for reasoning. The obtained results are fused with the high-resolution frame vectors generated by RBPN to obtain the target high-resolution frame vectors corresponding to the adjacent frames; for the generated inserted frames, the target low-resolution frame vectors corresponding to the two adjacent frames are spliced ​​with the low-resolution frame vectors of the generated inserted frames and the inter-frame motion representation vectors and input into the network for reasoning. The two obtained results are fused with the high-resolution frame vectors corresponding to the inserted frames generated by the previous module to obtain the target high-resolution frame vectors corresponding to the inserted frames. Finally, the three target high-resolution frame vectors are input into the reconstruction network to be restored into three target video frames. As shown in FIG. Figure 5 As shown, the optimization network 500 combines the low-resolution frame vector L of each frame in the original adjacent frames and the generated inserted frame. i , L i+1 and L i+t’ And the inter-frame motion representation vector F i→i+1 After splicing, the input is an upsampling module consisting of 5 residual blocks and 5 deconvolution layers. The two results are respectively compared with the high-resolution frame vector H generated by RBPN i and H i+1 Fusion calculation to obtain the target high-resolution frame vector corresponding to adjacent frames and The target high-resolution frame vector and Downsample and compare with the low-resolution frame vector L generated by RBPN i and L i+1 Fusion calculation to obtain the target low-resolution frame vector corresponding to adjacent frames and Then, the obtained target low-resolution frame vectors can be and The low-resolution frame vector L corresponding to the inserted frame generated by the spatiotemporal super-resolution network i+t’ And the inter-frame motion representation vector F i→i+1 After splicing, the input is an upsampling module consisting of 5 residual blocks and 5 deconvolution layers. The two results are compared with the high-resolution frame vector H corresponding to the inserted frame generated by the spatiotemporal super-resolution network. i+t’ The target high-resolution frame vector corresponding to the inserted frame is obtained by fusion calculation

[0076] Figure 6 FIG. 6 shows an overall flow chart of a video synchronization spatiotemporal super-resolution method 600 based on inter-frame feature alignment according to an embodiment of the present invention. Figure 6 As shown, the method 600 includes the following steps:

[0077] Step S1: Extract the video file frames one by one to form a video frame sequence Where K is the number of video frames, 1≤i≤K.

[0078] Step S2: For adjacent frames Perform feature extraction to obtain the feature vectors F corresponding to these two frames i 、F i+1 , and the obtained feature vector F i 、F i+1 Input the inter-frame motion alignment network to obtain the inter-frame motion representation vector F i→i+1 .

[0079] Step S3: The adjacent frames and the inter-frame motion representation vector F i→i+1 and adjacent frames and the inter-frame motion representation vector F i→i+1 Input RBPN to get video frames The corresponding high-resolution frame vector H i and the low-resolution frame vector L i And video frames The corresponding high-resolution frame vector H i+1 and the low-resolution frame vector L i+1 .

[0080] Step S4: The adjacent frames The corresponding high-resolution frame vector H i , H i+1 and the low-resolution frame vector L i , L i+1 And the inter-frame motion representation vector F i→i+1 Input the spatiotemporal super-resolution network to calculate and generate the high-resolution frame vector H corresponding to the inserted frame i+t’ and the low-resolution frame vector L i+t’ .

[0081] Step S5: The adjacent frames and the high-resolution frame vector H corresponding to the generated inserted frame i , H i+1 , H i+t’ and the low-resolution frame vector L i , L i+1 , L i+t’ And the inter-frame motion representation vector F i→i+1 Input optimization network to get adjacent frames and the target high-resolution frame vector corresponding to the generated inserted frame Then the target high-resolution frame vector The input reconstruction network is restored into three target video frames

[0082] Step S6: cyclically executing steps 2 to 5 on the video frame sequence, and after the last group of adjacent frames is executed, re-encoding the obtained target video frames into a video to obtain a high frame rate and high resolution video.

[0083] Figure 7 The flowchart of a video synchronization spatiotemporal super-resolution method 700 based on inter-frame feature alignment according to an embodiment of the present invention is shown. The method 700 starts at step 701, where the video pre-processing module 101 extracts frames of a video file one by one to form a video frame sequence.

[0084] In step 702, the inter-frame motion alignment module 102 performs feature extraction on each pair of adjacent frames in the video frame sequence to obtain feature vectors corresponding to the two adjacent frames, and obtains an inter-frame motion representation vector through an inter-frame motion alignment network.

[0085] In step 703, the spatiotemporal super-resolution module 103 uses a recurrent back projection network (RBPN) to super-resolution two adjacent frames to obtain high-resolution frame vectors corresponding to the two adjacent frames, and obtains low-resolution frame vectors corresponding to the two adjacent frames by downsampling.

[0086] In step 704, the spatiotemporal super-resolution module 103 generates a high-resolution frame vector corresponding to the inserted frame using the spatiotemporal super-resolution network constructed using the upsampling module and the pyramid module, and obtains a low-resolution frame vector corresponding to the inserted frame by downsampling.

[0087] In step 705, the optimization reconstruction module 104 optimizes the high-resolution frame vectors and low-resolution frame vectors of the two adjacent frames and the inserted frame generated by the spatiotemporal super-resolution module 103 to obtain target high-resolution vectors of the two adjacent frames and the inserted frame, and generates three target video frames by upsampling.

[0088] In step 706 , the video reconstruction module 105 re-encodes each target video frame generated after processing each pair of adjacent frames in the video frame sequence into a video to obtain a video with a high frame rate and high resolution.

[0089] What has been described above includes examples of various aspects of the claimed subject matter. Of course, it is not possible to describe every conceivable combination of components or methods for the purpose of describing the claimed subject matter, but one of ordinary skill in the art will recognize that many further combinations and permutations of the claimed subject matter are possible. Thus, the disclosed subject matter is intended to encompass all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims.

Claims

1. A method for generating high frame rate and high resolution video based on synchronous spatiotemporal super-resolution, characterized in that: The method comprises: Extract frames from the video file one by one to form a video frame sequence; For each pair of adjacent frames in the video frame sequence, the following operations are performed: Extracting features from the adjacent frames to obtain feature vectors corresponding to the adjacent frames, and obtaining inter-frame motion representation vectors through an inter-frame motion alignment network; Using a recurrent back projection network (RBPN) to super-resolution the adjacent frames to obtain a high-resolution frame vector and a low-resolution frame vector corresponding to the adjacent frames; A spatiotemporal super-resolution network constructed using an upsampling module and a pyramid module is used to generate a high-resolution frame vector and a low-resolution frame vector corresponding to the inserted frame; Optimizing the high-resolution frame vectors corresponding to the adjacent frames and the inserted frames to obtain target high-resolution vectors corresponding to the adjacent frames and the inserted frames and generating three target video frames by upsampling; and Each target video frame generated after processing each pair of adjacent frames in the video frame sequence is re-encoded into a video with a high frame rate and high resolution.

2. The method according to claim 1, characterized in that The obtaining of the inter-frame motion representation vector by the inter-frame motion alignment network further comprises: The feature vectors corresponding to the adjacent frames are concatenated and input into the convolution layer for preliminary alignment; Input bottleneck convolutional layer to reduce the number of channels; and The input residual blocks are aligned to obtain the aligned inter-frame motion representation vector.

3. The method according to claim 1, characterized in that The super-resolution of the adjacent frames by using a cyclic back projection network (RBPN) further comprises: The adjacent frames are spliced ​​and input into the convolution layer for fusion and dimension upgrading; spliced ​​with the inter-frame motion representation vector and then input into the RBPN to obtain a high-resolution frame vector corresponding to the adjacent frame; and The low-resolution frame vectors corresponding to the adjacent frames are obtained by downsampling.

4. The method according to claim 1, characterized in that The generating of the high-resolution frame vector and the low-resolution frame vector corresponding to the inserted frame by using the spatiotemporal super-resolution network further comprises: splicing the high-resolution frame vectors corresponding to the adjacent frames and the inter-frame motion representation vectors and inputting them into an upsampling module and a pyramid module composed of a residual block and a deconvolution module respectively; The outputs of the two are fused and calculated to obtain a high-resolution frame vector corresponding to the inserted frame; and A low-resolution frame vector corresponding to the inserted frame is obtained by downsampling.

5. The method according to claim 1, characterized in that Optimizing the high-resolution frame vectors corresponding to the adjacent frames and the inserted frames further includes: The result obtained by splicing the low-resolution frame vectors and the inter-frame motion representation vector of the adjacent frame and the inserted frame and performing upsampling is fused with the high-resolution frame vector of the adjacent frame to obtain a target high-resolution frame vector corresponding to the adjacent frame; Obtaining a target low-resolution frame vector corresponding to the adjacent frames by downsampling; and The target low-resolution frame vector corresponding to the adjacent frame is spliced ​​with the low-resolution frame vector of the inserted frame and the inter-frame motion representation vector, and the result of upsampling is fused with the high-resolution frame vector corresponding to the inserted frame to obtain the target high-resolution frame vector corresponding to the inserted frame.

6. A system for generating high frame rate and high resolution video based on synchronous spatiotemporal super-resolution, characterized in that: The system comprises: A video preprocessing module, wherein the video preprocessing module is configured to extract frames of the video file one by one to form a video frame sequence; An inter-frame motion alignment module, wherein the inter-frame motion alignment module is configured to: extract features from each pair of adjacent frames in the video frame sequence to obtain feature vectors corresponding to the adjacent frames, and obtain an inter-frame motion representation vector through an inter-frame motion alignment network; A space-time super-resolution module, wherein the space-time super-resolution module is configured to: Using a recurrent back projection network (RBPN) to super-resolution the adjacent frames to obtain a high-resolution frame vector and a low-resolution frame vector corresponding to the adjacent frames; A spatiotemporal super-resolution network constructed using an upsampling module and a pyramid module is used to generate a high-resolution frame vector and a low-resolution frame vector corresponding to the inserted frame; an optimization and reconstruction module, the optimization and reconstruction module being configured to optimize the high-resolution frame vectors corresponding to the adjacent frames and the inserted frames to obtain target high-resolution vectors corresponding to the adjacent frames and the inserted frames and to generate three target video frames by upsampling; and A video reconstruction module is configured to re-encode each target video frame generated after processing each pair of adjacent frames in the video frame sequence into a video with a high frame rate and high resolution.

7. The system according to claim 6, characterized in that The inter-frame motion alignment module is further configured to: The feature vectors corresponding to the adjacent frames are concatenated and input into the convolution layer for preliminary alignment; Input bottleneck convolutional layer to reduce the number of channels; and The input residual blocks are aligned to obtain the aligned inter-frame motion representation vector.

8. The system according to claim 6, characterized in that The spatiotemporal super-resolution module is further configured to: The adjacent frames are spliced ​​and input into the convolution layer for fusion and dimension upgrading; spliced ​​with the inter-frame motion representation vector and then input into the RBPN to obtain a high-resolution frame vector corresponding to the adjacent frame; as well as The low-resolution frame vectors corresponding to the adjacent frames are obtained by downsampling.

9. The system according to claim 6, characterized in that The spatiotemporal super-resolution module is further configured to: splicing the high-resolution frame vectors corresponding to the adjacent frames and the inter-frame motion representation vectors and inputting them into an upsampling module and a pyramid module composed of a residual block and a deconvolution module respectively; The outputs of the two are fused and calculated to obtain a high-resolution frame vector corresponding to the inserted frame; and A low-resolution frame vector corresponding to the inserted frame is obtained by downsampling.

10. The system according to claim 6, characterized in that The optimization and reconstruction module is further configured to: The result obtained by splicing the low-resolution frame vectors and the inter-frame motion representation vector of the adjacent frame and the inserted frame and performing upsampling is fused with the high-resolution frame vector of the adjacent frame to obtain a target high-resolution frame vector corresponding to the adjacent frame; Obtaining a target low-resolution frame vector corresponding to the adjacent frames by downsampling; as well as The target low-resolution frame vector corresponding to the adjacent frame is spliced ​​with the low-resolution frame vector of the inserted frame and the inter-frame motion representation vector, and the result of upsampling is fused with the high-resolution frame vector corresponding to the inserted frame to obtain the target high-resolution frame vector corresponding to the inserted frame.

Images