A video spatio-temporal super-resolution reconstruction method and device and a storage medium

Abstract

The application provides a video space-time super-resolution reconstruction method and device and a storage medium. The super-resolution reconstruction method comprises: acquiring a frame feature sequence of a to-be-reconstructed video in a feature domain; inputting the frame feature sequence of the to-be-reconstructed video into a trained video space-time super-resolution reconstruction model to output a recovered video sequence, wherein the video space-time super-resolution reconstruction model comprises: an initial feature extraction module; a bidirectional alignment module configured to generate an initial feature of a to-be-interpolated intermediate frame of adjacent odd frames based on initial features of the adjacent odd frames; a feature extraction and fusion module based on space-time information, comprising a space-time correction module and k2 cascaded residual blocks connected in sequence; and a reconstruction layer configured to recover the video sequence. The bidirectional alignment module is used to generate a to-be-interpolated even frame between each pair of adjacent odd frames, the space-time information of adjacent frames is used for space-time correction to complete feature refinement and correction, and a more smooth and natural reconstruction effect is obtained, and the model is more lightweight and flexible.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and particularly relates to a video spatial-temporal super-resolution reconstruction method and device and a storage medium. BACKGROUND

[0002] The research on spatial-temporal video super-resolution reconstruction (STVSR) has good practical value, and the target is to improve the spatial and temporal resolution of the video frame sequence at the same time. An effective reconstruction algorithm can reconstruct the texture information that conforms to visual perception and matches the real situation, and improve the visual quality or assist in completing more accurate advanced visual tasks.

[0003] In the traditional video reconstruction algorithm, the whole process is divided into two stages of time reconstruction and space reconstruction, that is, first, a video frame interpolation (VFI) algorithm is used to synthesize the missing low-resolution (LR) frame, and then a video super-resolution reconstruction (VSR) model is used to generate a high-resolution (HR) frame sequence according to the LR frame sequence. The two-stage reconstruction method will make the final STVSR model large in size, and the two-stage reconstruction method ignores the internal correlation between the VFI and the VSR, which is contrary to the natural properties of the video signal, resulting in a low-quality final reconstruction result and poor visual effect. SUMMARY

[0004] The present application aims to at least solve the technical problems in the prior art, and provide a video spatial-temporal super-resolution reconstruction method and device and a storage medium.

[0005] In order to achieve the above-mentioned object of the present application, according to a first aspect of the present application, the present application provides a video spatio-temporal super-resolution reconstruction method, comprising: acquiring a frame feature sequence of a to-be-reconstructed video in a feature domain; inputting the frame feature sequence of the to-be-reconstructed video into a trained video spatio-temporal super-resolution reconstruction model, wherein the video spatio-temporal super-resolution reconstruction model outputs a recovered video sequence, and the video spatio-temporal super-resolution reconstruction model comprises: an initial feature extraction module, configured to extract an initial feature of each frame of the to-be-reconstructed video, denoted as an odd frame initial feature; a bidirectional alignment module, configured to generate an initial feature of a to-be-interpolated intermediate frame of adjacent odd frames based on the initial feature of the adjacent odd frames, and the initial feature of the to-be-interpolated intermediate frame is denoted as an even frame initial feature; k1 cascaded feature extraction and fusion modules based on spatio-temporal information, wherein the odd frame initial feature and the even frame initial feature are simultaneously input into the k1 cascaded feature extraction and fusion modules based on spatio-temporal information to obtain a deep feature corresponding to each frame, and the feature extraction and fusion module based on spatio-temporal information comprises a spatio-temporal correction module and k2 cascaded residual blocks connected in sequence; k1 and k2 are both positive integers; and a reconstruction layer, configured to perform recovery processing on the deep features of all frames to obtain the recovered video sequence.

[0006] In order to achieve the above-mentioned object of the present application, according to a second aspect of the present application, the present application provides a device based on the video spatio-temporal super-resolution reconstruction method according to the first aspect of the present application, comprising: an acquisition module, configured to acquire a frame feature sequence of a to-be-reconstructed video in a feature domain; and a reconstruction module, provided with a trained video spatio-temporal super-resolution reconstruction model, configured to input the frame feature sequence of the to-be-reconstructed video into the trained video spatio-temporal super-resolution reconstruction model to obtain a recovered video sequence.

[0007] In order to achieve the above-mentioned object of the present application, according to a third aspect of the present application, the present application provides a computer readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the video spatio-temporal super-resolution reconstruction method according to the first aspect of the present application.

[0008] The application has the beneficial technical effects that: in the video space-time super-resolution reconstruction model of the application, all frames of a video to be reconstructed are set as odd frames, a bidirectional alignment module is used to generate an even frame to be interpolated between each pair of adjacent odd frames, the bidirectional alignment module includes a forward alignment process and a backward alignment process, the information of the reverse motion process is considered for the reconstruction when the video sequence is played in reverse, the bidirectional alignment process can make the reconstructed high-frame-rate video more natural in the time dimension, a single frame is inserted between the adjacent odd frames (i.e. between the adjacent frames of the video to be reconstructed), the feature level frame rate is preliminarily improved, and the high-frame-rate video is prepared for the subsequent feature extraction and fusion module based on space-time information; the feature extraction and fusion module based on space-time information uses a space-time correction strategy to mine the space-time correlation between the frame sequences, further extracts features by densely connecting a plurality of residual blocks to obtain deep features, and further refines and corrects the features by using the space-time information of adjacent frames for space-time correction, so that a more smooth and natural reconstruction effect is obtained. The video space-time super-resolution reconstruction model of the application uses bidirectional alignment when mining the information between frames, and processes the time information in the subsequent feature extraction and fusion module based on space-time information, compared with the traditional two-stage process of inserting frames first and then reconstructing, the model is more lightweight and flexible. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 is a flowchart of the video space-time super-resolution reconstruction method in embodiment 1 of the application;

[0010] Figure 2 is a structural schematic diagram of the video space-time super-resolution reconstruction model in an application scenario in embodiment 1 of the application;

[0011] Figure 3 is a structural schematic diagram of the bidirectional alignment module in embodiment 1 of the application;

[0012] Figure 4 is a general structural diagram of the feature extraction and fusion module based on space-time information in embodiment 1 of the application;

[0013] Figure 5 is a structural schematic diagram of the residual block in embodiment 1 of the application;

[0014] Figure 6 is a structural schematic diagram of the space-time correction module in embodiment 1 of the application;

[0015] Figure 7 is a structural schematic diagram of the time correction block in embodiment 1 of the application;

[0016] Figure 8 is a structural schematic diagram of the space correction block in embodiment 1 of the application;

[0017] Figure 9The structural schematic diagram of the forward alignment sub-module in the embodiment 1 of the present application is shown in the figure.

[0018] Figure 10 The deformable convolution schematic diagram in the embodiment 1 of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0020] In the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms “longitudinal”, “transverse”, “upper”, “lower”, “front”, “rear”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, “outer” and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0021] In the description of the present application, unless otherwise specified and limited, it should be noted that the terms “mounting”, “connection”, “connection” should be understood broadly, for example, it can be mechanical connection or electrical connection, it can be the communication between two elements, it can be direct connection or indirect connection through intermediate medium, and the specific meaning of the above terms can be understood by those skilled in the art according to the specific circumstances.

[0022] Embodiment 1

[0023] The present embodiment discloses a video space-time super-resolution reconstruction method, and a flowchart thereof is shown in the figure Figure 1 as shown, comprising:

[0024] Step S1, obtaining the frame feature sequence of the to-be-reconstructed video in the feature domain.

[0025] The video to be reconstructed is usually a low-frame-rate and low-resolution video sequence. In a network, in order to facilitate the propagation of a video signal, the video is usually encoded, that is, the video is represented with the least number of bits on the premise of retaining necessary information, which is likely to cause degradation, and noise is ubiquitous. Such a video is usually referred to as a low-resolution (LR) video. Super-resolution (SR) reconstruction technology is to recover a high-quality image and video containing a large amount of detailed texture and suitable for a human visual perception system as much as possible according to spatial or spatio-temporal information of an LR input through image processing technology, deep learning tools and the like. In the embodiment, the video to be reconstructed is represented as wherein t represents an index-related positive integer variable of a frame, N is a positive integer, the frames of the video to be reconstructed are defined as odd frames, and therefore the frame in the video to be reconstructed is represented by an odd index 2t-1. The purpose of the embodiment is to recover a corresponding high-frame-rate and high-resolution video frame sequence based on recovery of a corresponding high-frame-rate and high-resolution video frame sequence N frames are added, and the added N frames are taken as even frames. First, the reconstructed video is transformed from an image domain to a feature domain. Specifically, the image frames of the video to be reconstructed are sent into a convolution conversion layer one by one to convert the image frames from the image domain to the feature domain, to obtain frame features corresponding to each image frame, and all the frame features constitute a frame feature sequence of the video to be reconstructed. Preferably, the convolution conversion layer is arranged in a video spatio-temporal super-resolution reconstruction model.

[0026] In step S2, the frame feature sequence of the video to be reconstructed is input into the trained video spatio-temporal super-resolution reconstruction model.

[0027] The video spatio-temporal super-resolution reconstruction model comprises:

[0028] An initial feature extraction module is configured to extract initial features of each frame of the video to be reconstructed, which are referred to as odd frame initial features. The initial feature extraction module is preferably but not limited to composed of a plurality of residual blocks cascaded, such as 5 cascaded residual blocks. The specific structure of the residual block can refer to Figure 5 , which comprises a first 3x3 convolution, a first ReLU activation function processing unit, a second 3x3 convolution, a residual connection unit and a second ReLU activation function processing unit connected in sequence. The residual connection unit is configured to connect the input features of the first 3x3 convolution and the output features of the second 3x3 convolution in a residual manner. Each frame feature of the video to be reconstructed in the feature domain is processed by the initial feature extraction module to obtain the initial features corresponding to each frame. The initial features of all original frames (odd frames) can be represented as

[0029] The bidirectional alignment module generates an initial feature of an intermediate frame to be interpolated based on adjacent odd-numbered frame initial features, and records the initial feature of the intermediate frame to be interpolated as an even-numbered frame initial feature. All the even-numbered frame initial features are represented as At this point, there are 2N+1 frame features, and the frame rate at the feature level is preliminarily improved, which prepares for reconstructing the high frame rate video. In specific applications, multiple bidirectional alignment modules can be set, and the multiple bidirectional alignment modules respectively generate initial features of intermediate frames to be interpolated for different adjacent odd-numbered frame initial features.

[0030] k1 spatio-temporal information-based feature extraction and fusion modules, the odd-numbered frame initial features and the even-numbered frame initial features are simultaneously input into the k1 spatio-temporal information-based feature extraction and fusion modules to obtain deep features corresponding to each frame. As shown in Figure 4 , the spatio-temporal information-based feature extraction and fusion module includes a spatio-temporal correction module and k2 cascaded residual blocks connected in sequence, so as to fully utilize the spatio-temporal correlation between frames. The residual blocks are densely connected, and preferably, the residual blocks adopt the residual block structure as shown in Figure 5 . k1 and k2 are both positive integers. Specifically, the number of processing channels of the spatio-temporal information-based feature extraction and fusion module can be set according to needs and hardware conditions, such as Figure 2 , the number of processing channels of the spatio-temporal information-based feature extraction and fusion module is 5, which can simultaneously process initial features of 5 frames. If 2N+1 is greater than 5, the number of processing channels of the spatio-temporal information-based feature extraction and fusion module can be set to 2N+1 in batches. In this embodiment, preferably, k2 is 5, and k1 is set to 8.

[0031] The reconstruction layer performs recovery processing on the deep features of all the frames to obtain a recovered video sequence.

[0032] Figure 2 The structure of the video spatio-temporal super-resolution reconstruction model in one application scenario in this embodiment is shown. In this application scenario, the video spatio-temporal super-resolution reconstruction model simultaneously reconstructs and recovers three continuous frame features of a video to be reconstructed at a time, and the indices of the three frame features are 1, 3, and 5, respectively. As shown in Figure 2 , the initial feature extraction module extracts initial features of frame 1, frame 3, and frame 5, respectively. Two bidirectional alignment modules are provided. One bidirectional alignment module generates an intermediate frame 2 of frame 1 and frame 3, and the other bidirectional alignment module generates an intermediate frame 4 of frame 3 and frame 5. In this way, initial features of 5 frames are obtained in total. The initial features of the 5 frames are input into k1 spatio-temporal information-based feature extraction and fusion modules to obtain deep features of each frame. The deep features of the 5 frames are input into a reconstruction layer to recover super-resolution images of the 5 frames.

[0033] In step S3, the video spatio-temporal super-resolution reconstruction model outputs a recovered video sequence.

[0034] In the embodiment, to fully utilize the space-time correlation between frames, preferably, as shown in Figure 6 The space-time correction module comprises:

[0035] A first correction concatenation unit is configured to perform concatenation processing on the feature data of all the frames input into the space-time correction module, and represent the concatenated feature data as wherein C represents the number of feature channels, T represents the total number of frames, H represents the length of the image, and W represents the height of the image.

[0036] A time correction block is configured to capture time correlation information between frames based on the concatenated feature data output by the first correction concatenation unit, and perform time correction to obtain overall time correction feature data.

[0037] A segmentation unit is configured to segment the overall time correction feature data along the time axis to obtain time correction feature data of each frame.

[0038] A space correction block is configured to perform space correction processing on the time correction feature data of all the frames to obtain space-time correction feature data of all the frames.

[0039] A residual connection unit is configured to perform residual processing on the feature data of all the frames input into the space-time correction module and the space-time correction feature data of all the frames output by the space correction block.

[0040] In the embodiment, further preferably, as shown in Figure 7 The time correction block comprises:

[0041] A global space average pooling unit is configured to aggregate the space information of the concatenated feature data output by the first correction concatenation unit, and assume that the feature data input into the global space average pooling unit is The space information aggregated by the global space average pooling unit is represented as After that, the importance map affected by the time position is learned using a local branch, and the time position invariant weight is used to aggregate the time information.

[0042] The local branch is connected with the global spatial average pooling unit and includes, in sequence, a first one-dimensional convolution, a first temporal rectified activation function unit, a second one-dimensional convolution, a second temporal rectified activation function unit, and a rescaling unit. The first temporal rectified activation function unit preferably but not limited to adopts a ReLU activation function, and the second temporal rectified activation function unit preferably but not limited to adopts a Sigmoid activation function. The one-dimensional convolution is a lightweight and sequence processing option compared with three-dimensional convolution and LSTM, which can greatly save the calculation amount and compress the parameter amount. The local branch uses one-dimensional convolution to preliminarily extract short-term relationships, and processes the features by using a batch normalization layer. This regularization strategy can also play an active role in model training, such as solving the gradient vanishing problem, and then a ReLU function is used to complete the nonlinear mapping of the result.

[0043] Let the output result after processing by the first one-dimensional convolution and the first temporal rectified activation function unit be: wherein δ is a ReLU function, C 1D (·) represents one-dimensional convolution, F local_channel_down represents features after channel attenuation. The second one-dimensional convolution is used to expand the channels to C, and a Sigmoid function is used to map the final importance weight according to its nonlinearity and gate mechanism M=F local_channel_up =σ(C 1D (F local_channel_down ));wherein σ is a Sigmoid function, F local_channel_up represents features after channel up-sampling. The plane weight mask M is expanded into a spatial kernel K in a copy manner to meet the premise of dimension matching, to prepare for the correction of the input:

[0044] The temporal correction multiplication unit performs an element-level multiplication operation on the feature data output by the rescaling unit and the spliced feature data output by the first correction splicing unit.

[0045] The global branch is connected with the global spatial average pooling unit and includes, in sequence, a first full connection layer, a third temporal rectified activation function unit, a second full connection layer, and a fourth temporal rectified activation function unit. The third temporal rectified activation function unit preferably but not limited to is a ReLU activation function, and the fourth temporal rectified activation function unit preferably but not limited to is a Sigmoid activation function. The global branch aggregates temporal information by using two consecutive full connection layers combined with corresponding activation functions, so as to learn an adaptive kernel by using long-term information. The first full connection layer and the third temporal rectified activation function unit obtain, after processing, the following: The second full connection layer and the fourth temporal rectified activation function unit obtain, after processing, the following: Fglobal = σ(C FC (F global_fc_1 )) where C FC (.) denotes the full connection operation.

[0046] The role of the global branch is to generate a temporal kernel according to the global context information. On the one hand, when processing different input video sequences, the global branch always generates a corresponding temporal kernel according to this process, so the temporal kernel has adaptivity. On the other hand, the adaptive kernel does not change with the change of the time position (time elapses), so it has position invariance. Given the high complexity of modeling both the channel and the time dimension, only the time relationship is modeled.

[0047] The temporal correction convolution processing unit is configured to perform convolution processing on the feature data output by the temporal correction multiplication unit and the feature data output by the global branch, specifically, channel-level convolution processing.

[0048] Preferably, the feature data output by the temporal correction convolution processing unit is further sequentially subjected to 3x3 convolution and 1x1 convolution processing.

[0049] In summary, the local branch and the global branch obtain local temporal dependency and global information respectively. In order to correct the features, first, the input features are corrected according to the local temporal dependency, that is, the local branch result K is multiplied with the input X at the element level: X local = K O X. The symbol O represents element-level multiplication operation, X local is the result after local correction. Then, X local is convolved with the learned temporal kernel in a channel-level manner to obtain the feature after temporal correction: The symbol represents convolution operation. Compared with ordinary convolution of aggregated spatio-temporal information, the temporal correction module improves performance in two aspects, one is that the weights of the global convolution kernel change adaptively with the change of the video, the other is that the information of the time dimension is dynamically learned according to the relationship within the video. The temporal correction module achieves good capture and utilization of time correlation.

[0050] In this embodiment, spatial correction is also needed before feature extraction to better utilize the information within each frame. Further preferably, as shown in FIG. 6, the spatial correction block includes: Figure 8

[0051] The spatial correction fusion unit acquires the fused features of all frames, and specifically performs:

[0052] For each odd frame, the temporal correction feature of the odd frame (such as X ​`i` represents a positive integer related to the frame number index. The frame number is concatenated with the temporal correction feature of the next adjacent even-numbered frame. The concatenated result is then processed through convolution to obtain the fusion feature of the odd-numbered frame. Preferably, a 1×1 convolution is used in the convolution process. Specifically, since there are no inserted frames (even-numbered frames) after the last frame (i.e., the last odd-numbered frame), it is concatenated with the previous adjacent inserted frame (even-numbered frame) and processed through convolution to obtain its corresponding fusion feature.

[0053] For each even-numbered frame, the time-corrected feature of the even-numbered frame is concatenated with the time-corrected feature of the previous adjacent odd-numbered frame. The concatenation result is processed by convolution (preferably 1×1 convolution), and the processing result is used as the offset input of the first modified deformable convolutional unit. The time-corrected feature of the previous adjacent odd-numbered frame of the even-numbered frame is used as the base input of the first modified deformable convolutional unit. The time-corrected feature of the even-numbered frame is concatenated with the time-corrected feature of the next adjacent odd-numbered frame. The concatenation result is processed by convolution (preferably 1×1 convolution), and the processing result is used as the offset input of the second modified deformable convolutional unit. The time-corrected feature of the next adjacent odd-numbered frame of the even-numbered frame is used as the base input of the second modified deformable convolutional unit. The time-corrected feature of the even-numbered frame, the output feature of the first modified deformable convolutional unit, and the output feature of the second modified deformable convolutional unit are concatenated. The concatenation result is processed by convolution (preferably 1×1 convolution) to obtain the fused feature of the even-numbered frame.

[0054] The spatial attention unit processes the fused features of each frame sequentially through two channel decaying convolutional layers (preferably including a first 3×3 convolution, a ReLU function unit, a second 3×3 convolution, and a Sigmoid function unit connected in sequence) to obtain an importance weight map of spatial location for each frame, resulting in a planar feature map of spatial location. At this point, the spatial importance of each feature is distinguished.

[0055] The spatial correction multiplication unit is used to perform element-wise multiplication of the temporal correction features of each frame with the importance weight map of that frame to obtain the spatiotemporal correction features of that frame.

[0056] In this embodiment, considering that reversing the video sequence can also provide information about the reverse motion process for reconstruction, using a bidirectional alignment process can make the reconstructed high frame rate video appear more natural in the time dimension. Therefore, more preferably, as shown in the example... Figure 3As shown, the bidirectional alignment module includes a forward alignment submodule, a backward alignment submodule, and a fusion submodule. The forward alignment submodule performs forward alignment on the initial features of adjacent odd-numbered frames to obtain forward-aligned features. The backward alignment submodule performs backward alignment on the initial features of adjacent odd-numbered frames to obtain backward-aligned features. The fusion submodule fuses the forward and backward alignment features to obtain the initial features of the intermediate frame to be interpolated between the adjacent odd-numbered frames, i.e., obtaining the initial features of the even-numbered frames located between the adjacent odd-numbered frames. The fusion submodule preferably includes, but is not limited to, a stitching unit, a 1×1 convolution, a ReLU activation function, and a 3×3 convolution connected in sequence. The stitching unit is used to stitch together the forward and backward alignment features. The bidirectional alignment module uses adjacent frame features... and As input, the forward alignment features and backward alignment features are obtained by processing the data through the modified forward alignment submodule and backward alignment submodule based on the DCN multi-branch cascaded feedback structure, respectively. The two features are then concatenated and convolved to output the alignment features.

[0057] In this embodiment, the forward alignment submodule includes:

[0058] The multi-level feature acquisition unit takes the initial features of adjacent odd-numbered frames of the input forward alignment submodule as first-level features. The initial size of the paired input adjacent odd-numbered frames is C×H×W. The first-level features are processed by a 2-stride convolution to obtain second-level features. The size of the second-level features is H / 2×W / 2. The stride convolution can encode spatial relationships, thereby implicitly preserving spatial information. In order to preserve information as much as possible, the first-level features are sampled with an 8×8 convolution kernel with a 4-stride sampling to obtain the first sampling result. The second-level features are sampled with a smaller 4×4 convolution kernel with a 2-stride sampling to obtain the second sampling result. The first sampling result and the second sampling result are fused by convolution (preferably a 1×1 convolution) to obtain the third-level features.

[0059] The three-branch processing branch includes a third aligned deformable convolutional unit. The third-level features are processed through convolution (preferably 1×1 convolution) and activation functions (preferably ReLU activation function) to obtain the offset input of the third aligned deformable convolutional unit. The input uses the earlier feature term from the third-level features (such as...) ) serves as the basic input for the third aligned deformable convolutional unit;

[0060] The two-branch processing branch includes a second aligned deformable convolutional unit and a two-branch concatenation unit. Secondary features are processed through convolution (preferably 3×3 convolution) and activation functions (preferably ReLU activation function) to obtain the two-branch convolution processing result. The offset input of the third aligned deformable convolutional unit undergoes first alignment upsampling processing. The two-branch concatenation unit concatenates the first alignment upsampling result with the two-branch convolution processing result, and performs convolution processing (preferably 1×1 convolution) on the concatenated result to obtain the offset input of the second aligned deformable convolutional unit. The offset input is then obtained using the earlier feature term in the secondary features (such as...). ) serves as the basic input for the second aligned deformable convolutional unit;

[0061] One branch processing branch includes a first aligned deformable convolutional unit, a receptive field expansion module, and a first branch concatenation unit. It processes the first-level features through convolution (preferably 3×3 convolution) and an activation function (preferably ReLU activation function) to obtain the first-branch convolution processing result. It then performs second-alignment upsampling processing on the offset input of the second aligned deformable convolutional unit. The first-branch concatenation unit concatenates the second-alignment upsampling processing result with the first-branch convolution processing result. The concatenated result is input (preferably after 1×1 convolution processing) to the receptive field expansion module to obtain the offset input of the first aligned deformable convolutional unit, using the earlier feature terms in the first-level features (such as...) ) serves as the basic input for the first aligned deformable convolutional unit;

[0062] The alignment and splicing fusion module performs a first fusion upsampling on the output features of the third alignment deformable convolutional unit, splices the first fusion upsampling result with the output features of the second alignment deformable convolutional unit to obtain a first fusion splicing result, performs a second fusion upsampling on the first fusion splicing result, splices the second fusion upsampling result with the output features of the first alignment deformable convolutional unit to obtain a second fusion splicing result, and performs convolution processing (preferably 3×3 convolution) on the second fusion splicing result to obtain the forward alignment features of adjacent odd-numbered frames.

[0063] In this embodiment, the backward alignment submodule has the same structure as the forward alignment submodule, except that the basic inputs of the first, second, and third aligned deformable convolutional units in the processing flow become later feature terms, such as... The backward alignment submodule and the forward alignment submodule effectively utilize local information and global temporal information within the frame, thereby improving the reconstruction quality.

[0064] The first, second, and third aligned deformable convolutional units all employ a deformable convolutional network (DCN), which can simultaneously align and aggregate temporal features to better utilize global contextual information. The computation process of DCN is as follows: Figure 10 As shown, it was proposed because, among all geometric invariants, CNNs only possess a certain degree of translation invariance, making them unable to effectively learn information such as rotation, scaling, and distortion. This process first utilizes a regular mesh with center point p0. Sample the input X. Where the coordinates of p0 are (0, 0), the offset Δp is learned using additional ordinary convolution. n The position of the convolution kernel will then change according to this offset. The specific deformable convolution is represented by the following formula:

[0065]

[0066] Through this process, deformable convolution can learn information from the input using convolutions of arbitrary shapes, rather than regular convolutions. Here, DCN, as a plug-and-play lightweight module, can adaptively learn offsets from the fused features of adjacent frames, and then apply these offsets to the input based on whether forward or backward alignment is used. or The deformed convolution is completed on top. Forward alignment can be described by the formula:

[0067]

[0068] Similarly, backward alignment inputs features from adjacent frames. and The concatenation order is reversed and used as input. The processing network structure is consistent with the forward alignment; backward alignment can be represented as:

[0069]

[0070] Finally, the bidirectional alignment features (forward alignment) are aligned using a convolution operation. Back Alignment To merge:

[0071]

[0072] Where C1 and C3 represent 1×1 and 3×3 convolutions respectively, and δ is the ReLU function.

[0073] In this embodiment, as Figure 9As shown, the expanded receptive field module includes: a first expansion branch, which performs a first expansion convolution on the features input to the expanded receptive field module, preferably through a 1×1 convolution; a second expansion branch, which performs a second expansion convolution on the features input to the expanded receptive field module, preferably through a 3×3 convolution; a third expansion branch, which sequentially performs a third expansion convolution (preferably through a 3×3 convolution), an expansion activation function (preferably a ReLU activation function), and a fourth expansion convolution (preferably through a 3×3 convolution) on the features input to the expanded receptive field module; an expansion splicing unit, which splices the output features of the first expansion branch, the second expansion branch, and the third expansion branch; and a fifth expansion convolutional layer (preferably through a 1×1 convolution), which performs a fifth expansion convolution on the features output by the expansion splicing unit.

[0074] Experimental verification was performed on the video spatiotemporal super-resolution reconstruction model provided in this embodiment:

[0075] Experiment 1: Validation of the bidirectional alignment module

[0076] The bidirectional alignment network, as a tool for frame feature interpolation, not only provides front-to-back fusion information for the frame to be inserted, but also utilizes the back-to-front reverse motion process to provide supplementary information, thus making the changes in the video frame sequence more natural. The effectiveness of the bidirectional alignment network was verified through ablation experiments. The experimental results are shown in Table 1. The model was tested on the Vid4 (slow-motion video sequence) and Vimeo-Fast test sets using PSNR (dB) / SSIM as the evaluation metric. On Vid4, compared to Experiment 1 (without alignment) and Experiment 2 (only forward alignment), Experiment 3 (bidirectional alignment) improved the PSNR value by 0.62dB and 0.35dB, respectively. On the Vimeo-Fast test set, Experiment 3 differed from Experiment 2 by 1.06dB, while widening the gap with Experiment 1 to 1.74dB. It can be seen that compared to frame interpolation without alignment, Experiment 2 demonstrates that forward alignment alone can effectively learn the spatiotemporal correlation of features. Further comparison of bidirectional alignment networks and single forward alignment processes confirms that bidirectional alignment can indeed utilize "reverse video" and obtain supplementary information from it.

[0077] Table 1 evaluates the effectiveness of the bidirectional alignment network on Vid4 and Vimeo-Fast using PSNR (dB) / SSIM.

[0078]

[0079] Experiment 2: Validation of the spatiotemporal correction module

[0080] The spatial correction scheme mainly consists of two parts: temporal attention and spatial attention. Its function is to fully extract the spatiotemporal information implicit in the frame sequence. The benchmark experiment without the correction scheme replaces the spatiotemporal correction module with a series of ordinary convolutions. In addition, it includes temporal correction, spatial correction, and spatiotemporal correction. Similarly, PSNR (dB) / SSIM is used as the evaluation metric on Vid4 and Vimeo-Fast, and the experimental results are shown in Table 2. According to the data in Table 2, on the Vid4 dataset, compared with Experiment 1, Experiment 2 improved by 0.67dB, Experiment 3 improved by 0.49dB, while Experiment 4 increased the gap to 1.14dB. Analysis shows that since slow motion largely leaves the foreground and background visible, temporal correction promotes feature learning and representation by providing supplementary information from adjacent frames, while spatial correction enhances the reconstruction of a single frame image through spatial information. Combining the two effectively corrects features based on adjacent frames, thereby improving the reconstruction quality. On Vimeo-Fast, Experiment 2 showed a 0.71 dB improvement over Experiment 1, Experiment 3 a 0.56 dB improvement, and Experiment 4 a 1.32 dB improvement. It can be seen that spatiotemporal correction significantly improves fast motion compared to slow motion. This means that the module can effectively utilize temporal context information and spatial location importance to process large movements, indicating that the spatiotemporal information-based feature extraction and fusion module plays a positive role in reconstruction.

[0081] Table 4.4 Evaluation of the effectiveness of the spatiotemporal correction module on Vid4 and Vimeo-Fast using PSNR (dB) / SSIM.

[0082]

[0083] Experiment 3: Validation of the effectiveness of deformable convolution

[0084] Deformable convolutions, used as alignment strategies, allow the model to learn information such as rotation and distortion. To illustrate their impact on reconstruction performance, ablation experiments were designed. The quantitative evaluation results of PSNR (dB) / SSIM on Vid4 and Vimeo-Fast are shown in Table 3. To verify the role of deformable convolutions in the model, Experiment 1 replaced the DCN in the bidirectional alignment module with the corresponding ordinary convolution. Features of adjacent frames at the same level were fused only by convolution. Furthermore, the fusion process of inserted frame features in the spatiotemporal information-based feature extraction and fusion module was simplified to a process similar to that of odd-numbered frame features, i.e., merging the left and right adjacent frames and then performing ordinary convolution.

[0085] Table 3 verifies the impact of deformable convolution on reconstruction performance on Vid4 and Vimeo-Fast using PSNR (dB) / SSIM as the metric.

[0086]

[0087] As can be seen from the quantitative evaluation in Table 3, the application of deformable convolution significantly improves reconstruction performance for both slow and fast motion. The PSNR gain reaches 1.07 dB on Vid4 and 1.29 dB on Vimeo-Fast. Based on the functionality of deformable convolution, the model trained in Experiment 2 can be aligned using the DCN process to simulate inter-frame motion; while the fusion operation of ordinary convolution, although it correlates information in adjacent frames, this correlation is limited to translation and cannot learn other transformations.

[0088] Experiment 4: Comparison with Advanced Methods

[0089] Currently, spatiotemporal super-resolution reconstruction (SSR) tasks for video sequences are mainly divided into two categories: one is a two-stage approach that performs VFI first and then completes VSR, and the other is to generate only feature-level intermediate frames without explicitly inserting pixel frames. Representative methods from these two categories are compared on Vid4 and Vimeo tests. VFI methods include SepConv, Super-SloMo, and DAIN; VSR models include BI, RCAN, RBPN, EDVR, and the multi-branch multi-resolution cross-projection model (MBMR); and complete one-stage methods include STARnet, Zooming Slow-Mo, and TMNet. Quantization results using PSNR (dB) / SSIM as the evaluation metric are shown in Table 4.

[0090] According to Table 4, the best performing two-stage method is the combination of DAIN+EDVR, which performs best on all datasets, but it still lags behind the performance of the one-stage method. Among the one-stage methods, the video spatiotemporal super-resolution reconstruction algorithm based on bidirectional alignment and spatiotemporal correction proposed in this application shows relatively stable performance, ranking first or second in scores on different test sets. Specifically, on the Vid4, Vimeo-Fast, and Vimeo-Med datasets, this model achieves the best results, improving the PSNR score by 0.05dB, 0.21dB, and 0.09dB respectively compared to the second-best model, TMNet. Although the PSNR score on Vimeo-Slow is slightly lower than TMNet, the SSIM still reflects that the image structure is well restored. Analysis of the dataset characteristics shows that, on slow and medium-speed motion datasets, bidirectional alignment can fully utilize the foreground and background information provided by adjacent frames. Since the motion is not intense, spatiotemporal information can be used for calibration. It is worth noting that the bidirectional alignment and spatiotemporal correction model of this application has outstanding performance on the Vimeo-Fast dataset, meaning that its advantages are particularly evident in the handling of intense motion scenes. Although the differences between frames are large at this time, the spatiotemporal correction of features can enhance their expressive power, which is of great significance for improving motion blur.

[0091] Table 4 compares the proposed reconstruction model with several advanced algorithms on the Vid4 and Vimeo test sets, using PSNR (dB) / SSIM as the evaluation metric.

[0092]

[0093]

[0094] The video spatiotemporal super-resolution reconstruction model provided in this embodiment uses a spatiotemporal information-based feature extraction and fusion module to continuously correct and calibrate depth features using temporal and spatial information, thereby synthesizing more accurate structural information. This will improve motion blur and artifacts to a certain extent, and it has good adaptability even when dealing with fast motion. The reconstruction results of the method proposed in this application also have better visual performance compared with the reconstruction results of other one-stage methods. It can adapt well to all scenarios.

[0095] Example 2

[0096] This embodiment provides an apparatus based on the video spatiotemporal super-resolution reconstruction method provided in Embodiment 1, comprising: an acquisition module for acquiring the frame feature sequence of the video to be reconstructed in the feature domain; and a reconstruction module equipped with a trained video spatiotemporal super-resolution reconstruction model, which inputs the frame feature sequence of the video to be reconstructed into the trained video spatiotemporal super-resolution reconstruction model to obtain the recovered video sequence.

[0097] Example 3

[0098] This embodiment provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the video spatiotemporal super-resolution reconstruction method provided in Embodiment 1.

[0099] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0100] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A video spatiotemporal super-resolution reconstruction method, characterized in that, include: Obtain the frame feature sequence of the video to be reconstructed in the feature domain, input the frame feature sequence of the video to be reconstructed into a trained video spatiotemporal super-resolution reconstruction model, and the video spatiotemporal super-resolution reconstruction model outputs a recovered video sequence. The video spatiotemporal super-resolution reconstruction model includes: The initial feature extraction module is used to extract the initial features of each frame of the video to be reconstructed, denoted as the initial features of odd-numbered frames; The bidirectional alignment module generates the initial features of the intermediate frame to be interpolated based on the initial features of the adjacent odd-numbered frames, and records the initial features of the intermediate frame to be interpolated as the initial features of the even-numbered frames. A cascaded feature extraction and fusion module based on spatiotemporal information, with initial features from odd-numbered frames and even-numbered frames input simultaneously. A series of cascaded spatiotemporal information-based feature extraction and fusion modules obtain deep features corresponding to each frame. The spatiotemporal information-based feature extraction and fusion modules include a spatiotemporal correction module and... A series of cascaded residual blocks; and All are positive integers; The reconstruction layer performs deep feature recovery processing on all frames to obtain the recovered video sequence; wherein, the spatiotemporal correction module includes: The first correction and splicing unit is used to splice the feature data of all frames input to the spatiotemporal correction module. The time correction block captures the time-related information between frames based on the splicing feature data output by the first correction splicing unit to perform time correction and obtain the overall time correction feature; The segmentation unit divides the overall time correction feature along the time axis to obtain the time correction feature of each frame; Spatial correction block: Spatial correction is applied to the temporal correction features of all frames to obtain the spatiotemporal correction features of all frames. The residual connection unit performs residual processing on the feature data of all frames input to the spatiotemporal correction module and the spatiotemporal correction features of all frames output by the spatial correction block.

2. The video spatiotemporal super-resolution reconstruction method as described in claim 1, characterized in that, The time correction block includes: The global spatial average pooling unit is used to aggregate the spatial information of the splicing feature data output by the first corrected splicing unit; The local branch is connected to the global spatial average pooling unit, and includes a first one-dimensional convolution, a first time-corrected activation function unit, a second one-dimensional convolution, a second time-corrected activation function unit, and a rescaling unit connected in sequence. The time-corrected multiplication unit performs element-wise multiplication on the feature data output by the rescaling unit and the concatenated feature data output by the first correction concatenation unit. The global branch, connected to the global spatial average pooling unit, includes a first fully connected layer, a third time-corrected activation function unit, a second fully connected layer, and a fourth time-corrected activation function unit connected in sequence. The temporally corrected convolution processing unit is used to perform convolution processing on the feature data output by the temporally corrected multiplication unit and the feature data output by the global branch.

3. The video spatiotemporal super-resolution reconstruction method as described in claim 1 or 2, characterized in that, Spatial correction blocks include: The spatial correction fusion unit acquires the fusion features of all frames and performs the following actions: For each odd frame, the temporal correction feature of the odd frame is concatenated with the temporal correction feature of the next adjacent even frame, and the concatenation result is processed by convolution to obtain the fused feature of the odd frame. For each even-numbered frame, the temporal correction feature of the even-numbered frame is concatenated with the temporal correction feature of the previous adjacent odd-numbered frame. The concatenated result is processed by convolution, and the result is used as the offset input of the first modified deformable convolutional unit. The temporal correction feature of the previous adjacent odd-numbered frame is used as the base input of the first modified deformable convolutional unit. The temporal correction feature of the even-numbered frame is concatenated with the temporal correction feature of the next adjacent odd-numbered frame. The concatenated result is processed by convolution, and the result is used as the offset input of the second modified deformable convolutional unit. The temporal correction feature of the next adjacent odd-numbered frame is used as the base input of the second modified deformable convolutional unit. The temporal correction feature of the even-numbered frame, the output feature of the first modified deformable convolutional unit, and the output feature of the second modified deformable convolutional unit are concatenated. The concatenated result is processed by convolution to obtain the fused feature of the even-numbered frame. The spatial attention unit processes the fused features of each frame sequentially through two channels of decaying convolutional layers to obtain the importance weight map of each frame with respect to spatial location. The spatial correction multiplication unit is used to perform element-wise multiplication of the temporal correction features of each frame with the importance weight map of that frame to obtain the spatiotemporal correction features of that frame.

4. The video spatiotemporal super-resolution reconstruction method as described in claim 1 or 2, characterized in that, The bidirectional alignment module includes a forward alignment submodule, a backward alignment submodule, and a fusion submodule. The forward alignment submodule performs forward alignment on the initial features of adjacent odd-numbered frames to obtain forward-aligned features. The backward alignment submodule performs backward alignment on the initial features of adjacent odd-numbered frames to obtain backward-aligned features. The fusion submodule merges the forward alignment features and the backward alignment features to obtain the initial features of the intermediate frame to be interpolated between the adjacent odd-numbered frames, that is, to obtain the initial features of the even-numbered frames located between the adjacent odd-numbered frames.

5. The video spatiotemporal super-resolution reconstruction method as described in claim 4, characterized in that, The forward alignment submodule includes: The multi-level feature acquisition unit takes the initial features of adjacent odd-numbered frames of the input forward alignment submodule as first-level features; processes the first-level features using a 2-step convolution to obtain second-level features; performs 4-step sampling on the first-level features to obtain the first sampling result; performs 2-step sampling on the second-level features to obtain the second sampling result; and fuses the first sampling result and the second sampling result through convolution to obtain the third-level features. The three-branch processing branch includes the third aligned deformable convolutional unit. The three-level features are processed by convolution and activation functions to obtain the offset input of the third aligned deformable convolutional unit. The feature terms at the beginning of the three-level features are used as the basic input of the third aligned deformable convolutional unit. The two-branch processing branch includes a second aligned deformable convolutional unit and a two-branch concatenation unit. The secondary features are processed by convolution and activation functions to obtain the two-branch convolution processing result. The offset input of the third aligned deformable convolutional unit is processed by the first aligned upsampling processing result. The two-branch concatenation unit concatenates the first aligned upsampling processing result with the two-branch convolution processing result. The concatenation result is processed by convolution to obtain the offset input of the second aligned deformable convolutional unit. The feature term at the beginning of the secondary features is used as the basic input of the second aligned deformable convolutional unit. The first branch processing branch includes a first aligned deformable convolutional unit, an expanded receptive field module, and a first branch concatenation unit. It processes the first-level features through convolution and activation functions to obtain the first branch convolution processing result. It performs second alignment upsampling processing on the offset input of the second aligned deformable convolutional unit. The first branch concatenation unit concatenates the second alignment upsampling processing result with the first branch convolution processing result. The concatenation result is input into the expanded receptive field module to obtain the offset input of the first aligned deformable convolutional unit. The first feature term in the first-level features is used as the basic input of the first aligned deformable convolutional unit. The alignment and fusion module performs a first fusion upsampling on the output features of the third alignment deformable convolutional unit, concatenates the first fusion upsampling result with the output features of the second alignment deformable convolutional unit to obtain a first fusion concatenation result, performs a second fusion upsampling on the first fusion concatenation result, concatenates the second fusion upsampling result with the output features of the first alignment deformable convolutional unit to obtain a second fusion concatenation result, and performs convolution processing on the second fusion concatenation result to obtain the forward alignment features of adjacent odd-numbered frames.

6. The video spatiotemporal super-resolution reconstruction method as described in claim 5, characterized in that, The expanded receptive field module includes: The first expansion branch performs a first expansion convolution processing on the features of the input expanded receptive field module; The second expansion branch performs a second expansion convolution on the features of the input expanded receptive field module; The third expansion branch sequentially processes the features of the input expanded receptive field module by the third expansion convolution, the expanded activation function, and the fourth expansion convolution. Expand the splicing unit to splice the output features of the first, second, and third expanded branches; The fifth enlarged convolutional layer performs fifth enlarged convolution processing on the features output by the enlarged splicing unit.

7. The video spatiotemporal super-resolution reconstruction method as described in claim 3, characterized in that, The bidirectional alignment module includes a forward alignment submodule, a backward alignment submodule, and a fusion submodule. The forward alignment submodule performs forward alignment on the initial features of adjacent odd-numbered frames to obtain forward-aligned features. The backward alignment submodule performs backward alignment on the initial features of adjacent odd-numbered frames to obtain backward-aligned features. The fusion submodule merges the forward alignment features and the backward alignment features to obtain the initial features of the intermediate frame to be interpolated between the adjacent odd-numbered frames, that is, to obtain the initial features of the even-numbered frames located between the adjacent odd-numbered frames.

8. An apparatus based on the video spatiotemporal super-resolution reconstruction method according to any one of claims 1-7, characterized in that, include: The acquisition module is used to acquire the frame feature sequence of the video to be reconstructed in the feature domain; The reconstruction module is equipped with a pre-trained video spatiotemporal super-resolution reconstruction model. The frame feature sequence of the video to be reconstructed is input into the pre-trained video spatiotemporal super-resolution reconstruction model to obtain the recovered video sequence.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the video spatiotemporal super-resolution reconstruction method as described in any one of claims 1 to 7.

Images