Video space-time super-resolution method and device

Abstract

The invention discloses a video space-time super-resolution method and device. The method comprises the following steps: receiving a video to be processed; a preset model is adopted to process complete sequence image frames corresponding to the to-be-processed video, fusion features of the complete sequence image frames, motion features of the complete sequence image frames and content features of the complete sequence image frames are obtained, the fusion features of the complete sequence image frames are at least used for representing time sequence information of each image frame in the complete sequence image frames, and the motion features of the complete sequence image frames are at least used for representing time sequence information of each image frame in the complete sequence image frames; the action features of the complete-sequence image frames are at least used for representing the motion information of each image frame in the complete-sequence image frames, and the content features of the complete-sequence image frames are at least used for representing the content information of each image frame in the complete-sequence image frames; and determining a target video according to the fusion feature of the complete sequence image frame, the action feature of the complete sequence image frame and the content feature of the complete sequence image frame, the target video being a video after super-resolution reconstruction corresponding to the to-be-processed video.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and more specifically, to a video spatiotemporal super-resolution method and apparatus. Background Technology

[0002] Currently, in related technologies, the only methods for performing spatiotemporal super-resolution reconstruction on videos are: 1. Enlarging each frame of the video and then interpolating to generate an intermediate frame of the enlarged size; 2. Generating intermediate frames from the video first, then enlarging both the original video frames and the generated intermediate frames, which is spatial super-resolution; 3. Performing spatiotemporal super-resolution on the video simultaneously using a network model, but this model essentially still performs super-resolution on the image first and then interpolates frames, or vice versa, and is not a true one-stage process. The two-stage method leads to the accumulation of errors, affecting the final generation effect. The shortcomings of the two-stage method are that different networks have different features for image and video extraction, which means that the feature maps generated in the first stage cannot be reused in the second stage. This wastes resources, but more importantly, it results in insufficient feature extraction depth and limited use of features, leading to a large deviation between the overall effect and the local effect. Furthermore, the errors in the first stage are treated as natural input samples, lacking error correction capabilities. This causes new content to be generated on the erroneous intermediate results, resulting in the accumulation of erroneous information. As a result, the final generation result is accompanied by serious erroneous calculation information, causing the video to have a lot of artifacts and be blurry. Summary of the Invention

[0003] This application provides a video spatiotemporal super-resolution method and apparatus to at least solve the technical problem that related technologies use a two-stage approach of amplification and frame interpolation to reconstruct video spatiotemporal super-resolution, resulting in low clarity of the reconstructed video.

[0004] According to one aspect of the embodiments of this application, a video spatiotemporal super-resolution method is provided, comprising: receiving a video to be processed; processing the full sequence of image frames corresponding to the video to be processed using a preset model to obtain fusion features, motion features, and content features of the full sequence of image frames, wherein the fusion features of the full sequence of image frames are used at least to characterize the temporal information of each image frame in the full sequence of image frames, the motion features of the full sequence of image frames are used at least to represent the motion information of each image frame in the full sequence of image frames, and the content features of the full sequence of image frames are used at least to characterize the content information of each image frame in the full sequence of image frames; determining a target video based on the fusion features, motion features, and content features of the full sequence of image frames, wherein the target video is the super-resolution reconstructed video corresponding to the video to be processed.

[0005] Optionally, the fusion features are determined as follows: a preset model is used to process each image frame in the entire sequence of image frames using a bidirectional background detection algorithm based on Gaussian mixture distribution to obtain forward motion foreground information and backward motion foreground information corresponding to each image frame; the forward motion foreground information and backward motion foreground information corresponding to each image frame are binarized to obtain the time step of each image frame; each image frame and the time step of each image frame are combined to obtain the time step information corresponding to each image frame; the time step information corresponding to each image frame is processed to obtain the fusion features of the entire sequence of image frames.

[0006] Optionally, processing the time step information corresponding to each image frame to obtain the fusion features of the entire sequence of image frames includes: extracting features from the time step information corresponding to each image frame to obtain a multi-layer feature map for each image frame, wherein the size of the multi-layer feature map decreases sequentially; determining the offset of each feature map based on the offset of the feature map of the previous layer; aligning the multi-layer feature map with the multi-layer feature maps of adjacent image frames of each image frame according to the offset of each feature map to obtain aligned features; and determining the fusion features of the entire sequence of image frames based on the aligned features.

[0007] Optionally, determining the fused features based on the aligned features includes: determining the aligned features and the bidirectional mixed features as target features; dividing the target features into past feature states, present feature states, and future feature states; inputting the past feature states, present feature states, and future feature states into three channels respectively to obtain three feature maps, and concatenating the three feature maps to obtain a concatenated feature map; and determining the fused features of the entire sequence of image frames based on the concatenated feature map.

[0008] Optionally, the method further includes: extracting a forward optical flow map and a reverse optical flow map from the time step information corresponding to each image frame using an optical flow model in a preset model; extracting features from the forward optical flow map and the reverse optical flow map respectively to obtain forward features and reverse features; and fusing the forward features and the reverse features to obtain bidirectional hybrid features.

[0009] Optionally, the content features are determined as follows: any patch in the current image frame is selected as a query vector, and the patches at corresponding positions in the adjacent image frames before and after the patch corresponding to the query vector are respectively determined as key vectors and value vectors; an attention weight matrix is ​​determined based on the query vector and the key vectors, wherein the attention weight matrix is ​​used to represent the correlation between patches in the previous image frame and patches in adjacent image frames; the attention weight matrix is ​​used to weight the patches in the adjacent image frames to obtain the content features of the patches in the current image frame.

[0010] Optionally, the action features are determined by: assigning position codes to patches in each image frame; and weighting the position codes of each patch using the attention weight matrix to obtain the action features of the entire sequence of image frames.

[0011] Optionally, determining the target video based on the fusion features, motion features, and content features of the entire sequence of image frames includes: determining the residual of the target video based on the fusion features, motion features, and content features of the entire sequence of image frames; and generating the target video based on the residual of the target video.

[0012] According to another aspect of the embodiments of this application, a video spatiotemporal super-resolution apparatus is also provided, comprising: a receiving module for receiving a video to be processed; a processing module for processing the full sequence of image frames corresponding to the video to be processed using a preset model to obtain fusion features, motion features, and content features of the full sequence of image frames, wherein the fusion features of the full sequence of image frames are used at least to characterize the temporal information of each image frame in the full sequence of image frames, the motion features of the full sequence of image frames are used at least to represent the motion information of each image frame in the full sequence of image frames, and the content features of the full sequence of image frames are used at least to characterize the content information of each image frame in the full sequence of image frames; and a determining module for determining a target video based on the fusion features, motion features, and content features of the full sequence of image frames, wherein the target video is the super-resolution reconstructed video corresponding to the video to be processed.

[0013] According to another aspect of the embodiments of this application, a computer device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described video spatiotemporal super-resolution method.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described video spatiotemporal super-resolution method.

[0015] In this embodiment, a video to be processed is received; a preset model is used to process the full sequence of image frames corresponding to the video to be processed, to obtain the fusion features, motion features, and content features of the full sequence of image frames. The fusion features are used to characterize the temporal information of each image frame in the full sequence of image frames; the motion features are used to represent the motion information of each image frame in the full sequence of image frames; and the content features are used to characterize the content information of each image frame in the full sequence of image frames. Based on the fusion of the full sequence of image frames... The target video is determined by the features, motion features, and content features of the entire sequence of image frames. The target video is the super-resolution reconstructed video corresponding to the video to be processed. The video to be processed is processed by a preset model to obtain multiple features of the video to be processed. Based on the multiple features of the video to be processed, the reconstructed target video is determined. This achieves the goal of avoiding the reconstruction of the video through a two-stage reconstruction method, thereby improving the technical effect of the reconstructed video clarity. This solves the technical problem that related technologies use a two-stage method of magnification and frame interpolation to perform spatiotemporal super-resolution reconstruction of video, resulting in low clarity of the reconstructed video. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a video spatiotemporal super-resolution method according to an embodiment of this application;

[0018] Figure 2 This is a flowchart of a video spatiotemporal super-resolution method according to an embodiment of this application;

[0019] Figure 3 This is a framework diagram of a video spatiotemporal super-resolution method according to an embodiment of this application;

[0020] Figure 4 This is a flowchart of a loop propagation method according to an embodiment of this application;

[0021] Figure 5 This is a flowchart illustrating the processing of content information and action information according to an embodiment of this application;

[0022] Figure 6 This is a flowchart illustrating the extraction of content features and action features according to an embodiment of this application;

[0023] Figure 7 This is a flowchart of a spatiotemporal super-resolution decoding according to an embodiment of this application;

[0024] Figure 8 This is a structural diagram of a video spatiotemporal super-resolution device according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0028] To better understand the embodiments of this application, the technical terms used in the embodiments of this application are explained as follows:

[0029] Resolution: A video consists of a series of images, each of which is sequential in time. The number of pixels it has is called the resolution, which is usually length by width, such as 1920×1080.

[0030] Video frame interpolation: Each image in a video is called a frame, and the number of frames played per second is called the frame rate. Video frame interpolation aims to increase the frame rate by using existing video frames to generate intermediate frames between each pair of frames, supplementing the details in the video playback. Generating frames in time also generates resolution in time, hence it is also called temporal super-resolution.

[0031] Video super-resolution: For an image, the process of super-resolution reconstruction is the process of increasing its resolution. This results in more pixels and a larger image size, achieving spatial super-resolution, or simply super-resolution. Performing super-resolution reconstruction on each frame of a video is called video (spatial) super-resolution reconstruction.

[0032] Attention: The attention mechanism originates from the human cognitive system, allowing people to filter and focus on specific information from various sensory inputs. In deep learning, the attention mechanism mimics this ability, reinforcing specific data features and adding weights to all information, thus making full use of areas with high attention weights.

[0033] Deep learning: Deep learning is currently the mainstream method for artificial intelligence models. It involves building deep neural networks to continuously analyze and mine the features of the input, and to perform computational processing on the deep features. This process uses convolutional neural networks, attention mechanisms, Transformer networks, and other technologies.

[0034] Transformer is a specific structural model based on deep neural networks. This model possesses unique encoding and decoding mechanisms, enabling efficient data processing and transformation. During data processing, it employs methods such as multi-head self-attention to encode inputs into Q, K, and V vectors, calculate the relationships between inputs and their own internal relationships, and perform multi-dimensional analysis and integration of the data to achieve accurate prediction and classification tasks.

[0035] Spatiotemporal sliding window: Due to memory limitations, video processing cannot handle the entire video simultaneously. Therefore, a fixed number of elements are processed sequentially. This sliding processing of content is called a sliding window, or simply a sliding window. A sliding window on a temporal sequence, such as sliding a fixed number of frames or images at the same location at different times, is called a temporal sliding window. A sliding window on a spatial sequence, such as decomposing a set of images and sliding the resulting blocks, is called a spatial sliding window. Simultaneous sliding in both time and space constitutes a spatiotemporal sliding window.

[0036] Features: In computer vision, after an image is input into a deep neural network, it undergoes operations such as convolution or attention calculations. For example, the dimensions of an image might be (3, 1, 256, 256), where 1 represents the number of images (in this case, one image), 3 represents the number of channels (typically red, green, and blue channels, meaning it's a superposition of three images), and the last two 256 represent the height and width of the image, indicating that each channel is a 256x256 matrix. A batch of images after feature extraction is also typically represented as (C, T, HW), where C represents the number of channels, T represents the number of frames, T represents the height, and T represents the width.

[0037] To address the problems existing in related technologies, this application provides a video spatiotemporal super-resolution method, which can be implemented in... Figure 1 The computer terminal shown is explained below.

[0038] The video spatiotemporal super-resolution method embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal for implementing a video spatiotemporal super-resolution method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0039] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0040] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video spatiotemporal super-resolution method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned video spatiotemporal super-resolution method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0041] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0042] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0043] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0044] In the above operating environment, this application provides an embodiment of a video spatiotemporal super-resolution method. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0045] Figure 2 This is a flowchart of a video spatiotemporal super-resolution method according to an embodiment of this application, such as...Figure 2 As shown, the method includes the following steps:

[0046] Step S202: Receive the video to be processed;

[0047] Step S204: Process the full sequence image frames corresponding to the video to be processed using a preset model to obtain the fusion features, motion features, and content features of the full sequence image frames. The fusion features of the full sequence image frames are used to characterize the temporal information of each image frame in the full sequence image frames. The motion features of the full sequence image frames are used to represent the motion information of each image frame in the full sequence image frames. The content features of the full sequence image frames are used to characterize the content information of each image frame in the full sequence image frames.

[0048] Step S206: Determine the target video based on the fusion features of the full sequence image frames, the action features of the full sequence image frames, and the content features of the full sequence image frames, wherein the target video is the super-resolution reconstructed video corresponding to the video to be processed.

[0049] Through steps S202 to S206 above, the process involves receiving a video to be processed; processing the full sequence image frames corresponding to the video to be processed using a preset model to obtain the fusion features, motion features, and content features of the full sequence image frames. The fusion features of the full sequence image frames are used to characterize at least the temporal information of each image frame in the full sequence image frames; the motion features of the full sequence image frames are used to represent at least the motion information of each image frame in the full sequence image frames; and the content features of the full sequence image frames are used to characterize at least the content information of each image frame in the full sequence image frames. Based on the full sequence image... The target video is determined by the fusion features of the images, the motion features of the entire sequence of images, and the content features of the entire sequence of images. The target video is the super-resolution reconstructed video corresponding to the video to be processed. A preset model is used to process the video to be processed, obtaining various features of the video to be processed. Based on these features, the reconstructed target video is determined. This avoids the need for a two-stage reconstruction method, thereby improving the clarity of the reconstructed video. This solves the technical problem of low clarity in reconstructed videos due to the two-stage method of upscaling and frame interpolation used in related technologies for spatiotemporal super-resolution video reconstruction. A detailed explanation follows.

[0050] To better implement the video spatiotemporal super-resolution method proposed in the embodiments of this application, a framework for implementing the video spatiotemporal super-resolution method is also proposed in the embodiments of this application, such as... Figure 3As shown, the video input module is used to extract the background subtraction mask from the input and combine the time step input with prior knowledge; the feature extraction and alignment module is mainly used to extract multi-level features from the time step and align the feature information through pyramid variability convolution; the loop propagation module is mainly used to accumulate long-term features of the video in a loop, so that the feature information of the previous time step and the subsequent time step is accumulated when processing each time step; the full-time asymptotic fusion module is mainly used to perform full-time fusion of the features accumulated by bidirectional hybrid propagation; the motion and content extraction module is mainly used to extract full-sequence motion feature information and full-sequence content feature information from the inter-frame full-sequence fusion features through Transformer and spatiotemporal sliding window; the low-frequency intermediate frame generation module mainly acts as a base to supplement information for the next step; the spatiotemporal super-resolution decoding module mainly uses the full-sequence information corresponding to each frame obtained by the previous full-time asymptotic fusion module, as well as the full-sequence motion feature information and full-sequence content feature information calculated by the motion and content extraction module, to parse and generate the video spatiotemporal super-resolution residual, which is then attached to the base after bicubic upsampling of the low-resolution sequence image.

[0051] In some embodiments of this application, the fusion features are determined as follows: a preset model is used to process each image frame in the full sequence of image frames using a bidirectional background detection algorithm based on Gaussian mixture distribution to obtain forward motion foreground information and backward motion foreground information corresponding to each image frame; the forward motion foreground information and backward motion foreground information corresponding to each image frame are binarized to obtain the time step of each image frame; each image frame and the time step of each image frame are combined to obtain the time step information corresponding to each image frame; the time step information corresponding to each image frame is processed to obtain the fusion features of the full sequence of image frames.

[0052] Specifically, an input time step is constructed through a video input module. A low-frame-rate, low-resolution video (the video to be processed) is input, and a bidirectional background detection algorithm based on Gaussian mixture distribution is applied to each frame to obtain forward and backward motion foreground information for each frame. This process extracts prior knowledge, focusing more on foreground information during spatiotemporal super-resolution reconstruction. After binarizing the two foreground information pieces for each frame, the corresponding time step information is obtained by combining them.

[0053] The specific steps for processing the time step information corresponding to each image frame to obtain the fusion features of the entire sequence of image frames are as follows: Feature extraction is performed on the time step information corresponding to each image frame to obtain a multi-layer feature map for each image frame, wherein the size of the multi-layer feature map decreases sequentially; the offset of each feature map is determined based on the offset of the feature map of the previous layer; the multi-layer feature map is aligned with the multi-layer feature maps of adjacent image frames of each image frame according to the offset of each feature map to obtain aligned features; the fusion features of the entire sequence of image frames are determined based on the aligned features.

[0054] Specifically, feature extraction and alignment are performed on the time step information corresponding to each image frame. Based on the pyramid cascade thinning principle most commonly used in optical flow, this application embodiment designs a new pyramid deformable alignment module to perform feature extraction and alignment on the time step. Each input image of each time step is processed by strided convolutional layers to extract features, resulting in three levels of features of size 1 / 2, 1 / 4, and 1 / 8. The features of the previous level serve as the kernel for the deformable convolution of the next level. The offset of the current level is concatenated with the offset of the next level after interpolation and upsampling. The offset of the deformable convolution kernel of the current level is obtained by calculating and adjusting the dimensions using convolution, and then applied to the deformable convolution of the current level. After the pyramid cascade deformable convolution from coarse to fine, the features of the target frame aligned to that frame (adjacent image frames) are finally obtained.

[0055] For example, for each image frame's corresponding time step information, each frame image undergoes feature extraction through strided convolutional layers, generating feature maps at multiple scales, with scales corresponding to 1 / 2, 1 / 4, and 1 / 8 of the original image, respectively. These feature maps contain information at different levels of image abstraction and will be used for subsequent alignment and fusion. For each feature layer, using the offsets from the previous layer and the offsets calculated for the current layer, a deformable convolutional kernel offset suitable for the current feature layer is generated by convolution calculation and dimension adjustment. Deformable convolution is applied: based on the calculated offsets, the feature map of the current layer undergoes dynamic spatial transformation to align with the target frame's position, handling feature displacement caused by object motion. This process is repeated until all three feature layers are aligned. It should be noted that for each feature layer, feature alignment with adjacent frames (previous or next frame) on that layer is performed.

[0056] It should also be noted that the bidirectional hybrid features can be determined in the following way: using the optical flow model in the preset model, the forward optical flow map and the reverse optical flow map are extracted from the time step information corresponding to each image frame; the forward optical flow map and the reverse optical flow map are respectively subjected to feature extraction to obtain forward features and reverse features; the forward features and the reverse features are fused to obtain the bidirectional hybrid features.

[0057] By combining forward and backward feature blending, each frame can capture information from all time steps in the sequence, not just its immediate neighbors. This enhances the model's perception of the entire video sequence, providing rich temporal and spatial features for subsequent full-time asymptotic fusion. Through forward recurrent propagation, the model accumulates temporal information from the past (beginning of the video) to the current frame. During each frame processing, the model considers not only the features of the current frame but also the accumulated features from previous frames, performing feature fusion and information updates to obtain the forward blended feature state for the current frame. The backward recurrent propagation model accumulates temporal information from the future (end of the video) to the current frame, processing each frame accordingly. Similar to forward propagation, backward propagation also considers the feature information of the current and future frames to obtain the backward blended feature state for each frame.

[0058] Furthermore, the specific steps for determining the fused features based on the aligned features are as follows: the aligned features and the bidirectional mixed features are determined as target features; the target features are divided into past feature states, present feature states, and future feature states; the past feature states, present feature states, and future feature states are input into three channels respectively to obtain three feature maps, and the three feature maps are stitched together to obtain a stitched feature map; the fused features of the entire sequence image frame are determined based on the stitched feature map.

[0059] Specifically, a cyclical propagation module iteratively extracts bidirectional feature information accumulated up to the current time step from each frame. This novel design allows each time step to utilize the accumulated features from all time steps in the temporal sequence for reconstruction. Compared to iterative propagation frameworks, it enables long-range modeling and accumulates more information, providing more insights into object motion and occlusion. Compared to cyclical propagation frameworks, it avoids the information imbalance problem caused by the order of time steps during reconstruction. This approach allows each frame to perceive the entire time sequence. Figure 4As shown, the input is first processed with prior information. A trained optical flow model is used to extract forward and backward optical flow maps from the input image sequence, and the input frames are combined to form the prior information input. For each input frame, a recurrent propagation model is used to forward and backward input sequences respectively to obtain the mixed features of forward and backward directions.

[0060] The aligned features and bidirectional mixed features are progressively fused across three channels using a full-time progressive fusion module. Feature states from the past, present, and future are fed into three channels in the progressive fusion residual block. Each channel first passes through a convolutional layer with a receptive field of 3 followed by an activation layer; this combination is referred to as the convolutional block below. The feature maps computed on each channel are concatenated. The aggregated deep feature map contains a large amount of time-related information, requiring a convolutional layer to further utilize this temporal information. After feature fusion and dimensionality reduction, the hidden state is input into a non-local attention block, resulting in a feature state containing global information. This feature state is then concatenated with the feature states from the three channels before concatenation, ensuring each channel contains global context information. Each channel is then convolved and residually connected to obtain the three-channel output of the block. After multiple full-sequence progressive fusions, the information of the fused features across the entire sequence for each frame is obtained.

[0061] It's important to clarify that past feature states refer to the feature information preceding the current frame (historical frames) in the time series. These features are accumulated through forward propagation and encompass the dynamic changes and historical motion information of the video sequence from the starting frame to the current frame. Current feature states specifically refer to the feature information obtained after processing by the feature extraction and alignment module for the current frame. They reflect the direct features and details of the video content at the current moment in the time series. Future feature states refer to the feature information following the current frame (future frames) in the time series. These features are accumulated through backward propagation and encompass the anticipated dynamic changes and motion information of the video sequence from the current frame to the ending frame.

[0062] In some embodiments of this application, content features are determined as follows: any patch in the current image frame is selected as a query vector, and patches at corresponding positions in adjacent image frames before and after the query vector are respectively determined as key vectors and value vectors; an attention weight matrix is ​​determined based on the query vector and the key vectors, wherein the attention weight matrix is ​​used to represent the correlation between patches in the previous image frame and patches in adjacent image frames; the attention weight matrix is ​​used to weight the patches in the adjacent image frames to obtain the content features of the patches in the current image frame.

[0063] The action features are determined by assigning position codes to patches in each image frame and weighting the position codes of each patch using the attention weight matrix to obtain the action features of the entire sequence of image frames.

[0064] Specifically, the process of extracting action and content features is as follows: By reusing the feature attention map, the module explicitly extracts inter-frame content feature information and motion estimation information. The content feature information directly provides content clues for generating low-resolution intermediate frames, and the motion information after motion estimation warps the input frame, providing more references for generating intermediate frame residuals. For example... Figure 5 As shown, the motion and content extraction modules can be stacked to obtain hierarchical motion and appearance information. For any patch in the current frame, it is used as a query vector, and its temporal neighbors are used as key and value vectors, respectively, to derive an attention map representing its temporal relevance. The attention map is then used to aggregate the appearance features of neighbors to represent the current region as context. Additionally, the attention map is used to weight the displacements of neighbors to obtain approximate motion vectors for patches from the current frame to adjacent frames. The specific calculation logic within the motion and content parser is as follows: Figure 6 As shown, for two features (fusion features of two adjacent image frames), denoted as i and j, key vector and value vector pairs are calculated using the Q (query vector) of feature i and the K (key vector) and V (value vector) of feature j. Similarly, the Q of feature j is calculated using the K and V of feature i. Specifically, each Q block (patch) of feature j is calculated with all K and V of feature i. The Q block is first calculated with K to obtain an attention map. The attention map represents the weight information, that is, the degree of correlation between a certain block of feature i and feature j at all positions. If its value vector is large, it means that the position of this block of feature i is highly correlated with the corresponding position of feature j, and the information of this block is emphasized in subsequent super-resolution reconstruction. The attention map is multiplied by the value vector information of feature j, that is, the weight information of the corresponding position is multiplied with the V of feature j. The value vector of the corresponding position is obtained according to the degree of correlation. The value vector is residually concatenated with feature i, that is, added to obtain the content feature C. Multiplying the attention map with the coordinate encoding of feature j allows for a direct extraction of the correlation between the block and each position. Simultaneously, the attention map implicitly extracts motion actions. After multiplying with the position encoding, the correlation level indicates which part of feature i the block originates from. This process is achieved by subtracting the coordinate encoding with correlation from the position encoding of feature j. Finally, the obtained content features and motion information are used for motion estimation and appearance refinement to synthesize intermediate frames.

[0065] After obtaining the fusion features, motion features, and content features of the entire sequence of image frames, a target video is determined based on these features. The residual of the target video is then determined based on these features. Finally, the target video is generated based on the residual.

[0066] The spatiotemporal super-resolution decoding module analyzes the full-time progressive fusion features, motion information features, and full-sequence content information features generated in previous steps, ultimately obtaining a high frame rate, high-resolution residual (the residual of the target video). This step also enables feature reuse, avoiding the impact of intermediate errors on the generation effect. Furthermore, thanks to the full-sequence propagation mechanism, the generated high frame rate, high-resolution video frames exhibit strong long-term distance modeling capabilities, stable frame generation, and low average and variance peak signal-to-noise ratio. The use of the Transformer mechanism in the motion and content extraction module allows for explicit extraction of motion information, significantly improving artifact handling in the generated high frame rate, high-resolution video frames. The explicit extraction of full-sequence content information also contributes to good occlusion handling in the generated high frame rate, high-resolution video frames. Internal calculations are as follows... Figure 7As shown, the input consists of full-sequence fusion feature information from the previous and next frames, full-sequence action feature information, full-sequence content feature information, and the full-sequence fusion feature information of the current frame. These are continuously convolved, stacked, and fused to obtain the final output, which is used to directly generate video spatiotemporal super-resolution. The full-sequence fusion feature information is obtained through processing by the full-time asymptotic fusion module. It integrates the feature information of each frame in the video sequence, including not only the information of the current frame but also the full-sequence temporal information from past frames to future frames. This feature, processed by the loop propagation module and the full-time asymptotic fusion module, can capture the long-term dependencies and temporal features of the video content, helping the model to fully understand the dynamic changes of the video. Action features represent the action and motion information between video frames, including object displacement, rotation, scaling, etc. These features are crucial for reconstructing motion details in video frames, especially when dealing with occluded and fast-moving objects. By explicitly extracting the full-sequence motion information between video frames, action features provide key dynamic cues for video frame interpolation and spatiotemporal super-resolution processes, helping to reduce artifacts and motion blur in the generated frames. Content features represent specific content information within or between image frames, such as texture, color, and shape. They are calculated through the content feature extraction component of the inter-frame attention module, combining information from the entire sequence to enable the model to more accurately understand and reconstruct details within video frames. Content features directly provide content cues for generating high frame rate, high-resolution video frames, ensuring that reconstructed frames are not only temporally accurate but also spatially detailed and possess high-quality visual effects.

[0067] It should be noted that during the generation of the target video, a baseline prediction can first be generated using bicubic upsampling or other simple upsampling methods. This baseline prediction serves as an initial high-resolution or high-frame-rate video frame (the background after bicubic upsampling of the low-resolution image sequence). The residual of the target video represents the difference between the high-resolution, high-frame-rate video (the target video) and the baseline prediction. The generated residual is then combined with the baseline prediction result, typically by adding them pixel by pixel, to obtain the target video.

[0068] It is understood that the video spatiotemporal super-resolution method provided in this application embodiment uses a single-stage full-sequence video spatiotemporal super-resolution reconstruction method. By using a one-stage model, it achieves full-sequence feature extraction and simultaneously generates spatiotemporally super-resolution frames for the features, enabling in-depth feature extraction and utilization, and avoiding the cumulative impact of errors through a unified overall model generation. By constructing a full-sequence propagation mechanism, namely through the joint action of a feature extraction and alignment module, a loop propagation module, and a full-time progressive fusion module, a full-sequence propagation mechanism is constructed. This allows the present invention to utilize the information of the entire video sequence when generating each frame, providing sufficient information for the model. Furthermore, through the action and content extraction module, the full-sequence action feature information and full-sequence content feature information between frames are explicitly extracted, providing explicit features for spatiotemporal super-resolution, solving the content generation under occlusion conditions, and reducing possible artifacts and blurring.

[0069] Alternatively, the preset model can be trained in the following way: The relevant training parameters are as follows: using the ADAM optimizer, β1 is set to 0.9, β2 is set to 0.999, the learning rate is set to 2e-4, and the value of α in the loss function is set to 2 × 10-3. The training takes about 300,000 rounds.

[0070] Figure 8 A video spatiotemporal super-resolution device is shown, the device comprising:

[0071] Receiver module 80 is used to receive the video to be processed;

[0072] The processing module 82 is used to process the full sequence image frames corresponding to the video to be processed using a preset model to obtain the fusion features of the full sequence image frames, the motion features of the full sequence image frames, and the content features of the full sequence image frames. The fusion features of the full sequence image frames are used to characterize the temporal information of each image frame in the full sequence image frames. The motion features of the full sequence image frames are used to represent the motion information of each image frame in the full sequence image frames. The content features of the full sequence image frames are used to characterize the content information of each image frame in the full sequence image frames.

[0073] The determining module 84 is used to determine the target video based on the fusion features of the full sequence image frames, the action features of the full sequence image frames, and the content features of the full sequence image frames, wherein the target video is the super-resolution reconstructed video corresponding to the video to be processed.

[0074] The aforementioned video spatiotemporal super-resolution device employs the following steps: receiving a video to be processed; processing the entire sequence of image frames corresponding to the video to be processed using a preset model to obtain the fusion features, motion features, and content features of the entire sequence of image frames. The fusion features of the entire sequence of image frames are used to characterize at least the temporal information of each image frame in the entire sequence of image frames; the motion features of the entire sequence of image frames are used to represent at least the motion information of each image frame in the entire sequence of image frames; and the content features of the entire sequence of image frames are used to characterize at least the content information of each image frame in the entire sequence of image frames. Based on the entire sequence of image frames... The target video is determined by fusing features, motion features of the entire sequence of image frames, and content features of the entire sequence of image frames. The target video is the super-resolution reconstructed video corresponding to the video to be processed. The video to be processed is processed by a preset model to obtain multiple features of the video to be processed. Based on the multiple features of the video to be processed, the reconstructed target video is determined. This achieves the goal of avoiding the reconstruction of the video through a two-stage reconstruction method, thereby improving the technical effect of the reconstructed video's clarity. This solves the technical problem that related technologies use a two-stage method of magnification and frame interpolation to perform spatiotemporal super-resolution reconstruction of videos, resulting in low clarity of the reconstructed video.

[0075] The processing module 82 includes: a fusion submodule, used to perform bidirectional background detection algorithm based on Gaussian mixture distribution on each image frame in the full sequence of image frames using a preset model to obtain forward motion foreground information and backward motion foreground information corresponding to each image frame; to perform binarization processing on the forward motion foreground information and backward motion foreground information corresponding to each image frame to obtain the time step of each image frame; to combine each image frame and the time step of each image frame to obtain the time step information corresponding to each image frame; and to process the time step information corresponding to each image frame to obtain the fusion feature of the full sequence of image frames.

[0076] The fusion submodule includes a fusion unit, used to process the time step information corresponding to each image frame to obtain the fusion features of the entire sequence of image frames. This includes: extracting features from the time step information corresponding to each image frame to obtain a multi-layer feature map for each image frame, wherein the size of the multi-layer feature maps decreases sequentially; determining the offset of each feature map based on the offset of the feature map of the previous layer; aligning the multi-layer feature maps with the multi-layer feature maps of adjacent image frames of each image frame according to the offset of each feature map to obtain aligned features; and determining the fusion features of the entire sequence of image frames based on the aligned features.

[0077] The fusion unit includes a fusion subunit, used to determine the fusion features based on the aligned features, including: determining the aligned features and bidirectional mixed features as target features; dividing the target features into past feature states, present feature states, and future feature states; inputting the past feature states, present feature states, and future feature states into three channels respectively to obtain three feature maps, and concatenating the three feature maps to obtain a concatenated feature map; and determining the fusion features of the entire sequence image frames based on the concatenated feature map.

[0078] The fusion submodule also includes a bidirectional unit, which is used to extract a forward optical flow map and a reverse optical flow map from the time step information corresponding to each image frame using the optical flow model in the preset model; to extract features from the forward optical flow map and the reverse optical flow map respectively to obtain forward features and reverse features; and to fuse the forward features and the reverse features to obtain bidirectional hybrid features.

[0079] The processing module 82 further includes a content submodule and an action submodule. The content submodule is used to select any patch in the current image frame as a query vector, and to determine the corresponding patches in the adjacent image frames before and after the query vector as key vectors and value vectors, respectively. An attention weight matrix is ​​determined based on the query vector and the key vectors, wherein the attention weight matrix is ​​used to represent the correlation between patches in the previous image frame and patches in adjacent image frames. The attention weight matrix is ​​used to weight the patches in the adjacent image frames to obtain the content features of the patches in the current image frame.

[0080] The action submodule is used to assign position codes to patches in each image frame; the attention weight matrix is ​​used to weight the position codes of each patch to obtain the action features of the entire sequence of image frames.

[0081] The aforementioned video spatiotemporal super-resolution device further includes: a determination submodule, used to determine a target video based on the fusion features of the full sequence of image frames, the motion features of the full sequence of image frames, and the content features of the full sequence of image frames, including: determining the residual of the target video based on the fusion features of the full sequence of image frames, the motion features of the full sequence of image frames, and the content features of the full sequence of image frames; and generating the target video based on the residual of the target video.

[0082] It should be noted that, Figure 8 The video spatiotemporal super-resolution device shown is used to perform Figure 2 The video spatiotemporal super-resolution method shown above, therefore, the relevant explanations and descriptions in the video spatiotemporal super-resolution method also apply to this video spatiotemporal super-resolution device, and will not be repeated here.

[0083] This application also provides a computer device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute the above-described video spatiotemporal super-resolution method.

[0084] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the video spatiotemporal super-resolution method in this application.

[0085] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0086] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0088] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0090] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0091] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A video spatiotemporal super-resolution method, characterized in that, include: Receive the video to be processed; A preset model is used to process the full sequence of image frames corresponding to the video to be processed, to obtain the fusion features, motion features, and content features of the full sequence of image frames. The fusion features are used to characterize the temporal information of each image frame in the full sequence of image frames, the motion features are used to represent the motion information of each image frame in the full sequence of image frames, and the content features are used to characterize the content information of each image frame in the full sequence of image frames. The target video is determined based on the fusion features of the full sequence of image frames, the motion features of the full sequence of image frames, and the content features of the full sequence of image frames, wherein the target video is the super-resolution reconstructed video corresponding to the video to be processed.

2. The method according to claim 1, characterized in that, The fusion features are determined in the following ways: A pre-defined model is used to process each image frame in the entire sequence of image frames using a bidirectional background detection algorithm based on Gaussian mixture distribution, to obtain the forward motion foreground information and backward motion foreground information corresponding to each image frame; The forward motion foreground information and backward motion foreground information corresponding to each image frame are binarized to obtain the time step of each image frame; By combining each image frame with its time step, the time step information corresponding to each image frame is obtained; The time step information corresponding to each image frame is processed to obtain the fusion features of the entire sequence of image frames.

3. The method according to claim 2, characterized in that, The time step information corresponding to each image frame is processed to obtain the fusion features of the entire sequence of image frames, including: Feature extraction is performed on the time step information corresponding to each image frame to obtain a multi-layer feature map for each image frame, wherein the size of the multi-layer feature map decreases sequentially. The offset of each feature map is determined based on the offset of the feature map of the previous layer. The multi-layer feature map is aligned with the multi-layer feature map of the adjacent image frames of each image frame based on the offset of each layer feature map to obtain the aligned features. The fusion features of the entire sequence of image frames are determined based on the aligned features.

4. The method according to claim 3, characterized in that, Determining the fused features based on the aligned features includes: The aligned features and the bidirectional blended features are determined as the target features; The target features are divided into past feature states, present feature states, and future feature states; The past feature state, the present feature state, and the future feature state are respectively input into three channels to obtain three feature maps, and the three feature maps are then stitched together to obtain a stitched feature map. The fusion features of the entire sequence of image frames are determined based on the stitched feature map.

5. The method according to claim 2, characterized in that, The method further includes: The forward optical flow map and the reverse optical flow map are extracted from the time step information corresponding to each image frame using the optical flow model in the preset model. Feature extraction is performed on the forward optical flow map and the reverse optical flow map respectively to obtain forward features and reverse features; The positive and negative features are fused to obtain a bidirectional hybrid feature.

6. The method according to claim 1, characterized in that, The content features are determined in the following ways: Select any patch in the current image frame as the query vector, and determine the corresponding patches in the adjacent image frames before and after the patch corresponding to the query vector as the key vector and value vector, respectively. An attention weight matrix is ​​determined based on the query vector and the key vector, wherein the attention weight matrix is ​​used to represent the correlation between patches in the previous image frame and patches in the adjacent image frame; The attention weight matrix is ​​used to weight the patches in the adjacent image frames to obtain the content features of the patches in the current image frame.

7. The method according to claim 6, characterized in that, The action characteristics are determined in the following ways: Assign position codes to patches in each image frame; The positional encoding of each patch is weighted using the attention weight matrix to obtain the action features of the entire sequence of image frames.

8. The method according to claim 6, characterized in that, Determining the target video based on the fusion features of the entire sequence of image frames, the motion features of the entire sequence of image frames, and the content features of the entire sequence of image frames includes: The residual of the target video is determined based on the fusion features of the full sequence of image frames, the motion features of the full sequence of image frames, and the content features of the full sequence of image frames; The target video is generated based on the residual of the target video.

9. A video spatiotemporal super-resolution device, characterized in that, include: The receiving module is used to receive the video to be processed; The processing module is used to process the full sequence of image frames corresponding to the video to be processed using a preset model to obtain the fusion features, motion features, and content features of the full sequence of image frames. The fusion features of the full sequence of image frames are used to characterize the temporal information of each image frame in the full sequence of image frames. The motion features of the full sequence of image frames are used to represent the motion information of each image frame in the full sequence of image frames. The content features of the full sequence of image frames are used to characterize the content information of each image frame in the full sequence of image frames. The determination module is used to determine the target video based on the fusion features of the full sequence of image frames, the action features of the full sequence of image frames, and the content features of the full sequence of image frames, wherein the target video is the super-resolution reconstructed video corresponding to the video to be processed.

10. A computer device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; The processor, connected to the memory, is used to execute the video spatiotemporal super-resolution method according to any one of claims 1 to 8.

Images