Video space-time super-division method and device based on instant access, equipment and storage medium

Abstract

The invention discloses a video space-time super-division method and device based on instant access, equipment and a storage medium. The method comprises the following steps: acquiring input frame image data and a feature map extraction model; obtaining an intermediate frame feature map based on the input frame image data and a feature map extraction model; acquiring an initial backward hidden state; inputting the intermediate frame feature map and the initial backward hidden state into a backward circulation module to obtain a target backward hidden state; updating the backward hidden state of the target into the access container; acquiring an initial forward hiding state; inputting the intermediate frame feature map, the target backward hiding state and the initial forward hiding state into a forward circulation module to obtain a target forward hiding state; and inputting the target forward hidden state and the target backward hidden state to a fusion reconstruction module to obtain a target video. According to the invention, the forward circulation module uses the hidden state of the backward circulation module, the space-time information of all frames can be fully utilized, and the super-resolution effect is improved.

Description

Technical Field

[0001] The present invention belongs to the field of video image processing, and in particular relates to a video spatiotemporal super-resolution method, device, equipment and storage medium based on instant access. Background Art

[0002] Video image processing has always been the focus of researchers in the field of computer vision and is an important research direction in this field. Its core work includes tasks such as video image restoration, compression, and super-resolution. The goal of video spatiotemporal super-resolution is to reconstruct a realistic video with high resolution and high frame rate from a given low-resolution and low-frame-rate video. This type of problem involves not only the synthesis of intermediate frames, but also the super-resolution processing of video frames. It is a complex problem that combines two subtasks and is a very challenging research direction.

[0003] In film production, high-speed cameras and high-resolution cameras have been widely used to capture more detailed and clear images and enhance the visual effects of movies. However, due to the high hardware requirements and expensive system prices of these devices, this type of technology has not been widely used in other fields. In contrast, deep learning-based video spatiotemporal super-resolution technology only requires low frame rate and low resolution video sequences to generate high-quality high-resolution and high frame rate videos. Due to its wide application in slow motion generation, film production, HDTV upgrades and other fields, this technology has attracted more and more attention.

[0004] The task of video spatiotemporal super-resolution aims to improve the spatial and temporal resolution of video sequences. The core challenge is how to make full use of the spatiotemporal information in video sequences. Traditional methods (such as Shechtman et al.) usually optimize the model through hand-designed regularization, prior knowledge and specific assumptions, but these limitations often make it difficult for the model to build accurate spatiotemporal associations in complex patterns. In recent years, some studies have proposed to achieve spatiotemporal super-resolution by jointly learning video super-resolution and video interpolation techniques. Xiang et al. first explored the local temporal context for feature interpolation and then used the global temporal context to complete the super-resolution processing. On this basis, Xu et al. further proposed the use of a local temporal feature comparison module to mine local motion cues for refinement. However, the above methods still fail to achieve accurate and comprehensive modeling of the spatiotemporal associations of video sequences.

[0005] The storage and retrieval network is designed to store potentially useful information for subsequent deeper fusion and refinement. This method has been widely used in natural language processing and video object segmentation. Based on this, we designed an instant access scheme to fully exploit the spatiotemporal correlation in the video, thereby achieving the improvement of spatiotemporal super-resolution of the video.

[0006] The inventor has conducted a survey on some of the latest video spatiotemporal super-resolution patents in recent years, and made a detailed comparison between the present invention and the current patents. Details are as follows:

[0007] Chinese patent document number CN116668738A, publication (announcement) date 2023.08.29. A video spatiotemporal super-resolution method is disclosed. The method first extracts the initial features of each frame in the video to be reconstructed through an initial feature extraction module, which is called the odd frame initial features. Subsequently, a bidirectional alignment module is used to generate the corresponding initial features of the intermediate frames to be interpolated based on the initial features of adjacent odd frames, which are called the even frame initial features. Next, the initial features of the odd and even frames are simultaneously input into the feature extraction and fusion module based on spatiotemporal information to obtain the deep features of each frame. Finally, the deep features of all frames are restored to obtain a reconstructed video sequence. However, this method only utilizes the information between adjacent frames and fails to fully utilize the correlation between long-distance frames.

[0008] Chinese patent document number CN114757828A, publication (announcement) date 2022.07.15. A Transformer-based video spatiotemporal super-resolution method is disclosed. The method first performs temporal feature fusion through the Transformer's attention mechanism to generate a high-frame-rate, low-resolution video sequence; then, based on the Transformer's spatiotemporal feature fusion, a high-frame-rate, high-resolution video sequence is reconstructed. However, this method only utilizes low-frame-rate image sequences in the interpolation process, and fails to fully combine the two subtasks of video interpolation and video super-resolution to achieve a closer joint optimization.

[0009] Chinese patent document number CN113034380A, publication (announcement) date 2021.06.25. A video spatiotemporal super-resolution method based on improved deformable convolution correction is disclosed. The method uses a feature extraction module to extract feature maps from input low-resolution adjacent video frames, uses an inter-frame correction module to perform correction processing based on the feature maps corresponding to adjacent video frames to synthesize intermediate frame feature maps, and uses an image reconstruction module to extract inter-frame and intra-frame features from the input intermediate frame feature maps and the feature maps corresponding to adjacent video frames, and reconstructs and outputs a high-resolution and high-frame-rate image sequence. However, this method only uses a low-resolution frame sequence to reconstruct high-resolution frames, and the low-resolution interpolation does not utilize the information of the high-resolution feature map in the video super-resolution task.

[0010] Chinese patent document number CN112862675A, publication (announcement) date 2021.05.28. A video spatiotemporal super-resolution method is disclosed, which utilizes the relationship between time and space, jointly learns in time and space, and directly super-reconstructs from low-resolution LR to high-resolution HR in multi-resolution ST-MR, thereby obtaining rich multi-scale features to enhance spatial resolution S-SR and temporal resolution T-SR, and finally forms a spatiotemporal super-resolution network STAMnet to solve the problem that ST-MR only processes contextual information separately in time and space. However, this method only uses the enhanced super-resolution process of the low-resolution image sequence, and does not fully utilize the rich spatial information of the super-resolution feature map to repair the loss of the low-resolution interpolation result. . Summary of the invention

[0011] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a video spatiotemporal super-resolution method, device, equipment and storage medium based on real-time access to solve the deficiencies in the prior art.......

[0012] According to one aspect of the present application, a video spatiotemporal super-resolution method based on instant access is disclosed. The method is implemented based on a timely access video spatiotemporal super-resolution system. The system includes a backward loop module, a forward loop module and a fusion reconstruction module. The method includes:

[0013] Obtain input frame image data and feature map extraction model;

[0014] Based on the input frame image data and the feature map extraction model, an intermediate frame feature map is obtained;

[0015] Get the initial backward hidden state;

[0016] Inputting the intermediate frame feature map and the initial backward hidden state into the backward loop module to obtain a target backward hidden state;

[0017] Updating the target backward hidden state into the access container;

[0018] Get the initial forward hidden state;

[0019] Inputting the intermediate frame feature map, the target backward hidden state and the initial forward hidden state into the forward loop module to obtain the target forward hidden state;

[0020] The target forward hidden state and the target backward hidden state are input into the fusion reconstruction module to obtain the target video.

[0021] In some embodiments, obtaining the intermediate frame feature map based on the input frame image data and the feature map extraction model includes:

[0022] The input frame image data is input into the feature map extraction model so that the feature map extraction model performs feature extraction based on linear interpolation to obtain the intermediate frame feature map.

[0023] In some embodiments, the system further includes an adjacent frame acquisition submodule, a remote frame acquisition submodule, and a dynamic selection module, the initial backward hidden state includes a first initial adjacent frame hidden state and a first initial remote frame hidden state, wherein the first initial adjacent frame hidden state represents a hidden state of a future adjacent frame, and the first initial remote frame hidden state represents a hidden state of a future remote frame; the step of inputting the intermediate frame feature map and the initial backward hidden state into the backward loop module to obtain a target backward hidden state includes:

[0024] Inputting the intermediate frame feature map and the first adjacent frame hidden state into the adjacent frame acquisition submodule to obtain a first aligned hidden state;

[0025] The first aligned hidden state is determined based on the following formula:

[0026]

[0027] In the formula, is the first alignment hidden state;

[0028] is the feature map of the middle frame;

[0029] is the first initial adjacent frame hidden state;

[0030] AF is the adjacent frame acquisition submodule;

[0031] Inputting the intermediate frame feature map and the first remote frame hidden state into the remote frame acquisition submodule to obtain a backward decoding resolution;

[0032] The first decoding resolution can be determined based on the following formula:

[0033]

[0034] In the formula, is the first decoding resolution;

[0035] DF is the remote frame acquisition submodule;

[0036] is the first initial remote frame hiding state;

[0037] Inputting the adjacent frame aligned hidden state, the first decoding resolution, and the intermediate frame feature map into the dynamic selection module to obtain the target backward hidden state;

[0038] The target backward hidden state is determined based on the following formula:

[0039]

[0040] In the formula, is a first target adjacent frame hidden state in the target backward hidden state, wherein the first target adjacent frame hidden state represents a hidden state of a future adjacent intermediate frame;

[0041] is a first target remote frame hidden state in the target backward hidden state, wherein the first target remote frame hidden state represents a hidden state of a future remote intermediate frame;

[0042] DS is a dynamic selection module.

[0043] In some embodiments, the initial forward hidden state includes a second initial adjacent frame hidden state and a second initial remote frame hidden state, wherein the second initial adjacent frame hidden state represents a hidden state in the past to an adjacent frame, and the second initial remote frame hidden state represents a hidden state in the past to a remote frame, and the inputting the intermediate frame feature map, the target backward hidden state, and the initial forward hidden state into the forward loop module to obtain the target forward hidden state includes:

[0044] Inputting the intermediate feature map, the second initial adjacent frame hidden state, and the first initial adjacent frame hidden state into the forward adjacent frame acquisition submodule to obtain an aligned hidden state;

[0045] The aligned hidden state is obtained based on the following formula:

[0046]

[0047] In the formula, is the first aligned hidden state in the aligned hidden states;

[0048] is a second aligned hidden state in the aligned hidden states;

[0049] is the feature map of the middle frame;

[0050] is the second initial adjacent frame hidden state;

[0051] is the first initial adjacent frame hidden state;

[0052] AF is the adjacent frame acquisition submodule;

[0053] Inputting the intermediate frame feature map, the second initial remote frame hidden state, and the first initial remote frame hidden state into the remote frame acquisition submodule to determine a decoding resolution;

[0054] The decoding resolution is determined based on the following formula:

[0055]

[0056] In the formula, is the first decoding resolution among the decoding resolutions;

[0057] is a second decoding resolution among the decoding resolutions;

[0058] is the feature map of the middle frame;

[0059] is the second initial remote frame hiding state;

[0060] is the first initial remote frame hiding state;

[0061] DF is the remote frame acquisition submodule;

[0062] The target forward hidden state is determined based on the intermediate frame feature map, the aligned hidden state, and the decoding resolution.

[0063] In some embodiments, determining the decoding resolution further comprises:

[0064] Downsampling the first initial remote frame hidden state, the second initial remote frame hidden state, and the intermediate frame feature map;

[0065] Among them, it can be realized based on the following formula:

[0066]

[0067] In the formula, is the second initial remote frame hiding state;

[0068] is the first initial remote frame hiding state;

[0069] is the feature map of the middle frame;

[0070] E stands for encoder, and C stands for channel connection operation;

[0071] To obtain the down-sampling value;

[0072] The down-sampling results are weighted summed to obtain the relevant components;

[0073] Among them, the relevant components can be determined based on the following formula:

[0074]

[0075] In the formula, is the second correlation component in the correlation components;

[0076] is the first correlation component among the correlation components;

[0077] The relevant components are decoded to obtain the decoding resolution.

[0078] Specifically, it can be implemented based on the following formula:

[0079]

[0080] In the formula, is the first decoding resolution among the decoding resolutions;

[0081] is a second decoding resolution among the decoding resolutions;

[0082] D stands for decoder.

[0083] In some embodiments, the dynamic selection submodule further includes a dynamically selected attention unit and a fused residual unit, and the determining the target forward hidden state based on the aligned hidden state and the decoding resolution further includes:

[0084] Inputting the aligned hidden state and the decoding resolution into the dynamic selection attention unit to obtain an attention map;

[0085] The attention map and the intermediate frame feature map are processed in combination with the fusion residual unit to obtain the target forward hidden state, wherein the target forward hidden state includes a second target adjacent frame hidden state and a second target remote frame hidden state in the target forward hidden state, the second target adjacent frame hidden state represents the hidden state from the past to the adjacent intermediate frame, and the second target remote frame hidden state represents the hidden state from the past to the remote intermediate frame;

[0086] Among them, the attention map can be obtained based on the following formula:

[0087]

[0088]

[0089] In the formula, CA stands for cross attention;

[0090] is the attention map obtained by the feature map of the middle frame relative to the second aligned hidden state;

[0091] is the attention map obtained by the feature map of the intermediate frame relative to the first aligned hidden state;

[0092] The attention map obtained by the intermediate frame feature map relative to the second decoding resolution;

[0093] It is the attention map obtained for the intermediate frame feature map relative to the first decoding resolution.

[0094] According to another aspect of the present application, a video spatiotemporal super-resolution device based on real-time access is also disclosed, the device comprising:

[0095] A first acquisition module, used to acquire input frame image data and a feature map extraction model;

[0096] An intermediate frame feature map obtaining module, used to obtain an intermediate frame feature map based on the input frame image data and the feature map extraction model;

[0097] A second acquisition module, used to acquire an initial backward hidden state;

[0098] A target backward hidden state obtaining module, used for inputting the intermediate frame feature map and the initial backward hidden state into the backward loop module to obtain a target backward hidden state;

[0099] An updating module, used for updating the target backward hidden state into the access container;

[0100] A third acquisition module is used to obtain an initial forward hidden state;

[0101] A target forward hidden state obtaining module, used for inputting the intermediate frame feature map, the target backward hidden state and the initial forward hidden state into the forward loop module to obtain the target forward hidden state;

[0102] The target video obtaining module is used to input the target forward hidden state and the target backward hidden state into the fusion reconstruction module to obtain the target video.

[0103] According to another aspect of the present application, an electronic device is also disclosed, which includes a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the various steps of the video spatiotemporal super-resolution method based on instant access as described above.

[0104] According to another aspect of the present application, a computer-readable storage medium is also disclosed, on which instructions are stored, characterized in that when the instructions are executed by a processor, the various steps of the video spatiotemporal super-resolution method based on real-time access as described in any of the above items are implemented.

[0105] The present invention includes but is not limited to the following beneficial effects: (1) The forward loop module of the present application uses the hidden state of the backward loop module, which can make full use of the spatiotemporal information of all frames and improve the super-resolution effect; (2) The present application can use super-resolution technology to upgrade low-resolution video frames to high-resolution, thereby improving the quality of the video; (3) The application is based on an instant access system implementation, which can process the input frame image data immediately after obtaining it, thereby improving the processing speed and allowing the method to be used for real-time video stream processing; (4) The method can adapt to different video content and quality requirements by adjusting the parameters of the feature map extraction model, the backward and forward loop modules, and the fusion reconstruction module, and has high flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0107] Figure 1 is a flow chart of a video spatiotemporal super-resolution method based on instant access according to an embodiment of the present application;

[0108] Figure 2 It is a schematic diagram of determining the coding resolution of the video spatiotemporal super-resolution method based on instant access according to an embodiment of the present application;

[0109] Figure 3 It is a principle diagram for determining an attention map of a video spatiotemporal super-resolution method based on instant access according to an embodiment of the present application;

[0110] Figure 4 It is a principle diagram for determining the target forward hidden state of the video spatiotemporal super-resolution method based on instant access in an embodiment of the present application;

[0111] Figure 5 is a structural block diagram of a video spatiotemporal super-resolution device based on real-time access according to an embodiment of the present application;

[0112] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0113] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0114] For ease of understanding, the specific process of the embodiment of the present invention is described below. Specifically, Figure 1 This is a flow chart of the video spatiotemporal super-resolution method based on real-time access of the present application, the steps comprising:

[0115] S100, obtaining input frame image data and feature map extraction model.

[0116] Specifically, seven consecutive frames can be randomly selected from a video, and each frame can be randomly cropped into video frames with two resolutions of 128*128 and 32*32. The 128*128 video frame is used as a high-resolution and high-frame-rate label, and the 32*32 resolution odd-index video frame is used as input frame image data. The feature map extraction model can be pre-trained.

[0117] S102: Obtain an intermediate frame feature map based on the input frame image data and the feature map extraction model.

[0118] Specifically, this step may be to input the input frame image data into the feature map extraction model, so that the feature map extraction model performs feature extraction based on linear interpolation to obtain an intermediate frame feature map.

[0119] S104: Obtain an initial backward hidden state.

[0120] Specifically, the initial backward hidden state can be predetermined and stored in a memory, and can be directly called from the memory when needed. The initial backward hidden state includes a first initial adjacent frame hidden state and a first initial remote frame hidden state, wherein the first initial adjacent frame hidden state represents the hidden state of the future adjacent frame, and the first initial remote frame hidden state represents the hidden state of the future remote frame. It can be understood that the middle frame can be the current frame, the frame adjacent to the middle frame can be the adjacent frame, the frame at a certain distance forward from the middle frame can be the past frame, and the frame at a certain distance backward from the middle frame can be the future frame.

[0121] S106: Input the intermediate frame feature map and the initial backward hidden state into the backward loop module to obtain the target backward hidden state.

[0122] Specifically, before introducing this step, the video spatiotemporal super-resolution system based on real-time access is further introduced. The system also includes an adjacent frame acquisition submodule, a remote frame acquisition submodule and a dynamic selection module. The step S106 can be implemented based on the following method:

[0123] Inputting the intermediate frame feature map and the first adjacent frame hidden state into the adjacent frame acquisition submodule to obtain a first aligned hidden state;

[0124] The first aligned hidden state is determined based on the following formula:

[0125]

[0126] In the formula, is the first alignment hidden state;

[0127] is the feature map of the middle frame;

[0128] is the first initial adjacent frame hidden state;

[0129] AF is the adjacent frame acquisition submodule;

[0130] Input the intermediate frame feature map and the first remote frame hidden state into the remote frame acquisition submodule to obtain the backward decoding resolution;

[0131] The first decoding resolution can be determined based on the following formula:

[0132]

[0133] In the formula, is the first decoding resolution;

[0134] DF is the remote frame acquisition submodule; its principle is to represent the stored hidden state and super-resolution features as a key-value format and a query component. In this way, the query feature is compared with the hidden state in the key space and the relevant associated value is retrieved;

[0135] is the first initial remote frame hiding state;

[0136] Input the adjacent frame aligned hidden states, the first decoding resolution, and the intermediate frame feature map into a dynamic selection module to obtain a target backward hidden state;

[0137] Among them, the target backward hidden state is determined based on the following formula:

[0138]

[0139] In the formula, is a first target adjacent frame hidden state in the target backward hidden state, the first target adjacent frame hidden state representing a hidden state of a future adjacent intermediate frame;

[0140] is a first target remote frame hidden state in the target backward hidden state, the first target remote frame hidden state representing a hidden state of a future remote intermediate frame;

[0141] DS is a dynamic selection module used to guide fusion.

[0142] S108: Update the target backward hidden state into the access container.

[0143] Specifically, the access container is equivalent to a memory, and the obtained target backward hidden state can be updated and stored in the access container first.

[0144] S110, obtaining an initial forward hidden state.

[0145] Specifically, the initial forward hidden state includes a second initial adjacent frame hidden state and a second initial remote frame hidden state, wherein the second initial adjacent frame hidden state represents the hidden state in the past to the adjacent frame, and the second initial remote frame hidden state represents the hidden state in the past to the remote frame.

[0146] S112, inputting the intermediate frame feature map, the target backward hidden state and the initial forward hidden state into the forward loop module to obtain the target forward hidden state.

[0147] Specifically, step S112 can be implemented in the following manner:

[0148] Inputting the intermediate feature map, the second initial adjacent frame hidden state, and the first initial adjacent frame hidden state into a forward adjacent frame acquisition submodule to obtain an aligned hidden state;

[0149] Among them, the aligned hidden state is obtained based on the following formula:

[0150]

[0151] In the formula, is the first aligned hidden state in the aligned hidden states;

[0152] is a second aligned hidden state in the aligned hidden states;

[0153] is the feature map of the middle frame;

[0154] is the second initial adjacent frame hidden state;

[0155] is the first initial adjacent frame hidden state;

[0156] AF is the adjacent frame acquisition submodule, which consists of a temporal deformable network. This module implicitly aggregates information components from adjacent frames and obtains aligned hidden states through a series of motion offset prediction and deformable convolution operations.

[0157] Inputting the intermediate frame feature map, the second initial remote frame hidden state, and the first initial remote frame hidden state into the remote frame acquisition submodule to determine a decoding resolution;

[0158] Specific, combined Figure 2 And the following formula is used to determine the decoding resolution:

[0159]

[0160] In the formula, is the first decoding resolution among the decoding resolutions;

[0161] is a second decoding resolution among the decoding resolutions;

[0162] is the feature map of the middle frame;

[0163] is the second initial remote frame hiding state;

[0164] is the first initial remote frame hiding state;

[0165] DF is the remote frame acquisition submodule;

[0166] Understandably, the adjacent frame acquisition submodule shows its effectiveness in capturing inter-frame motion between adjacent frames. However, due to the large motion between the hidden state and super-resolution features in the bidirectional container, the adjacent frame acquisition block becomes unstable when training for deformable alignment, and the overflow of motion offset severely degrades the model performance. Therefore, we propose the distant frame acquisition block to acquire and collect the hidden state from the bidirectional container, see Figure 2 The principle is to represent the stored hidden states and super-resolution features as a key-value format and a query component. In this way, we compare the query feature with the hidden state in the key space and retrieve the relevant associated value.

[0167] Determine the target forward hidden state based on the intermediate frame feature map, the aligned hidden state, and the decoding resolution.

[0168] In one example, the dynamic selection submodule further includes dynamically selecting an attention unit and a fusion residual unit, and determining the target forward hidden state based on the aligned hidden state and the decoding resolution further includes:

[0169] Input the aligned hidden state and the decoding resolution into the dynamic selection attention unit to obtain the attention map;

[0170] Specific, combined Figure 3 And the following formula to get the attention map:

[0171]

[0172] In the formula, CA stands for cross attention;

[0173] is the attention map obtained by the feature map of the middle frame relative to the second aligned hidden state;

[0174] is the attention map obtained by the feature map of the intermediate frame relative to the first aligned hidden state;

[0175] The attention map obtained by the intermediate frame feature map relative to the second decoding resolution;

[0176] It is the attention map obtained for the intermediate frame feature map relative to the first decoding resolution.

[0177] The attention map and the intermediate frame feature map are processed with the fusion residual unit to obtain the target forward hidden state The target forward hidden state includes the second target adjacent frame hidden state in the target forward hidden state. and the second target remote frame hidden state The second target adjacent frame hidden state represents the hidden state from the past to the adjacent intermediate frame, and the second target remote frame hidden state represents the hidden state from the past to the remote intermediate frame;

[0178] Specific, combined Figure 4 , get the target forward hidden state

[0179] S114, inputting the target forward hidden state and the target backward hidden state into a fusion reconstruction module to obtain a target video.

[0180] In one example, after the target forward hidden state and the target backward hidden state are input into the fusion reconstruction module, the target video is directly output.

[0181] Further, in another example, after the backward loop module and the forward loop module, the target forward hidden state and the target backward hidden state are respectively input to the hybrid fusion module to learn the structure and detail components, and then input to the reconstruction module consisting of two pixel upsampling layers and a series of "convolution layer-LeakyReLU activation function-convolution layer" operations to generate high-resolution (4×) and high-frame rate (2×) videos. In order to optimize the entire network, we use the reconstruction loss function:

[0182]

[0183] Where L r is the specific loss value, and Respectively represent the generated image frame, the generated image frame detail component and the generated image frame structure component; and They represent the real image frame, the real image frame detail component and the real image frame structure component respectively. The real image frame structure component is obtained by bicubic sampling, and the real image frame detail component is the residual between bicubic sampling and the real video frame. is the Charbonnier penalty function, and χ represents the error between the model prediction value and the true value, for example etc., used to measure the gap between the model output and the real data in different aspects, and the constant ω is set to 10 -3 The loss value here is to update the network parameters and optimize the network model. If the loss value converges, the training of the network model is stopped, otherwise it returns to continue training.

[0184] Furthermore, in yet another example, the present application also provides another example of determining the decoding resolution:

[0185] Specifically, it includes downsampling the first initial remote frame hidden state, the second initial remote frame hidden state, and the intermediate frame feature map;

[0186] Among them, it can be realized based on the following formula:

[0187]

[0188] In the formula, is the second initial remote frame hiding state;

[0189] is the first initial remote frame hiding state;

[0190] is the feature map of the middle frame;

[0191] E stands for encoder, and C stands for channel connection operation;

[0192] To obtain the down-sampling value;

[0193] The down-sampling results are weighted summed to obtain the relevant components;

[0194] Among them, the relevant components can be determined based on the following formula:

[0195]

[0196] In the formula, is the second correlation component in the correlation components;

[0197] is the first correlation component among the correlation components;

[0198] The relevant components are decoded to obtain the decoded resolution.

[0199] Specifically, it can be implemented based on the following formula:

[0200]

[0201] In the formula, is the first decoding resolution among the decoding resolutions;

[0202] is a second decoding resolution among the decoding resolutions;

[0203] D stands for decoder.

[0204] It is understandable that the embodiments of the present application can provide long-distance spatiotemporal correlations to complement each other. In order to better utilize the long-distance video frame super-resolution information, we design a bidirectional loop mechanism for obtaining the super-resolution information of past, current and future frames, and storing the updated super-resolution information, so as to aggregate the spatiotemporal information of all frames and reconstruct the video sequence.

[0205] According to another aspect of the present application, a video spatiotemporal super-resolution device based on real-time access is also disclosed, specifically, Figure 5 As shown, the device comprises:

[0206] A first acquisition module, used to acquire input frame image data and a feature map extraction model;

[0207] An intermediate frame feature map obtaining module is used to obtain an intermediate frame feature map based on input frame image data and a feature map extraction model;

[0208] A second acquisition module, used to acquire an initial backward hidden state;

[0209] A target backward hidden state obtaining module is used to input the intermediate frame feature map and the initial backward hidden state into the backward loop module to obtain the target backward hidden state;

[0210] An update module, used for updating the target backward hidden state into the access container;

[0211] A third acquisition module is used to obtain an initial forward hidden state;

[0212] A target forward hidden state obtaining module is used to input the intermediate frame feature map, the target backward hidden state and the initial forward hidden state into the forward loop module to obtain the target forward hidden state;

[0213] The target video obtaining module is used to input the target forward hidden state and the target backward hidden state into the fusion reconstruction module to obtain the target video.

[0214] The application introduction of the relevant modules of the device in this example can refer to the relevant introduction of the principle of the above method, which will not be repeated here.

[0215] According to another aspect of the present application, the present application also discloses an electronic device, which includes a memory and at least one processor, wherein instructions are stored in the memory; at least one processor calls the instructions in the memory so that the electronic device executes the various steps of the above-mentioned video spatiotemporal super-resolution method based on instant access.

[0216] above Figure 6The video spatiotemporal super-resolution device based on instant access in the embodiment of the present invention is described in detail from the perspective of modular functional entities, and the electronic device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0217] Figure 6 6 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 can be short-term storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the electronic device 600. Furthermore, the processor 610 can be configured to communicate with the storage medium 630 to execute a series of instruction operations in the storage medium 630 on the electronic device 600.

[0218] The electronic device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input and output interfaces 650, and / or one or more operating systems 631, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 6 The structure of the electronic device shown does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine some components, or arrange the components differently.

[0219] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of a video spatiotemporal super-resolution method based on instant access.

[0220] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0221] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0222] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video spatiotemporal super-resolution method based on real-time access, characterized in that: The method is implemented based on a timely access video spatiotemporal super-resolution system, the system includes a backward loop module, a forward loop module and a fusion reconstruction module, and the method includes: Obtain input frame image data and feature map extraction model; Based on the input frame image data and the feature map extraction model, an intermediate frame feature map is obtained; Get the initial backward hidden state; Inputting the intermediate frame feature map and the initial backward hidden state into the backward loop module to obtain a target backward hidden state; Updating the target backward hidden state into the access container; Get the initial forward hidden state; Inputting the intermediate frame feature map, the target backward hidden state and the initial forward hidden state into the forward loop module to obtain the target forward hidden state; The target forward hidden state and the target backward hidden state are input into the fusion reconstruction module to obtain the target video.

2. The video spatiotemporal super-resolution method based on real-time access according to claim 1, characterized in that: The step of obtaining the intermediate frame feature map based on the input frame image data and the feature map extraction model comprises: The input frame image data is input into the feature map extraction model so that the feature map extraction model performs feature extraction based on linear interpolation to obtain the intermediate frame feature map.

3. The video spatiotemporal super-resolution method based on real-time access according to claim 1, characterized in that: The system further includes an adjacent frame acquisition submodule, a remote frame acquisition submodule and a dynamic selection module, wherein the initial backward hidden state includes a first initial adjacent frame hidden state and a first initial remote frame hidden state, wherein the first initial adjacent frame hidden state represents a hidden state of a future adjacent frame, and the first initial remote frame hidden state represents a hidden state of a future remote frame; the step of inputting the intermediate frame feature map and the initial backward hidden state into the backward loop module to obtain a target backward hidden state includes: Inputting the intermediate frame feature map and the first adjacent frame hidden state into the adjacent frame acquisition submodule to obtain a first aligned hidden state; The first aligned hidden state is determined based on the following formula: In the formula, is the first alignment hidden state; is the feature map of the middle frame; is the first initial adjacent frame hidden state; AF is the adjacent frame acquisition submodule; Inputting the intermediate frame feature map and the first remote frame hidden state into the remote frame acquisition submodule to obtain a backward decoding resolution; The first decoding resolution can be determined based on the following formula: In the formula, is the first decoding resolution; DF is the remote frame acquisition submodule; is the first initial remote frame hiding state; Inputting the adjacent frame aligned hidden state, the first decoding resolution, and the intermediate frame feature map into the dynamic selection module to obtain the target backward hidden state; The target backward hidden state is determined based on the following formula: In the formula, is a first target adjacent frame hidden state in the target backward hidden state, wherein the first target adjacent frame hidden state represents a hidden state of a future adjacent intermediate frame; is a first target remote frame hidden state in the target backward hidden state, wherein the first target remote frame hidden state represents a hidden state of a future remote intermediate frame; DS is a dynamic selection module.

4. The video spatiotemporal super-resolution method based on real-time access according to claim 1, characterized in that: The initial forward hidden state includes a second initial adjacent frame hidden state and a second initial remote frame hidden state, wherein the second initial adjacent frame hidden state represents a hidden state from the past to the adjacent frame, and the second initial remote frame hidden state represents a hidden state from the past to the remote frame, and the inputting the intermediate frame feature map, the target backward hidden state, and the initial forward hidden state into the forward loop module to obtain the target forward hidden state includes: Inputting the intermediate feature map, the second initial adjacent frame hidden state, and the first initial adjacent frame hidden state into the forward adjacent frame acquisition submodule to obtain an aligned hidden state; The aligned hidden state is obtained based on the following formula: In the formula, is the first aligned hidden state in the aligned hidden states; is a second aligned hidden state in the aligned hidden states; is the feature map of the middle frame; is the second initial adjacent frame hidden state; is the first initial adjacent frame hidden state; AF is the adjacent frame acquisition submodule; Inputting the intermediate frame feature map, the second initial remote frame hidden state, and the first initial remote frame hidden state into the remote frame acquisition submodule to determine a decoding resolution; The decoding resolution is determined based on the following formula: In the formula, is the first decoding resolution among the decoding resolutions; is a second decoding resolution among the decoding resolutions; is the feature map of the middle frame; is the second initial remote frame hiding state; is the first initial remote frame hiding state; DF is the remote frame acquisition submodule; The target forward hidden state is determined based on the intermediate frame feature map, the aligned hidden state, and the decoding resolution.

5. The video spatiotemporal super-resolution method based on real-time access according to claim 1, characterized in that: Determining the decoding resolution further comprises: Downsampling the first initial remote frame hidden state, the second initial remote frame hidden state, and the intermediate frame feature map; Among them, it can be realized based on the following formula: In the formula, is the second initial remote frame hiding state; is the first initial remote frame hiding state; is the feature map of the middle frame; E stands for encoder, and C stands for channel connection operation; To obtain the down-sampling value; The down-sampling results are weighted summed to obtain the relevant components; Among them, the relevant components can be determined based on the following formula: In the formula, is the second correlation component in the correlation components; is the first correlation component among the correlation components; The relevant components are decoded to obtain the decoding resolution. Specifically, it can be implemented based on the following formula: In the formula, is the first decoding resolution among the decoding resolutions; is a second decoding resolution among the decoding resolutions; D stands for decoder.

6. The video spatiotemporal super-resolution method based on real-time access according to claim 1, characterized in that: The dynamic selection submodule further includes a dynamic selection attention unit and a fusion residual unit, and the determining of the target forward hidden state based on the aligned hidden state and the decoding resolution further includes: Inputting the aligned hidden state and the decoding resolution into the dynamic selection attention unit to obtain an attention map; The attention map and the intermediate frame feature map are processed in combination with the fusion residual unit to obtain the target forward hidden state, wherein the target forward hidden state includes a second target adjacent frame hidden state and a second target remote frame hidden state in the target forward hidden state, the second target adjacent frame hidden state represents the hidden state from the past to the adjacent intermediate frame, and the second target remote frame hidden state represents the hidden state from the past to the remote intermediate frame; Among them, the attention map can be obtained based on the following formula: In the formula, CA stands for cross attention; is the attention map obtained by the feature map of the middle frame relative to the second aligned hidden state; is the attention map obtained by the feature map of the intermediate frame relative to the first aligned hidden state; The attention map obtained by the intermediate frame feature map relative to the second decoding resolution; It is the attention map obtained for the intermediate frame feature map relative to the first decoding resolution.

7. A video spatiotemporal super-resolution device based on real-time access, characterized in that: The device comprises: A first acquisition module, used to acquire input frame image data and a feature map extraction model; An intermediate frame feature map obtaining module, used to obtain an intermediate frame feature map based on the input frame image data and the feature map extraction model; A second acquisition module, used to acquire an initial backward hidden state; A target backward hidden state obtaining module, used for inputting the intermediate frame feature map and the initial backward hidden state into the backward loop module to obtain a target backward hidden state; An updating module, used for updating the target backward hidden state into the access container; A third acquisition module is used to obtain an initial forward hidden state; A target forward hidden state obtaining module, used for inputting the intermediate frame feature map, the target backward hidden state and the initial forward hidden state into the forward loop module to obtain the target forward hidden state; The target video obtaining module is used to input the target forward hidden state and the target backward hidden state into the fusion reconstruction module to obtain the target video.

8. An electronic device, characterized in that: The electronic device includes a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the electronic device executes the various steps of the video spatiotemporal super-resolution method based on real-time access as claimed in claims 1-6.

9. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by a processor, the steps of the video spatiotemporal super-resolution method based on real-time access as described in any one of claims 1 to 6 are implemented.

Images