Event camera driven video super-resolution reconstruction method, system and apparatus

Abstract

The application is directed to the characteristics of traditional frame cameras and new event cameras, and proposes an event camera driven video super-resolution reconstruction method, system and device, belonging to the technical field of video processing.The method of the application proposes an event adaptive synthesis module, which captures and learns long-term motion trajectories by utilizing the spatiotemporal correlation between video frames and event voxels through two steps of event modulation alignment and bidirectional cycle compensation, and facilitates adaptive interpolation and fusion of spatiotemporal features.In addition, the method of the application also proposes a local implicit video Transformer, which combines local implicit video neural functions with cross-scale spatiotemporal attention to learn continuous video representations for generating reliable videos of arbitrary spatial resolution and frame rate.The application can improve the frame rate and resolution of low-quality videos at any time and spatial scale through the driving of event camera data, significantly improving the efficiency, effectiveness and generalizability of the video spatiotemporal super-resolution reconstruction method.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, and in particular to an event camera-driven video super-resolution reconstruction method, system, and apparatus. Background Technology

[0002] Due to high storage costs and bandwidth limitations, most videos are stored or transmitted at relatively low resolutions and frame rates, resulting in a loss of detail. Reconstruction techniques are needed to improve video quality. Video spatiotemporal super-resolution, as a reconstruction technique to improve the frame rate and resolution of low-quality videos, can simultaneously improve both video clarity and smoothness, and has wide applications in film and television production and processing.

[0003] However, current methods generally fix the temporal and spatial reconstruction scales, significantly limiting their practical application. Existing continuous video spatiotemporal super-resolution reconstruction methods are constrained by the decoupled spatiotemporal continuous representation learning paradigm, resulting in poor generalization and efficiency at arbitrary scales. In particular, current reconstruction methods often rely solely on video frames, and due to the limited information provided by the input frames, reconstruction results are often poor when encountering scenes with complex inter-frame motion. Event cameras are novel biomimetic vision sensors that generate high dynamic range and temporal resolution event streams by asynchronously detecting brightness changes in each pixel. Recent methods utilizing events for video processing have shown great potential due to their ability to acquire more latent spatiotemporal information; however, there is currently a lack of event-driven methods for continuous video spatiotemporal super-resolution reconstruction. Summary of the Invention

[0004] Embodiments of the present invention provide an event camera-driven video super-resolution reconstruction method, system, and apparatus to solve technical problems existing in the prior art.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] Event camera-driven video super-resolution reconstruction methods include:

[0007] Based on two consecutive reference frames obtained from low frame rate and low resolution video, and the corresponding low resolution event stream, the low frequency event stream is transformed and the timing and polarity of the events are reversed to obtain a bidirectional event voxel grid.

[0008] Feature extraction operations are performed on two consecutive reference frames and a bidirectional event voxel grid to obtain the initial features of the reference frames, forward event features, and backward event features.

[0009] By using forward event features and backward event features, and then using a pyramid-cascaded deformable convolutional network to perform forward modulation alignment operations on the initial features of the reference frame and backward modulation alignment operations respectively, a potential inter-frame feature sequence is obtained.

[0010] The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain the first frame feature sequence. Based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining event information through bidirectional cyclic compensation operation. The second frame feature sequence is obtained through forward compensation operation and backward compensation operation in this global modeling process.

[0011] By utilizing a cross-scale spatiotemporal attention mechanism, a unified continuous representation of the frame features of the second frame feature sequence is learned in the spatiotemporal domain. Based on any given temporal upsampling factor and spatial upsampling factor, the frame features of the second frame feature sequence are mapped to high-resolution video frames through a local implicit video Transformer.

[0012] Preferably, the process of obtaining a bidirectional event voxel grid by performing transformation operations and reversing the timing and polarity of events on the low-frequency event stream based on two consecutive reference frames obtained from a low-frame-rate, low-resolution video and the corresponding low-resolution event stream includes:

[0013] The low-resolution event stream is converted into a voxel grid of the same size as the reference frame and divided into multiple segments; each voxel grid segment can represent the density information of a small area before and after a certain moment.

[0014] By reversing the event polarity and temporal direction of multiple voxel mesh segments, a bidirectional event voxel mesh is obtained.

[0015] Preferably, the process of performing feature extraction operations on two consecutive reference frames and a bidirectional event voxel grid to obtain the initial features of the reference frames, forward event features, and backward event features includes:

[0016] Two convolutional layers and five residual blocks are used to perform feature extraction operations on two consecutive reference frames and a bidirectional event voxel grid to obtain the initial features of the reference frames, forward event features and backward event features; the two convolutional layers have 5×5 convolutional kernels and 3×3 convolutional kernels with LeakyReLU activation, respectively.

[0017] Preferably, the process of obtaining the potential inter-frame feature sequence by using forward event features and backward event features and performing forward modulation alignment operations and backward modulation alignment operations on the initial features of the reference frame using a pyramid cascaded deformable convolutional network is executed in the event adaptive synthesis module.

[0018] The event adaptive synthesis module includes a first pyramid-cascaded deformable convolutional network, a second pyramid-cascaded deformable convolutional network, and a fusion layer arranged side by side;

[0019] The first pyramid-cascaded deformable convolutional network is used to perform forward multi-scale modulation alignment operation on the initial features of the reference frame using forward event features to obtain the forward inter-frame feature sequence.

[0020] The second pyramid-cascaded deformable convolutional network is used to perform backward multi-scale modulation alignment operation on the initial features of the reference frame using backward event features to obtain the backward inter-frame feature sequence.

[0021] The fusion layer is used to fuse the forward inter-frame feature sequences and the backward inter-frame feature sequences to obtain the potential inter-frame feature sequences.

[0022] Preferably, based on the first frame feature sequence, a global model is performed on the event features and image features in the first frame feature sequence by combining bidirectional cyclic compensation operations and event information; the global modeling performed in the process of obtaining the second frame feature sequence through forward compensation and backward compensation operations in this global modeling process is a bidirectional cyclic network.

[0023] The bidirectional loop network includes a forward loop compensation network, a backward loop compensation network, and an output module; the forward loop network has a first channel attention module and a forward loop block arranged sequentially along the data flow transmission direction, and the backward loop network has a second channel attention module and a backward loop block arranged sequentially along the data flow transmission direction.

[0024] The first channel attention module is used to fuse the frame features at a certain time step and the forward event features at a certain time step in the first frame feature sequence, and input them into the forward loop block for processing; the forward loop block is used to fuse the output of the backward loop block of the previous time step with the output of the first channel attention module at this certain time step.

[0025] The second channel attention module is used to fuse the frame features at a certain time step and the backward event features at a certain time step in the first frame feature sequence, and input them into the backward loop block for processing; the backward loop block is used to fuse the output of the forward loop block at the current time step, the output of the second channel attention module at that time step, and the output of the backward loop block at the next time step.

[0026] The output module is used to perform element-wise addition of the outputs of the forward recurrent compensation network and the backward recurrent compensation network at each time step to obtain the feature sequence of the second frame.

[0027] Preferably, by utilizing a cross-scale spatiotemporal attention mechanism, a unified continuous representation of the frame features of the second frame feature sequence is learned in the spatiotemporal domain. The process of mapping the frame features of the low-resolution second frame feature sequence to high-resolution video frames based on any given temporal upsampling factor and spatial upsampling factor includes:

[0028] Based on a certain temporal upsampling factor and spatial upsampling factor of the original video frame, the query coordinates of the high-resolution grid corresponding to the target frame, the sampling coordinates of the low-resolution grid corresponding to the feature map, and the cell encoding are obtained.

[0029] Based on the query coordinates of the high-resolution grid corresponding to the target frame and the sampling coordinates of the low-resolution grid corresponding to the feature map, the relative coordinates of the local sampling grid are obtained. The relative coordinates of the local sampling grid are then converted into cosine coding to obtain the position offset.

[0030] Perform a 3D convolution operation on the second frame feature sequence to convert the second frame feature sequence into a query feature sequence, a key feature sequence, and a value feature sequence;

[0031] The query feature sequence is upsampled using trilinear upsampling with the query coordinates of the high-resolution grid corresponding to the target frame to obtain the query sample data.

[0032] Based on the sampling coordinates of the low-resolution grid corresponding to the feature map, nearest neighbor local sampling is performed on the Key feature sequence and the Value feature sequence respectively to obtain Key sampling data and Value sampling data;

[0033] Attention operations are performed on the Query sampled data, Key sampled data, Value sampled data, and positional bias. Then, the Query sampled data is used again, and the result of the aforementioned attention operation and the unit encoding are input into the multilayer perceptron for processing to obtain high-resolution video frames.

[0034] In a second aspect, the present invention provides an event camera-driven video super-resolution reconstruction system, comprising:

[0035] The original video frame and event conversion unit is used to: perform conversion operations on the low-frequency event stream and reverse the timing and polarity operations of the events based on two consecutive reference frames obtained from a low-frame-rate, low-resolution video and the corresponding low-resolution event stream, to obtain a bidirectional event voxel grid.

[0036] The frame feature extraction unit is used for:

[0037] Feature extraction operations are performed on two consecutive reference frames and a bidirectional event voxel grid to obtain the initial features of the reference frames, forward event features, and backward event features.

[0038] By using forward event features and backward event features, and then using a pyramid-cascaded deformable convolutional network to perform forward modulation alignment operations on the initial features of the reference frame and backward modulation alignment operations respectively, a potential inter-frame feature sequence is obtained.

[0039] The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain the first frame feature sequence. Based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining event information through bidirectional cyclic compensation operation. The second frame feature sequence is obtained through forward compensation operation and backward compensation operation in this global modeling process.

[0040] The video frame reconstruction unit is used to: learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain using a cross-scale spatiotemporal attention mechanism, and map the frame features of the low-resolution second frame feature sequence to a high-resolution video frame based on any given temporal upsampling factor and spatial upsampling factor.

[0041] Thirdly, the present invention provides an event camera-driven video super-resolution reconstruction apparatus, the apparatus comprising: a processor; and a memory for storing processor-executable instructions;

[0042] The processor implements the above method by calling executable instructions.

[0043] As can be seen from the technical solutions provided by the embodiments of the present invention above, the present invention proposes an event camera-driven video super-resolution reconstruction method, system, and apparatus, belonging to the field of video processing technology, targeting the characteristics of traditional frame cameras and novel event cameras. The method of the present invention proposes an event adaptive synthesis module, which, through two steps of event modulation alignment and bidirectional cyclic compensation, utilizes the spatiotemporal correlation between video frames and event voxels to capture and learn long-term motion trajectories, facilitating adaptive interpolation and fusion of spatiotemporal features. Furthermore, the method of the present invention also proposes a local implicit video Transformer, which combines a local implicit video neural function with cross-scale spatiotemporal attention to learn continuous video representations for generating reliable videos with arbitrary spatial resolution and frame rate. Driven by event camera data, the present invention can improve the frame rate and resolution of low-quality videos at arbitrary temporal and spatial scales, significantly improving the efficiency, effectiveness, and generalization of video spatiotemporal super-resolution reconstruction methods.

[0044] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating the process of the event camera-driven video super-resolution reconstruction method provided by this invention.

[0047] Figure 2 A schematic diagram of the event modulation alignment process of the event camera-driven video super-resolution reconstruction method provided by the present invention;

[0048] Figure 3 A schematic diagram of the bidirectional cyclic compensation process of the event camera-driven video super-resolution reconstruction method provided by the present invention;

[0049] Figure 4 A schematic diagram of the local implicit video Transformer processing procedure of the event camera-driven video super-resolution reconstruction method provided by the present invention;

[0050] Figure 5 The present invention provides a logic block diagram of an event camera-driven video super-resolution reconstruction system. Detailed Implementation

[0051] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0052] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0054] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0055] See Figure 1 This invention provides an event camera-driven video super-resolution reconstruction method for spatiotemporal super-resolution reconstruction of continuous video, comprising the following steps:

[0056] Step 1: Based on two consecutive reference frames obtained from a low frame rate, low resolution video, and the corresponding low resolution event stream, perform transformation operations and reverse the timing and polarity operations of the low frequency event stream to obtain a bidirectional event voxel grid.

[0057] Step 2: Perform feature extraction operations on two consecutive reference frames and a bidirectional event voxel grid to obtain the initial features of the reference frames, forward event features, and backward event features;

[0058] Step 3: Using forward event features and backward event features, and using a pyramid-cascaded deformable convolutional network, perform forward modulation alignment operations and backward modulation alignment operations on the initial features of the reference frame to obtain the potential inter-frame feature sequence.

[0059] Step 4: Combine the initial features of the reference frame with the potential inter-frame feature sequence to obtain the first frame feature sequence; based on the first frame feature sequence, perform global modeling of the event features and image features in the first frame feature sequence through bidirectional cyclic compensation operation and in combination with event information; obtain the second frame feature sequence through forward compensation operation and backward compensation operation in this global modeling process;

[0060] Step 5: Utilize a cross-scale spatiotemporal attention mechanism to learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain. Based on any given temporal upsampling factor and spatial upsampling factor, map the frame features of the second frame feature sequence to high-resolution video frames through a local implicit video Transformer.

[0061] This invention, leveraging the characteristics of a novel event camera, firstly utilizes an event-adaptive synthesis module to fully leverage the high temporal resolution information provided by event voxels. Through two steps—event modulation alignment and bidirectional cyclic compensation—it first obtains the latent inter-frame features aligned to event voxels. Then, it comprehensively utilizes the event features to perform bidirectional adaptive compensation on the frame features, thereby synthesizing accurate frame features in the temporal domain. Finally, through a locally implicit video Transformer, it breaks the existing paradigm of implicit video representation, combining cross-scale spatiotemporal attention to learn a unified video representation in the spatiotemporal domain. It efficiently outputs high-frame-rate, high-resolution video frames at arbitrary temporal and spatial sampling factors as needed, ensuring the effectiveness and generalization of the reconstruction results.

[0062] In a preferred embodiment of the present invention, during step 1, since deep networks cannot directly process the sparse format of the original event data, it is necessary to convert the original event stream into a voxel grid of the same size as the input video frame and divided into several segments. Each event segment represents the density information of a small range before and after a certain moment. Simultaneously, considering that the backward event stream can represent the backward inter-frame motion relationship, the event voxel grid is inverted in terms of event polarity and temporal direction to obtain bidirectional event voxels.

[0063] In step 2, considering that event features contain richer motion information than frame features during the alignment process in step 3, two convolutional layers and five residual blocks are used to extract features from two consecutive reference frames and a bidirectional event voxel grid, respectively, to obtain the initial features of the reference frames, forward event features, and backward event features. The two convolutional layers have 5×5 convolutional kernels with LeakyReLU activation and 3×3 convolutional kernels, respectively.

[0064] In a preferred embodiment of the present invention, an event adaptive synthesis module is used in steps 3 and 4 to perform event modulation alignment and bidirectional cyclic compensation.

[0065] In step 3, given that several event segment features contain motion cues between frames, the reference frame features can be modulated and aligned in both the forward and backward directions to obtain potential inter-frame features. To this end, two pyramid-cascaded deformable convolutional networks are used (forward and backward) to scale the reference frame features and event features to three scales: ×1, ×1 / 2, and ×1 / 4. The forward and backward event segments are then modulated and aligned with the reference frame in both directions to obtain two inter-frame feature sequences. Finally, the inter-frame feature sequences from the two directions are concatenated on the channel and fused through a 1×1 convolutional layer to obtain the output of the event modulation and alignment step.

[0066] In one feasible embodiment, such as Figure 3As shown, the event adaptive synthesis module includes a first pyramid cascaded deformable convolutional network, a second pyramid cascaded deformable convolutional network, and a fusion layer arranged side by side.

[0067] The first pyramid-cascaded deformable convolutional network is used to perform forward multi-scale modulation alignment operation on the initial features of the reference frame using forward event features to obtain the forward inter-frame feature sequence.

[0068] The second pyramid-cascaded deformable convolutional network is used to perform backward multi-scale modulation alignment operation on the initial features of the reference frame using backward event features to obtain the backward inter-frame feature sequence.

[0069] The fusion layer is used to fuse the forward inter-frame feature sequences and the backward inter-frame feature sequences to obtain the potential inter-frame feature sequences.

[0070] In step 4, after combining the potential inter-frame features obtained in step 3 with the reference frame features to form a new frame feature sequence (referred to as the first frame feature sequence in the preferred embodiment of this invention), a bidirectional recurrent network is used to iteratively compensate the frame feature sequence in order to fully model the features of both the event and image modalities globally. In the backward recurrent block of each time step, the corresponding frame features and event features are fused through a channel attention module and then receive the output of the backward recurrent block of the previous time step; while in the forward recurrent block of each time step, the corresponding frame features and event features are also fused through a channel attention module, but while receiving the output of the forward recurrent block of the next time step, they also receive the output of the backward recurrent block corresponding to the current time step. Finally, in order to retain more effective information, the frame features before and after compensation are added element-wise to obtain a refined frame feature sequence, referred to as the second frame feature sequence in the preferred embodiment of this invention.

[0071] For example, in one feasible embodiment, the global modeling performed is a bidirectional cyclic network.

[0072] The bidirectional loop network includes a forward loop compensation network, a backward loop compensation network, and an output module; the forward loop network has a first channel attention module and a forward loop block arranged sequentially along the data flow transmission direction, and the backward loop network has a second channel attention module and a backward loop block arranged sequentially along the data flow transmission direction.

[0073] The first channel attention module is used to fuse the frame features at a certain time step and the forward event features at a certain time step in the first frame feature sequence, and input them into the forward loop block for processing; the forward loop block is used to fuse the output of the backward loop block of the previous time step with the output of the first channel attention module at this certain time step.

[0074] The second channel attention module is used to fuse the frame features at a certain time step and the backward event features at a certain time step in the first frame feature sequence, and input them into the backward loop block for processing; the backward loop block is used to fuse the output of the forward loop block at the current time step, the output of the second channel attention module at that time step, and the output of the backward loop block at the next time step.

[0075] The output module is used to perform element-wise addition of the outputs of the forward recurrent compensation network and the backward recurrent compensation network at each time step to obtain the feature sequence of the second frame.

[0076] In step 5, the query coordinates of the high-resolution grid corresponding to the target frame, the sampling coordinates of the feature map corresponding to the low-resolution grid, and the unit encoding are first obtained based on the input temporal and spatial upsampling factors. To better capture spatiotemporal dependencies, a novel local implicit video Transformer is proposed. The query is obtained from large-scale features on the high-resolution grid after trilinear upsampling, while the key and value are obtained from the local small-scale features of the nearest neighbor to the query on the original low-resolution grid, thus calculating a cross-scale spatiotemporal attention map. The query, key, and value are obtained from independent 3×3×3 three-dimensional convolutional layers, while the positional bias is obtained by cosine encoding the relative coordinates of the query coordinates in the local sampling grid. Finally, to further represent the continuous spatiotemporal domain, the output obtained from the cross-scale spatiotemporal attention mechanism is embedded and concatenated with the query and unit encoding, and after passing through a multilayer perceptron, a high-resolution target frame is output.

[0077] The specific process is as follows: Figure 4 As shown:

[0078] Based on a certain temporal upsampling factor and spatial upsampling factor of the original video frame, the query coordinates of the high-resolution grid corresponding to the target frame, the sampling coordinates of the low-resolution grid corresponding to the feature map, and the cell encoding are obtained respectively.

[0079] Based on the query coordinates of the high-resolution grid corresponding to the target frame and the sampling coordinates of the low-resolution grid corresponding to the feature map, the relative coordinates of the local sampling grid are obtained. The relative coordinates of the local sampling grid are then converted into cosine coding to obtain the position offset.

[0080] Perform a 3D convolution operation on the second frame feature sequence to convert the second frame feature sequence into a query feature sequence, a key feature sequence, and a value feature sequence;

[0081] The query feature sequence is upsampled using trilinear upsampling with the query coordinates of the high-resolution grid corresponding to the target frame to obtain the query sample data.

[0082] Based on the sampling coordinates of the low-resolution grid corresponding to the feature map, nearest neighbor local sampling operations are performed on the Key feature sequence and the Value feature sequence respectively to obtain Key sampling data and Value sampling data;

[0083] One path of the Query sampling data is directly input into the multilayer perceptron, while the other path performs attention operations with the Key sampling data, Value sampling data, and position offset. The result of the attention operation and the unit encoding are then input into the multilayer perceptron for processing to obtain high-resolution video frames.

[0084] Specifically, the 3D convolution operation on the feature sequence of the second frame is as follows:

[0085] First, the sequence is passed through a 3×3×3 convolutional layer, then divided into three paths, each passing through a 3×3×3 convolutional layer, to obtain the Query feature sequence, Key feature sequence, and Value feature sequence respectively.

[0086] The attention calculation process is as follows:

[0087] The query sampled data is first subjected to an inner product operation with the key sampled data. The result of the inner product operation is then subjected to a matrix addition operation with the position offset. After being processed by a Softmax kernel function, it is subjected to a matrix multiplication operation with the value sampled data, and finally input into the multilayer perceptron.

[0088] The multilayer perceptron processes the feature data and unit encoding from the two channels mentioned above, and finally outputs high frame rate and high resolution video frames at any time.

[0089] Figure 4 In the diagram, thick arrows indicate the transmission of coordinate data streams, while small arrows indicate the transmission of feature data streams.

[0090] The present invention also provides an embodiment to exemplarily demonstrate the effects of the present invention, specifically through a comparative experiment.

[0091] (1) Training and testing process

[0092] The experiment used an NVIDIA RTX 3090 GPU and the PyTorch framework to train and test the model. The batch size was set to 2, and the Charbonnier loss function and Adam optimizer were used for optimization. The training process consisted of two phases. In the first phase, the temporal and spatial upsampling factors were set to 8 and 4, respectively, and the training iterations were 450,000. In the second phase, the spatial upsampling factor was set to a uniform distribution of [1, 4], and the training iterations continued for another 150,000 iterations. Every 150,000 iterations, cosine annealing was used to reduce the learning rate from 1×10⁻⁶. -4Decay to 1×10 -7 The experiment designed a baseline model, EvEnhancer (7 event segments, 64 channels), and a lightweight model, EvEnhancer-light (5 event segments, 16 channels), by controlling the number of event segments and the number of channels used in the local implicit representation.

[0093] To better evaluate the spatiotemporal super-resolution reconstruction method for continuous video proposed in this invention, experiments were conducted using three different metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Tera Floating Point Operations (TFLOPs), and the number of model parameters to assess the performance of the reconstruction method.

[0094] Table 1. Classification of different methods, number of model parameters, and quality assessment results of fixed-scale reconstruction on the Adobe 240 test set (time scale × 8, spatial scale × 4).

[0095]

[0096] Table 2. Quality assessment results of different methods for reconstruction at arbitrary scales on the GoPro test set.

[0097]

[0098] Tables 1 and 2 show the PSNR and SSIM values ​​obtained by the proposed method and related methods under fixed-scale Adobe 240 dataset and arbitrary-scale GoPro dataset settings, respectively. Comparing the data in the two tables, it can be seen that the event-camera-driven continuous video spatiotemporal super-resolution reconstruction method proposed in this invention achieves superior performance compared to frame-based or event-based methods due to its lightweight model. In particular, it exhibits outstanding generalization performance under arbitrary-scale settings, fully demonstrating that the event-camera-driven strategy significantly improves the generalization and reliability of video spatiotemporal super-resolution reconstruction methods.

[0099] Table 3. Quality assessment results of different methods reconstructing on the BS-ERGB test set (fixed spatial scale × 4, varying temporal upsampling scale).

[0100]

[0101] Table 4. Quality assessment results of different methods in spatial scale ×2 reconstruction of the ALPIX-VSR test set.

[0102]

[0103] To further verify the effectiveness of the proposed method on real-world event datasets, the BS-ERGB and ALPIX-VSR datasets were selected for experiments. Event data in these datasets were all acquired using real-world event cameras. Tables 3 and 4 show the PSNR and SSIM values ​​obtained by the proposed method and related methods on the BS-ERGB dataset (fixed spatial scale × 4, varying temporal upsampling scale) and the ALPIX-VSR dataset (spatial scale × 2), respectively. It can be seen that the proposed method still demonstrates effectiveness in real-world event data scenarios.

[0104] Furthermore, to evaluate the efficiency improvement of the proposed solution, Table 5 shows the TFLOPs required for reconstruction on the GoPro dataset with a fixed spatial scale of ×4 and varying temporal upsampling scale. Since the MoTIF method consumes a large amount of GPU memory, it cannot be implemented with a single network inference when reconstructing larger temporal scales, requiring multiple inference iterations. In contrast, the proposed solution achieves optimal robustness while maintaining the best balance between computational cost and model parameter count (see Table 1), thus improving the efficiency, effectiveness, and versatility of video spatiotemporal super-resolution reconstruction methods.

[0105] Table 5 shows the TFLOPs required for reconstruction using different methods on the GoPro test set with a fixed spatial scale of ×4 and varying temporal upsampling scale (* indicates that a single inference causes memory overflow and requires repeated inferences).

[0106]

[0107] In a second aspect, the present invention provides an event camera-driven video super-resolution reconstruction system, comprising:

[0108] The original video frame and event conversion unit 501 is used to: based on two consecutive reference frames obtained from a low frame rate and low resolution video, and the corresponding low resolution event stream, perform conversion operations on the low frequency event stream, reverse the timing and polarity operations of the events, and obtain a bidirectional event voxel grid.

[0109] Frame feature extraction unit 502 is used for:

[0110] Feature extraction operations are performed on two consecutive reference frames and a bidirectional event voxel grid to obtain the initial features of the reference frames, forward event features, and backward event features.

[0111] By using forward event features and backward event features, and then using a pyramid-cascaded deformable convolutional network to perform forward modulation alignment operations on the initial features of the reference frame and backward modulation alignment operations respectively, a potential inter-frame feature sequence is obtained.

[0112] The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain the first frame feature sequence. Based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining event information through bidirectional cyclic compensation operation. The second frame feature sequence is obtained through forward compensation operation and backward compensation operation in this global modeling process.

[0113] The video frame reconstruction unit 503 is used to: learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain using a cross-scale spatiotemporal attention mechanism, and map the frame features of the low-resolution second frame feature sequence to a high-resolution video frame based on any given temporal upsampling factor and spatial upsampling factor.

[0114] Thirdly, the present invention provides an event camera-driven video super-resolution reconstruction apparatus, the apparatus comprising: a processor; and a memory for storing processor-executable instructions.

[0115] The processor implements the above method by calling executable instructions.

[0116] In summary, this invention proposes an event-camera-driven method and system for spatiotemporal super-resolution reconstruction of continuous video, belonging to the field of video processing technology, taking into account the characteristics of traditional frame cameras and novel event cameras. The method of this invention proposes an event-adaptive synthesis module, which, through two steps—event modulation alignment and bidirectional cyclic compensation—utilizes the spatiotemporal correlation between video frames and event voxels to capture and learn long-term motion trajectories, facilitating adaptive interpolation and fusion of spatiotemporal features. Furthermore, the method of this invention also proposes a locally implicit video Transformer, combining locally implicit video neural functions with cross-scale spatiotemporal attention to learn continuous video representations for generating reliable videos with arbitrary spatial resolution and frame rate. Driven by event camera data, this invention can improve the frame rate and resolution of low-quality videos at arbitrary temporal and spatial scales, significantly improving the efficiency, effectiveness, and generalization of video spatiotemporal super-resolution reconstruction methods.

[0117] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0118] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0119] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0120] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An event-camera-driven video super-resolution reconstruction method, characterized in that, include: Based on two consecutive reference frames obtained from low frame rate and low resolution video, and the corresponding low resolution event stream, the low frequency event stream is transformed and the timing and polarity of the events are reversed to obtain a bidirectional event voxel grid. Feature extraction operations are performed on two consecutive reference frames and a bidirectional event voxel grid to obtain the initial features of the reference frames, forward event features, and backward event features. By using forward event features and backward event features, and then using a pyramid-cascaded deformable convolutional network to perform forward modulation alignment operations on the initial features of the reference frame and backward modulation alignment operations respectively, a potential inter-frame feature sequence is obtained. The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain the first frame feature sequence. Based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining event information through bidirectional cyclic compensation operation. The second frame feature sequence is obtained through the forward compensation and backward compensation operations in this global modeling process; By utilizing a cross-scale spatiotemporal attention mechanism, a unified continuous representation of the frame features of the second frame feature sequence is learned in the spatiotemporal domain. Based on any given temporal upsampling factor and spatial upsampling factor, the frame features of the second frame feature sequence are mapped to high-resolution video frames through a local implicit video Transformer.

2. The method according to claim 1, characterized in that, The process of obtaining a bidirectional event voxel grid by performing transformation operations, reversing the timing and polarity of events on the low-frequency event stream, and converting two consecutive reference frames and corresponding low-resolution event streams obtained from low-frame-rate, low-resolution videos includes: The low-resolution event stream is converted into a voxel grid of the same size as the reference frame and divided into multiple segments; each voxel grid segment can represent the density information of a small area before and after a certain moment. By reversing the event polarity and temporal direction of multiple voxel mesh segments, a bidirectional event voxel mesh is obtained.

3. The method according to claim 2, characterized in that, The process of performing feature extraction operations on two consecutive reference frames and a bidirectional event voxel grid to obtain initial features of the reference frames, forward event features, and backward event features includes: Two convolutional layers and five residual blocks are used to perform feature extraction operations on two consecutive reference frames and a bidirectional event voxel grid to obtain the initial features of the reference frames, forward event features and backward event features; the two convolutional layers have 5×5 convolutional kernels and 3×3 convolutional kernels with LeakyReLU activation, respectively.

4. The method according to claim 3, characterized in that, The process of obtaining a potential inter-frame feature sequence by using forward event features and backward event features and performing forward modulation alignment operations and backward modulation alignment operations on the initial features of the reference frame using a pyramid cascaded deformable convolutional network is executed in the event adaptive synthesis module. The event adaptive synthesis module includes a first pyramid-cascaded deformable convolutional network, a second pyramid-cascaded deformable convolutional network, and a fusion layer arranged side by side; The first pyramid-cascaded deformable convolutional network is used to perform forward multi-scale modulation alignment operation on the initial features of the reference frame using forward event features to obtain the forward inter-frame feature sequence. The second pyramid-cascaded deformable convolutional network is used to perform backward multi-scale modulation alignment operation on the initial features of the reference frame using backward event features to obtain the backward inter-frame feature sequence. The fusion layer is used to fuse the forward inter-frame feature sequences and the backward inter-frame feature sequences to obtain the potential inter-frame feature sequences.

5. The method according to claim 4, characterized in that, The method described above involves using a bidirectional cyclic compensation operation based on the feature sequence of the first frame, combined with event information, to perform global modeling of the event features and image features in the feature sequence of the first frame. Through the forward compensation and backward compensation operations in this global modeling process, the global modeling performed in obtaining the feature sequence of the second frame is a bidirectional recurrent network. The bidirectional cyclic network includes a forward cyclic compensation network, a backward cyclic compensation network, and an output module; The forward loop network has a first channel attention module and a forward loop block arranged sequentially along the data flow transmission direction, and the backward loop network has a second channel attention module and a backward loop block arranged sequentially along the data flow transmission direction; The first channel attention module is used to fuse the frame features at a certain time step and the forward event features at a certain time step in the first frame feature sequence, and input them into the forward loop block for processing; the forward loop block is used to fuse the output of the backward loop block of the previous time step with the output of the first channel attention module at this certain time step. The second channel attention module is used to fuse the frame features at a certain time step and the backward event features at a certain time step in the first frame feature sequence, and input them into the backward loop block for processing. The backward loop block is used to fuse the output of the forward loop block at the current time step, the output of the second channel attention module at that time step, and the output of the backward loop block at the next time step. The output module is used to perform element-wise addition of the outputs of the forward recurrent compensation network and the backward recurrent compensation network at each time step to obtain the feature sequence of the second frame.

6. The method according to claim 5, characterized in that, The process of learning a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain using a cross-scale spatiotemporal attention mechanism, and mapping the frame features of the low-resolution second frame feature sequence to a high-resolution video frame based on any given temporal upsampling factor and spatial upsampling factor, includes: Based on a certain temporal upsampling factor and spatial upsampling factor of the original video frame, the query coordinates of the high-resolution grid corresponding to the target frame, the sampling coordinates of the low-resolution grid corresponding to the feature map, and the cell encoding are obtained. Based on the query coordinates of the high-resolution grid corresponding to the target frame and the sampling coordinates of the low-resolution grid corresponding to the feature map, the relative coordinates of the local sampling grid are obtained. The relative coordinates of the local sampling grid are then converted into cosine coding to obtain the position offset. Perform a 3D convolution operation on the second frame feature sequence to convert the second frame feature sequence into a query feature sequence, a key feature sequence, and a value feature sequence; The query feature sequence is upsampled using trilinear upsampling with the query coordinates of the high-resolution grid corresponding to the target frame to obtain the query sample data. Based on the sampling coordinates of the low-resolution grid corresponding to the feature map, nearest neighbor local sampling is performed on the Key feature sequence and the Value feature sequence respectively to obtain Key sampling data and Value sampling data; Attention operations are performed on the Query sampled data, Key sampled data, Value sampled data, and positional bias. Then, the Query sampled data is used again, and the result of the aforementioned attention operation and the unit encoding are input into the multilayer perceptron for processing to obtain high-resolution video frames.

7. An event camera-driven video super-resolution reconstruction system, characterized in that, include: The original video frame and event conversion unit is used to: perform conversion operations on the low-frequency event stream and reverse the timing and polarity operations of the events based on two consecutive reference frames obtained from a low-frame-rate, low-resolution video and the corresponding low-resolution event stream, to obtain a bidirectional event voxel grid. The frame feature extraction unit is used for: Feature extraction operations are performed on two consecutive reference frames and a bidirectional event voxel grid to obtain the initial features of the reference frames, forward event features, and backward event features. By using forward event features and backward event features, and then using a pyramid-cascaded deformable convolutional network to perform forward modulation alignment operations on the initial features of the reference frame and backward modulation alignment operations respectively, a potential inter-frame feature sequence is obtained. The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain the first frame feature sequence. Based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining event information through bidirectional cyclic compensation operation. The second frame feature sequence is obtained through the forward compensation and backward compensation operations in this global modeling process; The video frame reconstruction unit is used to: learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain using a cross-scale spatiotemporal attention mechanism, and map the frame features of the low-resolution second frame feature sequence to a high-resolution video frame based on any given temporal upsampling factor and spatial upsampling factor.

8. An event camera-driven video super-resolution reconstruction device, characterized in that, The device includes: a processor; and a memory for storing processor-executable instructions; The processor implements the method as described in any one of claims 1 to 6 by invoking the executable instructions.

Images