Event camera driven video super-resolution reconstruction method, system and device

Abstract

The invention provides a video super-resolution reconstruction method, system and device driven by an event camera aiming at the characteristics of a traditional frame camera and a novel event camera, and belongs to the technical field of video processing. According to the method, an event adaptive synthesis module is provided, through two steps of event modulation alignment and bidirectional cyclic compensation, a long-term motion track is captured and learned by using the spatial-temporal correlation between a video frame and an event voxel, and adaptive interpolation and fusion of spatial-temporal characteristics are facilitated. In addition, the invention also provides a local implicit video Transformer, which combines a local implicit video neural function with cross-scale space-time attention, learns continuous video representation, and is used for generating a reliable video with any spatial resolution and frame rate. According to the method, through driving of event camera data, the frame rate and the resolution of the low-quality video can be improved at any time and space scale, and the efficiency, the effectiveness and the generalization of the video space-time super-resolution reconstruction method are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to an event camera-driven video super-resolution reconstruction method, system and device. Background Art

[0002] Due to high storage costs and bandwidth limitations, most videos are stored or transmitted at relatively low resolutions and frame rates, resulting in loss of detailed information. This requires the use of reconstruction techniques to improve video quality. Video spatiotemporal super-resolution, a reconstruction technique that increases the frame rate and resolution of low-quality videos, can simultaneously improve both clarity and smoothness, and has a wide range of applications in film and television production and processing.

[0003] However, current related methods generally fix the temporal and spatial reconstruction scales, which greatly limits the practical application of related methods; and existing continuous video spatiotemporal super-resolution reconstruction methods are limited by the paradigm of decoupling spatiotemporal continuous representation learning, and have poor generalization and efficiency at arbitrary scales. In particular, current reconstruction methods are often based only on video frames. Due to the limited information provided by the input video frames, the reconstruction effect is often poor when encountering scenes with complex inter-frame motion. Event camera is a new type of bionic visual sensor that can generate event streams with high dynamic range and temporal resolution by asynchronously detecting the brightness changes of each pixel. Recent methods of using events for video processing have shown great potential due to their ability to obtain more potential spatiotemporal information, but there is currently a lack of related methods for achieving continuous video spatiotemporal super-resolution reconstruction through event-driven means. Summary of the Invention

[0004] Embodiments of the present invention provide an event camera-driven video super-resolution reconstruction method, system, and device, which are used to solve technical problems existing in the prior art.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0006] Event camera-driven video super-resolution reconstruction method, including:

[0007] Based on two consecutive reference frames obtained from a low-frame-rate, low-resolution video and the corresponding low-resolution event stream, the low-resolution event stream is transformed and the timing and polarity of the events are reversed to obtain a bidirectional event voxel grid.

[0008] Perform feature extraction operations on two consecutive reference frames and bidirectional event voxel grids respectively to obtain reference frame initial features, forward event features, and backward event features;

[0009] Through the forward event features and backward event features, the pyramid cascade deformable convolutional network is used to perform forward modulation alignment and backward modulation alignment operations on the initial features of the reference frame to obtain the potential inter-frame feature sequence;

[0010] The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain a first frame feature sequence; based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining the event information through a bidirectional cyclic compensation operation; the second frame feature sequence is obtained through a forward compensation operation and a backward compensation operation in the global modeling process;

[0011] Utilizing the cross-scale spatiotemporal attention mechanism, a unified continuous representation of the frame features of the second frame feature sequence is learned in the spatiotemporal domain. Based on any given temporal upsampling factor and spatial upsampling factor, the frame features of the second frame feature sequence are mapped to high-resolution video frames through a local implicit video Transformer.

[0012] Preferably, based on two consecutive reference frames obtained from a low-frame-rate, low-resolution video and the corresponding low-resolution event stream, a process of converting the low-resolution event stream and reversing the timing and polarity of the events to obtain a bidirectional event voxel grid includes:

[0013] Convert the low-resolution event stream into a voxel grid of the same size as the reference frame and divided into multiple segments; each segment of the voxel grid can represent the density information of a small range before and after a certain moment;

[0014] The multi-segment voxel grid is inverted in event polarity and time domain direction to obtain a bidirectional event voxel grid.

[0015] Preferably, the process of performing feature extraction operations on two consecutive reference frames and bidirectional event voxel grids to obtain reference frame initial features, forward event features, and backward event features includes:

[0016] Two convolutional layers and five residual blocks are used to perform feature extraction operations on two consecutive reference frames and bidirectional event voxel grids to obtain the initial features of the reference frames, forward event features, and backward event features; the two convolutional layers have a 5×5 convolution kernel and a 3×3 convolution kernel with LeakyReLU activation, respectively.

[0017] Preferably, the process of obtaining the potential inter-frame feature sequence by using the forward event features and the backward event features and performing the forward modulation alignment operation and the backward modulation alignment operation on the initial features of the reference frame using a pyramid cascade deformable convolutional network is performed in the event adaptive synthesis module;

[0018] The event adaptive synthesis module includes a first pyramid cascade deformable convolutional network, a second pyramid cascade deformable convolutional network and a fusion layer that are parallel to each other;

[0019] The first pyramid cascade deformable convolutional network is used to perform a forward multi-scale modulation alignment operation on the initial features of the reference frame using the forward event features to obtain a forward inter-frame feature sequence;

[0020] The second pyramid cascade deformable convolutional network is used to perform a backward multi-scale modulation alignment operation on the initial features of the reference frame using the backward event features to obtain a backward inter-frame feature sequence;

[0021] The fusion layer is used to fuse the forward inter-frame feature sequence and the backward inter-frame feature sequence to obtain the latent inter-frame feature sequence.

[0022] Preferably, based on the first frame feature sequence, a bidirectional cyclic compensation operation is performed in combination with event information to perform global modeling on the event features and image features in the first frame feature sequence; in the process of obtaining the second frame feature sequence through the forward compensation operation and the backward compensation operation in the global modeling process, the global modeling performed is a bidirectional cyclic network;

[0023] The bidirectional recurrent network includes a forward recurrent compensation network, a backward recurrent compensation network, and an output module; the forward recurrent network has a first channel attention module and a forward recurrent block arranged in sequence along the data stream transmission direction, and the backward recurrent network has a second channel attention module and a backward recurrent block arranged in sequence along the data stream transmission direction;

[0024] The first channel attention module is used to fuse the frame features of a certain time step in the first frame feature sequence and the forward event features of a certain time step, and input them into the forward loop block for processing; the forward loop block is used to fuse the output of the backward loop block of the previous time step with the output of the first channel attention module of the certain time step;

[0025] The second channel attention module is used to fuse the frame features of a certain time step in the first frame feature sequence and the backward event features of a certain time step, and input them into the backward loop block for processing; the backward loop block is used to fuse the output of the forward loop block of the current time step, the output of the second channel attention module of the certain time step, and the output of the backward loop block of the next time step;

[0026] The output module is used to perform element-by-element addition operation on the output of the forward loop compensation network at each time step and the output of the backward loop compensation network at each time step to obtain the second frame feature sequence.

[0027] Preferably, a cross-scale spatiotemporal attention mechanism is used to learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain, and a process of mapping the frame features of the low-resolution second frame feature sequence to a high-resolution video frame based on any given temporal upsampling factor and spatial upsampling factor includes:

[0028] Based on a certain temporal upsampling factor and a spatial upsampling factor of the original video frame, the query coordinates of the high-resolution grid corresponding to the target frame, the sampling coordinates and unit codes of the low-resolution grid corresponding to the feature map are obtained;

[0029] Based on the query coordinates of the high-resolution grid corresponding to the target frame and the sampling coordinates of the low-resolution grid corresponding to the feature map, the relative coordinates of the local sampling grid are obtained, and the relative coordinates of the local sampling grid are converted into cosine code to obtain a position offset;

[0030] Perform a three-dimensional convolution operation on the feature sequence of the second frame to convert the feature sequence of the second frame into a query feature sequence, a key feature sequence, and a value feature sequence;

[0031] Perform trilinear upsampling on the query feature sequence through the query coordinates of the high-resolution grid corresponding to the target frame to obtain the query sampling data;

[0032] Based on the sampling coordinates of the low-resolution grid corresponding to the feature map, the nearest neighbor local sampling is performed on the Key feature sequence and the Value feature sequence to obtain the Key sampling data and the Value sampling data;

[0033] Attention operation is performed on the query sampling data, key sampling data, value sampling data and position bias, and then the query sampling data is used again and input into the multi-layer perceptron together with the result of the above attention operation and the unit code for processing to obtain a high-resolution video frame.

[0034] In a second aspect, the present invention provides an event camera-driven video super-resolution reconstruction system, comprising:

[0035] The original video frame and event conversion unit is used to convert the low-resolution event stream based on two consecutive reference frames obtained from the low-frame-rate and low-resolution video and the corresponding low-resolution event stream, and invert the timing and polarity of the events to obtain a bidirectional event voxel grid;

[0036] Frame feature extraction unit, used to:

[0037] Perform feature extraction operations on two consecutive reference frames and bidirectional event voxel grids respectively to obtain reference frame initial features, forward event features, and backward event features;

[0038] Through the forward event features and backward event features, the pyramid cascade deformable convolutional network is used to perform forward modulation alignment and backward modulation alignment operations on the initial features of the reference frame to obtain the potential inter-frame feature sequence;

[0039] The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain a first frame feature sequence; based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining the event information through a bidirectional cyclic compensation operation; the second frame feature sequence is obtained through a forward compensation operation and a backward compensation operation in the global modeling process;

[0040] The video frame reconstruction unit is used to: utilize a cross-scale spatiotemporal attention mechanism to learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain, and map the frame features of the low-resolution second frame feature sequence to a high-resolution video frame based on any given temporal upsampling factor and spatial upsampling factor.

[0041] In a third aspect, the present invention provides an event camera driven video super-resolution reconstruction device, the device comprising: a processor; a memory for storing processor executable instructions;

[0042] The processor implements the above method by calling executable instructions.

[0043] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the present invention proposes an event camera-driven video super-resolution reconstruction method, system and device based on the characteristics of traditional frame cameras and new event cameras, which belongs to the field of video processing technology. The method of the present invention proposes an event adaptive synthesis module, which uses the spatiotemporal correlation between video frames and event voxels to capture and learn long-term motion trajectories through two steps of event modulation alignment and bidirectional cyclic compensation, and facilitates the adaptive interpolation and fusion of spatiotemporal features. In addition, the method of the present invention also proposes a local implicit video Transformer, which combines local implicit video neural functions with cross-scale spatiotemporal attention to learn continuous video representations for generating reliable videos of arbitrary spatial resolution and frame rate. Driven by event camera data, the present invention can improve the frame rate and resolution of low-quality videos at any time and space scale, significantly improving the efficiency, effectiveness and generalization of the video spatiotemporal super-resolution reconstruction method.

[0044] Additional aspects and advantages of the present invention will be set forth in part in the following description, will become apparent from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A processing flow chart of the event camera-driven video super-resolution reconstruction method provided by the present invention;

[0047] Figure 2 Schematic diagram of the event modulation alignment process of the event camera driven video super-resolution reconstruction method provided by the present invention;

[0048] Figure 3 A schematic diagram of a bidirectional cyclic compensation process of the event camera-driven video super-resolution reconstruction method provided by the present invention;

[0049] Figure 4 A schematic diagram of the processing process of the local implicit video Transformer of the event camera driven video super-resolution reconstruction method provided by the present invention;

[0050] Figure 5 This is a logic block diagram of an event camera-driven video super-resolution reconstruction system provided by the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0052] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.

[0054] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0055] See also Figure 1 The present invention provides an event camera driven video super-resolution reconstruction method for reconstructing continuous video spatiotemporal super-resolution, comprising the following steps:

[0056] Step 1: Based on two consecutive reference frames obtained from a low-frame-rate, low-resolution video and the corresponding low-resolution event stream, the low-resolution event stream is transformed and the timing and polarity of the events are reversed to obtain a bidirectional event voxel grid;

[0057] Step 2: Perform feature extraction operations on two consecutive reference frames and bidirectional event voxel grids to obtain reference frame initial features, forward event features, and backward event features;

[0058] Step 3: Using the forward event features and the backward event features, the pyramid cascade deformable convolutional network is used to perform forward modulation alignment and backward modulation alignment operations on the initial features of the reference frame to obtain the potential inter-frame feature sequence;

[0059] Step 4: Combine the initial features of the reference frame with the potential inter-frame feature sequence to obtain a first frame feature sequence; based on the first frame feature sequence, perform a bidirectional cyclic compensation operation and combine event information to perform global modeling on the event features and image features in the first frame feature sequence; obtain the second frame feature sequence through forward compensation and backward compensation operations in this global modeling process;

[0060] Step 5: Utilize the cross-scale spatiotemporal attention mechanism to learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain. Based on any given temporal upsampling factor and spatial upsampling factor, the frame features of the second frame feature sequence are mapped to high-resolution video frames through a local implicit video Transformer.

[0061] This invention, incorporating the characteristics of a novel event camera, first utilizes the high temporal resolution information provided by event voxels through an event adaptive synthesis module. Through two steps, event modulation alignment and bidirectional cyclic compensation, it first obtains the potential inter-frame features of event voxel alignment. Then, it comprehensively utilizes event features to perform bidirectional adaptive compensation on frame features, thereby synthesizing accurate frame features in the time domain. Finally, through a local implicit video Transformer, it breaks the existing paradigm of implicit video representation and combines cross-scale spatiotemporal attention to learn a unified video representation in the spatiotemporal domain. This allows for efficient output of high-frame-rate, high-resolution video frames based on any desired temporal and spatial upsampling factors, ensuring the effectiveness and generalizability of the reconstruction.

[0062] In the preferred embodiment of the present invention, during step 1, because deep networks cannot directly process the sparse format of the raw event data, the raw event stream is converted into a voxel grid of the same size as the input video frame and divided into several segments. Each event segment represents the density information of a small range before and after a certain moment. Furthermore, considering that the backward event stream can represent the backward inter-frame motion relationship, the event voxel grid is inverted in terms of event polarity and temporal direction to obtain bidirectional event voxels.

[0063] In step 2, considering that event features contain richer motion information than frame features during the alignment process in step 3, two convolutional layers and five residual blocks are used to extract features from two consecutive reference frames and the bidirectional event voxel grid, respectively, to obtain the reference frame initial features, forward event features, and backward event features. The two convolutional layers have a 5×5 convolution kernel and a 3×3 convolution kernel with LeakyReLU activation, respectively.

[0064] In a preferred embodiment of the present invention, an event adaptive synthesis module is used in steps 3 and 4 to perform event modulation alignment and bidirectional loop compensation.

[0065] In step 3, given that several event segmentation features contain motion cues across multiple frames, modulation alignment can be performed on the reference frame features in both the forward and backward directions to obtain potential inter-frame features. To this end, two pyramid cascaded deformable convolutional networks (FCNs) are used, one in each direction, to scale the reference frame features and event features to ×1, ×1 / 2, and ×1 / 4. The forward event segmentation and the backward event segmentation perform multi-scale modulation alignment on the reference frame in both directions, yielding two inter-frame feature sequences, one in each direction. Finally, the inter-frame feature sequences from both directions are concatenated across channels and fused through a 1×1 convolutional layer to yield the output of the event modulation alignment step.

[0066] In one possible embodiment, Figure 3As shown, the event adaptive synthesis module includes a first pyramid cascade deformable convolutional network, a second pyramid cascade deformable convolutional network and a fusion layer that are parallel to each other;

[0067] The first pyramid cascade deformable convolutional network is used to perform a forward multi-scale modulation alignment operation on the initial features of the reference frame using the forward event features to obtain a forward inter-frame feature sequence;

[0068] The second pyramid cascade deformable convolutional network is used to perform a backward multi-scale modulation alignment operation on the initial features of the reference frame using the backward event features to obtain a backward inter-frame feature sequence;

[0069] The fusion layer is used to fuse the forward inter-frame feature sequence and the backward inter-frame feature sequence to obtain the latent inter-frame feature sequence.

[0070] In step 4, after the potential inter-frame features obtained in step 3 above are combined with the reference frame features to form a new frame feature sequence (in the preferred embodiment provided by the present invention, it is referred to as the first frame feature sequence), in order to fully globally model the features of the two modalities of event and image, a bidirectional recurrent network is used to iteratively compensate the frame feature sequence. In the backward recurrent block of each time step, the corresponding frame features and event features are fused through a channel attention module and then receive the output of the backward recurrent block of the previous time step; and in the forward recurrent block of each time step, the corresponding frame features and event features are also fused through a channel attention module, but while receiving the output of the forward recurrent block of the next time step, the output of the backward recurrent block corresponding to the current time step is received. Finally, in order to retain more effective information, the frame features before and after compensation are added element by element to obtain a refined frame feature sequence, which in the preferred embodiment provided by the present invention is referred to as the second frame feature sequence.

[0071] For example, in one feasible embodiment, the global modeling is performed as a bidirectional recurrent network.

[0072] The bidirectional recurrent network includes a forward recurrent compensation network, a backward recurrent compensation network, and an output module; the forward recurrent network has a first channel attention module and a forward recurrent block arranged in sequence along the data stream transmission direction, and the backward recurrent network has a second channel attention module and a backward recurrent block arranged in sequence along the data stream transmission direction;

[0073] The first channel attention module is used to fuse the frame features of a certain time step in the first frame feature sequence and the forward event features of a certain time step, and input them into the forward loop block for processing; the forward loop block is used to fuse the output of the backward loop block of the previous time step with the output of the first channel attention module of the certain time step;

[0074] The second channel attention module is used to fuse the frame features of a certain time step in the first frame feature sequence and the backward event features of a certain time step, and input them into the backward loop block for processing; the backward loop block is used to fuse the output of the forward loop block of the current time step, the output of the second channel attention module of the certain time step, and the output of the backward loop block of the next time step;

[0075] The output module is used to perform element-by-element addition operation on the output of the forward loop compensation network at each time step and the output of the backward loop compensation network at each time step to obtain the second frame feature sequence.

[0076] In step 5, the query coordinates of the target frame corresponding to the high-resolution grid, the sampling coordinates of the feature map corresponding to the low-resolution grid, and the unit encoding are first obtained based on the input temporal and spatial upsampling factors. To better capture spatiotemporal dependencies, a new local implicit video Transformer is proposed. The query is obtained from the large-scale features on the high-resolution grid through trilinear upsampling, while the key and value are obtained from the local small-scale features of the query's nearest neighbors on the original low-resolution grid, thereby computing a cross-scale spatiotemporal attention map. The query, key, and value are obtained from independent 3×3×3 three-dimensional convolutional layers, while the position offset is obtained by cosine encoding the relative coordinates of the query coordinates in the local sampling grid. Finally, to further represent the continuous spatiotemporal domain, the output embedding obtained from the cross-scale spatiotemporal attention mechanism is concatenated with the query Qurey and unit encoding, and then passed through a multi-layer perceptron to output the high-resolution target frame.

[0077] The specific process is as follows Figure 4 As shown:

[0078] Based on a certain temporal upsampling factor and a spatial upsampling factor of the original video frame, the query coordinates of the high-resolution grid corresponding to the target frame and the sampling coordinates and unit codes of the low-resolution grid corresponding to the feature map are obtained respectively;

[0079] Based on the query coordinates of the high-resolution grid corresponding to the target frame and the sampling coordinates of the low-resolution grid corresponding to the feature map, the relative coordinates of the local sampling grid are obtained, and the relative coordinates of the local sampling grid are converted into cosine coding to obtain the position offset;

[0080] Perform a three-dimensional convolution operation on the feature sequence of the second frame to convert the feature sequence of the second frame into a query feature sequence, a key feature sequence, and a value feature sequence;

[0081] Perform trilinear upsampling on the query feature sequence through the query coordinates of the high-resolution grid corresponding to the target frame to obtain the query sampling data;

[0082] Based on the sampling coordinates of the low-resolution grid corresponding to the feature map, the nearest neighbor local sampling operation is performed on the Key feature sequence and the Value feature sequence to obtain the Key sampling data and the Value sampling data;

[0083] One path of query sampling data is directly input into the multi-layer perceptron, and the other path is used for attention operation with the key sampling data, value sampling data and position bias. The result of the attention operation and the unit code are then input into the multi-layer perceptron for processing to obtain high-resolution video frames.

[0084] The three-dimensional convolution operation on the feature sequence of the second frame is specifically as follows:

[0085] It first passes through a 3×3×3 convolutional layer, and then is divided into three paths, each of which passes through a 3×3×3 convolutional layer to obtain the Query feature sequence, Key feature sequence, and Value feature sequence respectively.

[0086] The specific process of attention operation is:

[0087] The query sampling data is first inner-producted with the key sampling data. The result of the inner-product operation is matrix-added together with the position bias. After being processed by a Softmax kernel function, it is matrix-multiplied with the value sampling data and finally input into the multi-layer perceptron.

[0088] The multi-layer perceptron processes the above two feature data and unit codes, and finally outputs high-frame-rate and high-resolution video frames at any time.

[0089] Figure 4 In the figure, the thick arrows represent the flow of coordinate data transmission, and the small arrows represent the flow of feature data transmission.

[0090] The present invention also provides an embodiment for exemplarily showing the effect of the present invention, which is specifically completed through a comparative experiment.

[0091] (1) Training and testing process

[0092] The experiment used an NVIDIA RTX 3090 GPU and the Pytorch framework to train and test the model. The batch size was set to 2, and the Charbonnier loss function and Adam optimizer were used for optimization. The training process was divided into two phases. In the first phase, the temporal and spatial upsampling factors were set to 8 and 4, respectively, and the training was repeated for 450,000 times. In the second phase, the spatial upsampling factor was set to a uniform distribution of [1, 4], and the training was continued for 150,000 times. Every 150,000 iterations, the learning rate was increased from 1×10 -4Decay to 1×10 -7 By controlling the number of event segments and the number of channels used in the local implicit representation, the experiment designed a baseline model EvEnhancer (number of event segments 7, number of channels 64) and a lightweight model EvEnhancer-light (number of event segments 5, number of channels 16).

[0093] In order to better evaluate the spatiotemporal super-resolution reconstruction method for continuous videos proposed in this invention, the experiment evaluates the performance of the reconstruction method through three different indicators: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Tera Floating Point Operations (TFLOPs), and the number of model parameters.

[0094] Table 1 Classification, model parameter counts, and quality evaluation results of fixed-scale reconstruction of the Adobe240 test set (time scale × 8, spatial scale × 4) for different methods

[0095]

[0096] Table 2 Quality evaluation results of different methods for arbitrary scale reconstruction of the GoPro test set

[0097]

[0098] Tables 1 and 2 show the PSNR and SSIM values ​​of the proposed method and related methods, respectively, tested on the fixed-scale Adobe240 dataset and the arbitrary-scale GoPro dataset. Comparing the data in the two tables, we can see that the proposed event-camera-driven spatiotemporal super-resolution reconstruction method for continuous video, leveraging its lightweight model, achieves leading performance compared to frame-based or event-based methods. In particular, it demonstrates outstanding generalization performance at arbitrary scales, fully demonstrating that the event-camera-driven strategy significantly improves the generalization and reliability of spatiotemporal super-resolution reconstruction methods for video.

[0099] Table 3 Quality evaluation results of different methods on the BS-ERGB test set reconstruction (fixed spatial scale × 4, changing temporal upsampling scale)

[0100]

[0101] Table 4 Quality evaluation results of different methods in the ALPIX-VSR test set spatial scale ×2 reconstruction

[0102]

[0103] To further validate the effectiveness of our proposed approach on real-world event datasets, we conducted experiments on the BS-ERGB and ALPIX-VSR datasets. The event data in these datasets was captured using real-world event cameras. Tables 3 and 4 show the PSNR and SSIM values ​​obtained by our proposed approach and related methods on the BS-ERGB dataset (fixed spatial scale × 4, variable temporal upsampling scale) and the ALPIX-VSR dataset (spatial scale × 2), respectively. This demonstrates the effectiveness of our proposed approach even in real-world event data scenarios.

[0104] Furthermore, to evaluate the efficiency improvements achieved by the proposed solution, Table 5 shows the TFLOPs required for reconstruction of the GoPro dataset using a fixed spatial scale of ×4 and varying temporal upsampling scales. Because the MoTIF method consumes significant video memory, reconstruction of larger temporal scales cannot be achieved with a single network inference, requiring multiple iterations. The proposed solution, however, achieves optimal robustness while striking an optimal balance between computational effort and the number of model parameters (see Table 1), improving the efficiency, effectiveness, and versatility of video spatiotemporal super-resolution reconstruction methods.

[0105] Table 5 TFLOPs required for reconstruction of the GoPro test set by different methods with a fixed spatial scale of ×4 and a changed temporal upsampling scale (* indicates that one inference caused a memory overflow and required repeated inferences)

[0106]

[0107] In a second aspect, the present invention provides an event camera-driven video super-resolution reconstruction system, comprising:

[0108] The original video frame and event conversion unit 501 is configured to: based on two consecutive reference frames obtained from a low-frame-rate, low-resolution video and the corresponding low-resolution event stream, perform a conversion operation on the low-resolution event stream, invert the timing and polarity of the events, and obtain a bidirectional event voxel grid;

[0109] The frame feature extraction unit 502 is configured to:

[0110] Perform feature extraction operations on two consecutive reference frames and bidirectional event voxel grids respectively to obtain reference frame initial features, forward event features, and backward event features;

[0111] Through the forward event features and backward event features, the pyramid cascade deformable convolutional network is used to perform forward modulation alignment and backward modulation alignment operations on the initial features of the reference frame to obtain the potential inter-frame feature sequence;

[0112] The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain a first frame feature sequence; based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining the event information through a bidirectional cyclic compensation operation; the second frame feature sequence is obtained through a forward compensation operation and a backward compensation operation in the global modeling process;

[0113] The video frame reconstruction unit 503 is used to: utilize a cross-scale spatiotemporal attention mechanism to learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain, and map the low-resolution frame features of the second frame feature sequence to a high-resolution video frame based on any given temporal upsampling factor and spatial upsampling factor.

[0114] In a third aspect, the present invention provides an event camera-driven video super-resolution reconstruction device, which includes: a processor; and a memory for storing processor-executable instructions.

[0115] The processor implements the above method by calling executable instructions.

[0116] In summary, the present invention proposes a method and system for spatiotemporal super-resolution reconstruction of continuous videos driven by event cameras, based on the characteristics of traditional frame cameras and new event cameras, and belongs to the field of video processing technology. The method of the present invention proposes an event adaptive synthesis module, which uses the spatiotemporal correlation between video frames and event voxels to capture and learn long-term motion trajectories through two steps of event modulation alignment and bidirectional cyclic compensation, and facilitates the adaptive interpolation and fusion of spatiotemporal features. In addition, the method of the present invention also proposes a local implicit video Transformer, which combines local implicit video neural functions with cross-scale spatiotemporal attention to learn continuous video representations for generating reliable videos of arbitrary spatial resolution and frame rate. Driven by event camera data, the present invention can improve the frame rate and resolution of low-quality videos at any time and space scale, significantly improving the efficiency, effectiveness and generalization of the video spatiotemporal super-resolution reconstruction method.

[0117] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0118] From the above description of the embodiments, it can be seen that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0119] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0120] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An event camera driven video super-resolution reconstruction method, characterized in that: include: Based on two consecutive reference frames obtained from a low-frame-rate, low-resolution video and the corresponding low-resolution event stream, the low-resolution event stream is transformed and the timing and polarity of the events are reversed to obtain a bidirectional event voxel grid. Perform feature extraction operations on two consecutive reference frames and bidirectional event voxel grids respectively to obtain reference frame initial features, forward event features, and backward event features; Through the forward event features and backward event features, the pyramid cascade deformable convolutional network is used to perform forward modulation alignment and backward modulation alignment operations on the initial features of the reference frame to obtain the potential inter-frame feature sequence; The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain the first frame feature sequence; based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining the event information through a bidirectional cyclic compensation operation; Obtaining a second frame feature sequence through forward compensation operation and backward compensation operation in the global modeling process; Utilizing the cross-scale spatiotemporal attention mechanism, a unified continuous representation of the frame features of the second frame feature sequence is learned in the spatiotemporal domain. Based on any given temporal upsampling factor and spatial upsampling factor, the frame features of the second frame feature sequence are mapped to high-resolution video frames through a local implicit video Transformer.

2. The method according to claim 1, characterized in that The process of converting the low-resolution event stream and reversing the timing and polarity of the events based on two consecutive reference frames obtained from the low-frame-rate, low-resolution video and the corresponding low-resolution event stream to obtain a bidirectional event voxel grid includes: Convert the low-resolution event stream into a voxel grid of the same size as the reference frame and divided into multiple segments; each segment of the voxel grid can represent the density information of a small range before and after a certain moment; The multi-segment voxel grid is inverted in event polarity and time domain direction to obtain a bidirectional event voxel grid.

3. The method according to claim 2, characterized in that The process of performing feature extraction operations on two consecutive reference frames and bidirectional event voxel grids to obtain reference frame initial features, forward event features, and backward event features includes: Two convolutional layers and five residual blocks are used to perform feature extraction operations on two consecutive reference frames and bidirectional event voxel grids to obtain the initial features of the reference frames, forward event features, and backward event features; the two convolutional layers have a 5×5 convolution kernel and a 3×3 convolution kernel with LeakyReLU activation, respectively.

4. The method according to claim 3, characterized in that The process of performing forward modulation alignment operation and backward modulation alignment operation on the initial features of the reference frame using the forward event features and the backward event features and using the pyramid cascade deformable convolutional network to obtain the potential inter-frame feature sequence is executed in the event adaptive synthesis module; The event adaptive synthesis module includes a first pyramid cascade deformable convolutional network, a second pyramid cascade deformable convolutional network and a fusion layer that are parallel to each other; The first pyramid cascade deformable convolutional network is used to perform a forward multi-scale modulation alignment operation on the initial features of the reference frame using the forward event features to obtain a forward inter-frame feature sequence; The second pyramid cascade deformable convolutional network is used to perform a backward multi-scale modulation alignment operation on the initial features of the reference frame using the backward event features to obtain a backward inter-frame feature sequence; The fusion layer is used to fuse the forward inter-frame feature sequence and the backward inter-frame feature sequence to obtain the latent inter-frame feature sequence.

5. The method according to claim 4, characterized in that The method comprises performing global modeling on the event features and image features in the first frame feature sequence by combining the event information through a bidirectional cyclic compensation operation based on the first frame feature sequence; Through the forward compensation operation and the backward compensation operation in the global modeling process, in the process of obtaining the second frame feature sequence, the global modeling performed is a bidirectional recurrent network; The bidirectional recurrent network includes a forward recurrent compensation network, a backward recurrent compensation network and an output module; The forward loop network has a first channel attention module and a forward loop block arranged in sequence along the data stream transmission direction, and the backward loop network has a second channel attention module and a backward loop block arranged in sequence along the data stream transmission direction; The first channel attention module is used to fuse the frame features of a certain time step in the first frame feature sequence and the forward event features of a certain time step, and input them into the forward loop block for processing; the forward loop block is used to fuse the output of the backward loop block of the previous time step with the output of the first channel attention module of the certain time step; The second channel attention module is used to fuse the frame features of a certain time step in the first frame feature sequence with the backward event features of a certain time step, and input them into the backward loop block for processing; The backward loop block is used to fuse the output of the forward loop block of the current time step, the output of the second channel attention module of the time step, and the output of the backward loop block of the next time step; The output module is used to perform element-by-element addition operation on the output of the forward loop compensation network at each time step and the output of the backward loop compensation network at each time step to obtain the second frame feature sequence.

6. The method according to claim 5, characterized in that The process of using the cross-scale spatiotemporal attention mechanism to learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain and mapping the low-resolution frame features of the second frame feature sequence to high-resolution video frames based on any given temporal upsampling factor and spatial upsampling factor includes: Based on a certain temporal upsampling factor and a spatial upsampling factor of the original video frame, the query coordinates of the high-resolution grid corresponding to the target frame, the sampling coordinates and unit codes of the low-resolution grid corresponding to the feature map are obtained; Based on the query coordinates of the high-resolution grid corresponding to the target frame and the sampling coordinates of the low-resolution grid corresponding to the feature map, the relative coordinates of the local sampling grid are obtained, and the relative coordinates of the local sampling grid are converted into cosine code to obtain a position offset; Perform a three-dimensional convolution operation on the feature sequence of the second frame to convert the feature sequence of the second frame into a query feature sequence, a key feature sequence, and a value feature sequence; Perform trilinear upsampling on the query feature sequence through the query coordinates of the high-resolution grid corresponding to the target frame to obtain the query sampling data; Based on the sampling coordinates of the low-resolution grid corresponding to the feature map, the nearest neighbor local sampling is performed on the Key feature sequence and the Value feature sequence to obtain the Key sampling data and the Value sampling data; Attention operation is performed on the query sampling data, key sampling data, value sampling data and position bias, and then the query sampling data is used again and input into the multi-layer perceptron together with the result of the above attention operation and the unit code for processing to obtain a high-resolution video frame.

7. An event camera driven video super-resolution reconstruction system, characterized in that: include: The original video frame and event conversion unit is used to convert the low-resolution event stream based on two consecutive reference frames obtained from the low-frame-rate and low-resolution video and the corresponding low-resolution event stream, and invert the timing and polarity of the events to obtain a bidirectional event voxel grid; Frame feature extraction unit, used to: Perform feature extraction operations on two consecutive reference frames and bidirectional event voxel grids respectively to obtain reference frame initial features, forward event features, and backward event features; Through the forward event features and backward event features, the pyramid cascade deformable convolutional network is used to perform forward modulation alignment and backward modulation alignment operations on the initial features of the reference frame to obtain the potential inter-frame feature sequence; The initial features of the reference frame are combined with the potential inter-frame feature sequence to obtain the first frame feature sequence; based on the first frame feature sequence, the event features and image features in the first frame feature sequence are globally modeled by combining the event information through a bidirectional cyclic compensation operation; Obtaining a second frame feature sequence through forward compensation operation and backward compensation operation in the global modeling process; The video frame reconstruction unit is used to: utilize a cross-scale spatiotemporal attention mechanism to learn a unified continuous representation of the frame features of the second frame feature sequence in the spatiotemporal domain, and map the frame features of the low-resolution second frame feature sequence to a high-resolution video frame based on any given temporal upsampling factor and spatial upsampling factor.

8. An event camera driven video super-resolution reconstruction device, characterized in that: The apparatus comprises: a processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 6 by calling the executable instructions.

Images