A video super-resolution method based on event data driving

Abstract

This invention discloses a video super-resolution method based on event data drive. The method comprises the following steps: 1. preparing training video data and corresponding event sequences, and segmenting the event sequences; 2. constructing a video super-resolution neural network model; 3. offline training the constructed video super-resolution neural network model; and 4. using the trained model for prediction to achieve video super-resolution. By utilizing an event data-driven approach, the invention can improve the video super-resolution restoration effect, thereby generating clear video frames.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular to a video super-resolution method driven by event data. Background Art

[0002] In recent years, video super-resolution technology has developed rapidly and has been widely applied in various fields, including remote sensing, video surveillance, and video compression. Compared with single-image super-resolution, video super-resolution is a more challenging task. In addition to improving image quality, video super-resolution also requires ensuring temporal consistency of the video.

[0003] Temporal alignment is a key step in video super-resolution technology, which refers to the alignment of multiple video frames based on temporal correspondence. A commonly used temporal alignment method is to use optical flow to align adjacent frames. However, the optical flow method has certain limitations. For example, the accuracy of optical flow will be affected in the case of lighting changes or rapid motion, which will lead to blurring or occlusion problems. In addition, some studies have tried to use deformable convolution for implicit alignment, but this method has problems in terms of computational complexity and random initialization of learnable offsets.

[0004] From the perspective of method classification, video super-resolution techniques can be mainly divided into two categories: implicit alignment methods and explicit alignment methods. Implicit alignment methods generally rely on the powerful modeling capabilities of deep neural networks to directly generate the current frame without paying too much attention to the alignment between adjacent frames. Explicit alignment methods generally rely on motion estimation and compensation techniques to reconstruct the current frame by aligning adjacent frames and extracting supplementary information. However, these methods still face challenges when facing difficult situations with complex textures, especially since high-frequency information is easily lost in low-resolution sequences, resulting in a lack of high-frequency information in the restored high-resolution sequences.

[0005] Although video super-resolution technology has made significant progress, there are still a series of unresolved problems and challenges in terms of temporal alignment, processing complex scenes, and recovering high-frequency information. These problems and challenges provide directions for further research. Summary of the Invention

[0006] In order to address the shortcomings of the above-mentioned existing technologies, the present invention proposes a video super-resolution method driven by event data. The method aims to effectively capture subtle changes in objects under high-speed motion and motion blur environments by introducing event data, thereby achieving more accurate inter-frame alignment and maintaining robustness under complex lighting conditions.

[0007] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0008] The video super-resolution method based on event data driving of the present invention is characterized in that it is performed according to the following steps:

[0009] Step 1: Obtain training video data and corresponding event sequences, and segment the event sequences:

[0010] Step 1.1.1, obtain a high-resolution video image set, denoted as Y = {y1, y2, ..., y i ,…,y N}, where y i represents the i-th high-resolution image, i = 1, 2, ..., N, where N is the total number of high-resolution images;

[0011] The high-resolution dataset Y is degraded to obtain a low-resolution video image set, denoted as X = {x1, x2, ..., x i ,...,x N}, where x i represents the i-th low-resolution image;

[0012] Step 1.1.2: Obtain the event sequence of the high-resolution video image set Y, denoted as E Y ={E1,...,E i ,...,E N}, where E i Represents the i-th high-resolution image y i The corresponding sequence of events;

[0013] For the event sequence E Y The event sequence of downgrading to generate low-resolution image set X is denoted as E X ={e1,...,e i ,...,e N},e i Represents the i-th low-resolution image x i The corresponding sequence of events;

[0014] Step 2: Construct a video super-resolution neural network, including: forward propagation module, back propagation module, cross-modal fusion module, and frame reconstruction module;

[0015] Step 2.1, the forward propagation module is used to i-1 、x i and e i-1 Processing is performed to obtain the i-th forward propagation feature map

[0016] Step 2.2: The back propagation module performs i+1 、x i and e iProcessing is performed to obtain the i-th back propagation feature map

[0017] Step 2.3, the cross-modal fusion module i-1 、x i and e i Processing is performed to obtain the cross-modal fusion feature F i ;

[0018] Step 2.4: The frame reconstruction module U is composed of multiple deconvolution layers and upsampling layers connected in series, and the i+1th forward propagation feature map is The i-1th back-propagation feature map and cross-modal fusion features F i After processing, the i-th super-resolution image is obtained Thus, we get a super-resolution video set

[0019] Step 3: Use formula (7) to construct the loss function L:

[0020]

[0021] In formula (7), ε is a non-negative constant;

[0022] Step 4: Use the gradient descent method to train the video super-resolution neural network and calculate the loss function L to update the network parameters. When the number of training iterations reaches the set number or the loss function L converges, the training stops, thereby obtaining the optimal video super-resolution model; it is used to process the low-resolution video image to obtain the corresponding clear high-resolution video image.

[0023] The event data-driven video super-resolution method of the present invention is also characterized in that the forward propagation module includes: a frame optical flow estimation network S, an event optical flow estimation network U, m residual connection layers and a feature fusion layer;

[0024] Step 2.1.1, the frame optical flow estimation network S is composed of an m1-layer downsampling convolutional layer and an m1-layer upsampling convolutional layer alternately connected;

[0025] The i-1th low-resolution image x i-1 and the i-th low-resolution image x i Input the frame optical flow estimation network S to perform optical flow estimation, and obtain the information about x i The i-th forward frame optical flow f i s,f ;

[0026] Using formula (1) to calculate the optical flow f of the i-th forward frame i s,fand the i-1th forward propagation feature map Perform warp transformation to obtain the i-th feature from the forward frame optical flow alignment

[0027]

[0028] In formula (1), warp represents the image warping transformation; when i = 1, let

[0029] Step 2.1.2: The event optical flow estimation network U is a customized lightweight neural network with an m2-layer "encoding-decoding" structure;

[0030] The i-1th event sequence e i-1 Input the event optical flow estimation network U to perform optical flow estimation, and obtain the information about x i The i-th forward event optical flow f i e,f ;

[0031] Using formula (2), the i-th forward event optical flow f i e,f and the i-1th forward propagation feature map Perform warp transformation to obtain the i-th feature from the forward event optical flow alignment

[0032]

[0033] Step 2.1.3: Features aligned with the optical flow of the i-th forward frame Features aligned with the i-th forward event optical flow The features are extracted through m non-shared residual connection layers respectively, and the frame alignment feature map and event alignment feature map are obtained accordingly. Then, after splicing along the feature channel, the i-th forward splicing feature map F is obtained. i f ;

[0034] Step 2.1.4: The feature fusion layer combines the i-th forward concatenated feature map F i f With the i-th low-resolution image x i After splicing, residual processing is performed to obtain the i-th forward propagation feature map

[0035] The back propagation module includes: a frame optical flow estimation network S and an event optical flow estimation network U that share parameters with the forward propagation module, m residual connection layers, and a feature fusion layer;

[0036] Step 2.2.1: The i+1th low-resolution image xi+1 and the i-th low-resolution image x i Input the frame optical flow estimation network S to perform optical flow estimation, and obtain the information about x i The i-th reverse frame optical flow f i s,b ;

[0037] Using formula (3) for the i-th reverse frame optical flow f i s,b and the i-1th back-propagation feature map Perform warp transformation to obtain the i-th feature from the reverse frame optical flow alignment

[0038]

[0039] In formula (3), when i=1, let

[0040] Step 2.2.2: The i-th event sequence e i Input the event optical flow estimation network U to perform optical flow estimation and obtain the i-th reverse event optical flow f i e,b ;

[0041] Using formula (4), the i-th reverse event optical flow f i e,b and the i-1th back-propagation feature map Perform warp transformation to obtain the i-th feature from event optical flow alignment

[0042]

[0043] Step 2.2.3, features of the i-th reverse frame optical flow alignment Features aligned with the optical flow of the i-th event The features are extracted through m non-shared residual connection layers respectively, and the frame alignment feature map and event alignment feature map are obtained accordingly. Then, after splicing along the feature channel, the i-th reverse splicing feature map F is obtained. i b ;

[0044] Step 2.2.4: The feature fusion layer combines the i-th reverse splicing feature map F i b With the i-th low-resolution image x i After splicing, residual processing is performed to obtain the i-th back propagation feature map

[0045] The cross-modal fusion module includes: a feature pyramid extractor FE, a channel attention branch, and a spatial attention branch;

[0046] Step 2.3.1, the feature pyramid extractor FE is composed of S convolutional layers;

[0047] The i-1th event sequence e i-1 , the i-th low-resolution image x i and the i-th event sequence e i They are respectively input into the feature pyramid extractor FE for feature extraction, and the pyramid features of each layer are obtained accordingly, where the i-1 event sequence e i-1 The s-th layer feature is recorded as The i-th low-resolution image x i The s-th layer feature is recorded as IF i s , the i-th event sequence e i The s-th layer feature is recorded as EF i s ;

[0048] Step 2.3.2, the channel attention branch takes the i-th low-resolution image x i The s-th layer feature IF i s Query vector The i-1th event sequence e i-1 The s-th layer features Projected into key-value pair vectors through two different convolutional layers and Thus, the i-th low-resolution image x is calculated using formula (5) i Cross-modal channel attention of the s-th layer feature

[0049]

[0050] In formula (5), c represents the key-value pair vector and The channel dimension, softmax represents the activation function;

[0051] Step 2.3.3, the spatial attention branch takes the i-th low-resolution image x i The s-th layer feature IF i s Query vector The i-th event sequence e i The s-th layer feature EF i s Projected into key-value pair vectors through two different convolutional layers and V i s , and thus use formula (6) to calculate the i-th low-resolution image x i Cross-modal spatial attention of the s-th layer features

[0052]

[0053] In formula (6), n represents the key-value pair vector and V i s spatial dimension;

[0054] Step 2.3.4: Cross-modal channel attention and cross-modal spatial attention Spliced ​​along the channel, and processed by convolution with a convolution kernel of 1×1, and then combined with the i-th low-resolution image x i The s-th layer feature IF i s Add together to get the s-th layer cross-modal fusion feature F of the i-th image i s , and then pass through m layers of decoding layers to obtain the cross-modal fusion feature F i .

[0055] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the video super-resolution method, and the processor is configured to execute the program stored in the memory.

[0056] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the video super-resolution method when executed by a processor.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. This paper designs a video super-resolution network based on event data, integrating event data into the video super-resolution task. Compared with current mainstream video super-resolution methods, this method not only improves the resolution quality of generated frames but also demonstrates greater robustness when handling complex scenes (such as high-speed motion, motion blur, and strong lighting changes).

[0059] 2. This invention innovatively extracts nonlinear optical flow from event data to align features of adjacent frames. This, combined with traditional frame-based optical flow alignment methods, is particularly suitable for scenes with large motion, enabling more accurate inter-frame alignment in video super-resolution applications.

[0060] 3. This paper constructs a cross-modal information fusion design that combines image frames and event data and uses a spatiotemporal attention mechanism to effectively mine the correlation information between them. This design can more effectively exploit the illumination robustness of event data, thereby achieving more robust frame restoration in video super-resolution.

[0061] 4. The present invention adopts a supervised training method for training, deeply embeds event information into the video super-resolution network, and improves the quality of the output frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of the process of the present invention;

[0063] Figure 2 This is a structural diagram of the forward propagation module, backpropagation module, cross-modal fusion module, and frame reconstruction module of the present invention;

[0064] Figure 3 This is a structural diagram of the cross-modal fusion of the present invention. DETAILED DESCRIPTION

[0065] In this embodiment, a video super-resolution method based on event data is an event-driven video alignment solution that uses event data to estimate nonlinear optical flow and builds a cross-modal information fusion module to generate high-quality high-resolution video frames. The main feature is that the data of the two modalities of event and frame image are fused and propagated in a bidirectional recurrent neural network. Figure 1 The specific steps are as follows:

[0066] Step 1: Prepare training video data and corresponding event sequences, and segment the event sequences:

[0067] Step 1.1.1, obtain a high-resolution video image set, denoted as Y = {y1, y2, ..., y i ,...,y N}, where y i represents the i-th high-resolution image, i = 1, 2, ..., N, N is the total number of high-resolution images; in this example, the total number of images during neural network training is N = 15; the high-resolution dataset Y is degraded to obtain a low-resolution video image set. In this example, the bilinear interpolation algorithm is used to degrade Y, X = {x1, x2, ..., x i ,…,x N}, where x i represents the i-th low-resolution image.

[0068] Step 1.1.2: Obtain the event sequence of the high-resolution video image set Y, denoted as E Y ={E1,...,Ei ,...,E N}, where E i Represents the i-th high-resolution image y i The corresponding event sequence; In this example, the event camera simulator ESIM is used to directly simulate the event data of the input video image set Y; Then the event sequence E Y The event sequence of downgrading to generate low-resolution image set X is denoted as E X ={e1,...,e i ,...,e N},e i Represents the i-th low-resolution image x i The corresponding sequence of events.

[0069] Step 2: Construct a video super-resolution neural network, such as Figure 1 As shown, it includes: forward propagation module, back propagation module, cross-modal fusion module, and frame reconstruction module;

[0070] Step 2.1, such as Figure 2 As shown in part a of the figure, the forward propagation module includes: a frame optical flow estimation network, an event optical flow estimation network, m residual connection layers (in this example, m = 15); and a feature fusion layer;

[0071] Step 2.1.1, the frame optical flow estimation network consists of an m1-layer downsampling convolution layer and an m1-layer upsampling convolution layer connected alternately. In this example, m1 = 5; by taking the i-1th low-resolution image x i-1 and the i-th low-resolution image x i Input frame optical flow estimation network S to perform optical flow estimation, and get the value of x i The i-th forward frame optical flow f i s,f ;

[0072] Formula (1) is to use the estimated forward frame optical flow f i s,f For the i-1th forward propagation feature map Perform warp transformation to obtain the i-th feature from the forward frame optical flow alignment Where warp represents the image distortion transformation:

[0073]

[0074] In formula (1), when i=1, let

[0075] Step 2.1.2: The event optical flow estimation network is a customized lightweight neural network with m2 layers of “encoding-decoding” structure; in this example, m2=5; by transforming the i-1th event sequence e i-1 Input event optical flow estimation network U to perform optical flow estimation and obtain information about x i The i-th forward event optical flow f i e,f .

[0076] Formula (2) is to use the estimated forward event optical flow f i e,f For the i-1th forward propagation feature map Perform warp transformation to obtain the i-th feature from the forward event optical flow alignment

[0077]

[0078] Step 2.1.3: Features from the optical flow alignment of the i-th forward frame Features aligned with the i-th forward event optical flow The features are extracted through m non-shared residual connection layers respectively, and the frame alignment feature map and event alignment feature map are obtained accordingly. Then, after splicing along the feature channel, the i-th forward splicing feature map F is obtained. i f .

[0079] Step 2.1.4, the feature fusion layer combines the i-th forward concatenated feature map F i f With the i-th low-resolution image x i After splicing, residual processing is performed to obtain the i-th forward propagation feature map

[0080] Step 2.2, such as Figure 2 As shown in part b of the figure, the back propagation module includes: a frame optical flow estimation network and an event optical flow estimation network that share parameters with the forward propagation module, m residual connection layers, and a feature fusion layer;

[0081] Step 2.2.1, by taking the i+1th low-resolution image x i+1 and the i-th low-resolution image x i Input frame optical flow estimation network S to perform optical flow estimation, and get the value of x i The i-th reverse frame optical flow f i s,b ;

[0082] Formula (3) is to use the estimated reverse frame optical flow f i s,b For the i-1th back propagation feature map Perform warp transformation to obtain the i-th feature from the reverse frame optical flow alignment

[0083]

[0084] In formula (3), when i=1, let

[0085] Step 2.2.2: By transforming the i-th event sequence e i Input the event optical flow estimation network to perform optical flow estimation and obtain the i-th reverse event optical flow f i e,b .

[0086] Formula (4) is to use the estimated reverse event optical flow f i e,b For the i-1th back propagation feature map Perform warp transformation to obtain the i-th feature from event optical flow alignment

[0087]

[0088] Step 2.2.3: Features from the i-th reverse frame optical flow alignment Features aligned with the optical flow of the i-th event The features are extracted through m non-shared residual connection layers respectively, and the frame alignment feature map and event alignment feature map are obtained accordingly. Then, after splicing along the feature channel, the i-th reverse splicing feature map F is obtained. i b .

[0089] Step 2.2.4, the feature fusion layer reversely concatenates the i-th feature map F i b With the i-th low-resolution image x i After splicing, residual processing is performed to obtain the i-th back propagation feature map

[0090] Step 2.3, such as Figure 2 As shown in part c of Figure 1, the cross-modal fusion module includes: feature pyramid extractor FE, channel attention branch, and spatial attention branch;

[0091] Step 2.3.1, the feature pyramid extractor FE consists of S layers of convolutional layers, and the size of the feature map becomes smaller and smaller as it goes up the layer;

[0092] The i-1th event sequence e i-1 , the i-th low-resolution image x i and the i-th event sequence ei They are input into the feature pyramid extractor FE for feature extraction, and the pyramid features of each layer are obtained accordingly. Figure 3 As shown, in this example, the number of extracted feature pyramid layers is 5, among which the event sequence e i-1 The s-th layer feature is recorded as Low-resolution image x i The s-th layer feature is recorded as IF i s , event sequence e i The s-th layer feature is recorded as EF i s .

[0093] Step 2.3.2, the channel attention branch takes the i-th low-resolution image x i The s-th layer feature IF i s Query vector The i-1th event sequence e i-1 The s-th layer features Projected into key-value pair vectors through two different convolutional layers and Thus, the i-th low-resolution image x is calculated using formula (5) i Cross-modal channel attention of the s-th layer feature

[0094]

[0095] Where c represents the key-value pair vector and The channel dimension of , and softmax represents the activation function.

[0096] Step 2.3.3, the spatial attention branch takes the i-th low-resolution image x i The s-th layer feature IF i s Query vector The i-th event sequence e i The s-th layer feature EF i s Projected into key-value pair vectors through two different convolutional layers and V i s , and thus use formula (6) to calculate the i-th low-resolution image x i Cross-modal spatial attention of the s-th layer features

[0097]

[0098] Where n represents the key-value pair vector and V i s spatial dimension.

[0099] Step 2.3.4, cross-modal channel attention and cross-modal spatial attention Spliced ​​along the channel, and processed by convolution with a convolution kernel of 1×1, and then combined with the i-th low-resolution image x i The s-th layer feature IF i s Add together to get the s-th layer cross-modal fusion feature F of the i-th image i s , and then pass through m layers of decoding layers to obtain the cross-modal fusion feature F i ;

[0100] Step 2.4, such as Figure 2 As shown in part d of the figure, the frame reconstruction module consists of multiple deconvolution layers and upsampling layers connected in series, and the i+1th forward propagation feature map is The i-1th back-propagation feature map and cross-modal fusion features F i After processing, the i-th high-resolution image is obtained The super-resolution video set is recorded as

[0101] Step 3: Use formula (7) to construct the loss function L:

[0102]

[0103] In formula (7), ε is a non-negative constant;

[0104] Step 4: Use the gradient descent method to train the video super-resolution neural network and calculate the loss function L to update the network parameters. When the number of training iterations reaches the set number or the loss function L converges, the training stops, thereby obtaining the optimal video super-resolution model; it is used to process the low-resolution video image to obtain the corresponding clear high-resolution video image.

[0105] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0106] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

Claims

1. A video super-resolution method based on event data driving, characterized in that: The steps are as follows: Step 1: Obtain training video data and corresponding event sequences, and segment the event sequences: Step 1.1.1, obtain a high-resolution video image set, denoted as Y = {y1, y2, ..., y i ,…,y N }, where y i represents the i-th high-resolution image, i = 1, 2, ..., N, N is the total number of high-resolution images; The high-resolution dataset Y is degraded to obtain a low-resolution video image set, denoted as X = {x1, x2, ..., x i ,...,x N }, where x i represents the i-th low-resolution image; Step 1.1.2: Obtain the event sequence of the high-resolution video image set Y, denoted as E Y ={E1,...,E i ,...,E N }, where E i Represents the i-th high-resolution image y i The corresponding sequence of events; For the event sequence E Y The event sequence of downgrading to generate low-resolution image set X is denoted as E X ={e1,...,e i ,...,e N },e i Represents the i-th low-resolution image x i The corresponding sequence of events; Step 2: Construct a video super-resolution neural network, including: forward propagation module, back propagation module, cross-modal fusion module, and frame reconstruction module; Step 2.1, the forward propagation module is used to i-1 、x i and e i-1 Processing is performed to obtain the i-th forward propagation feature map Step 2.2: The back propagation module performs i+1 、x i and e i Processing is performed to obtain the i-th back propagation feature map Step 2.3, the cross-modal fusion module i-1 、x i and e i Processing is performed to obtain the cross-modal fusion feature F i ; Step 2.4: The frame reconstruction module U is composed of multiple deconvolution layers and upsampling layers connected in series, and the i+1th forward propagation feature map is The i-1th back-propagation feature map and cross-modal fusion features F i After processing, the i-th super-resolution image is obtained Thus, we get a super-resolution video set Step 3: Use formula (7) to construct the loss function L: In formula (7), ε is a non-negative constant; Step 4: Use the gradient descent method to train the video super-resolution neural network and calculate the loss function L to update the network parameters. When the number of training iterations reaches the set number or the loss function L converges, the training stops, thereby obtaining the optimal video super-resolution model; it is used to process the low-resolution video image to obtain the corresponding clear high-resolution video image.

2. The event data driven video super-resolution method according to claim 1, characterized in that: The forward propagation module includes: a frame optical flow estimation network S, an event optical flow estimation network U, m residual connection layers and a feature fusion layer; Step 2.1.1, the frame optical flow estimation network S is composed of an m1-layer downsampling convolutional layer and an m1-layer upsampling convolutional layer alternately connected; The i-1th low-resolution image x i-1 and the i-th low-resolution image x i Input the frame optical flow estimation network S to perform optical flow estimation, and obtain the information about x i The i-th forward frame optical flow f i s,f ; Using formula (1) to calculate the optical flow f of the i-th forward frame i s,f and the i-1th forward propagation feature map Perform warp transformation to obtain the i-th feature from the forward frame optical flow alignment In formula (1), warp represents the image warping transformation; when i = 1, let Step 2.1.2: The event optical flow estimation network U is a customized lightweight neural network with an m2-layer "encoding-decoding" structure. The i-1th event sequence e i-1 Input the event optical flow estimation network U to perform optical flow estimation, and obtain the information about x i The i-th forward event optical flow f i e,f ; Using formula (2), the i-th forward event optical flow f i e,f and the i-1th forward propagation feature map Perform warp transformation to obtain the i-th feature from the forward event optical flow alignment Step 2.1.3: Features aligned with the optical flow of the i-th forward frame Features aligned with the i-th forward event optical flow The features are extracted through m non-shared residual connection layers respectively, and the frame alignment feature map and event alignment feature map are obtained accordingly. Then, after splicing along the feature channel, the i-th forward splicing feature map F is obtained. i f ; Step 2.1.4: The feature fusion layer combines the i-th forward concatenated feature map F i f With the i-th low-resolution image x i After splicing, residual processing is performed to obtain the i-th forward propagation feature map 3. The event data driven video super-resolution method according to claim 2, characterized in that: The back propagation module includes: a frame optical flow estimation network S and an event optical flow estimation network U that share parameters with the forward propagation module, m residual connection layers, and a feature fusion layer; Step 2.2.1: The i+1th low-resolution image x i+1 and the i-th low-resolution image x i Input the frame optical flow estimation network S to perform optical flow estimation, and obtain the information about x i The i-th reverse frame optical flow f i s,b ; Using formula (3) for the i-th reverse frame optical flow f i s,b and the i-1th back-propagation feature map Perform warp transformation to obtain the i-th feature from the reverse frame optical flow alignment In formula (3), when i=1, let Step 2.2.2: The i-th event sequence e i Input the event optical flow estimation network U to perform optical flow estimation and obtain the i-th reverse event optical flow f i e,b ; Using formula (4), the i-th reverse event optical flow f i e,b and the i-1th back-propagation feature map Perform warp transformation to obtain the i-th feature from event optical flow alignment Step 2.2.3, features of the i-th reverse frame optical flow alignment Features aligned with the optical flow of the i-th event The features are extracted through m non-shared residual connection layers respectively, and the frame alignment feature map and event alignment feature map are obtained accordingly. Then, after splicing along the feature channel, the i-th reverse splicing feature map F is obtained. i b ; Step 2.2.4: The feature fusion layer combines the i-th reverse splicing feature map F i b With the i-th low-resolution image x i After splicing, residual processing is performed to obtain the i-th back propagation feature map 4. The event data driven video super-resolution method according to claim 3, characterized in that: The cross-modal fusion module includes: a feature pyramid extractor FE, a channel attention branch, and a spatial attention branch; Step 2.3.1, the feature pyramid extractor FE is composed of S convolutional layers; The i-1th event sequence e i-1 , the i-th low-resolution image x i and the i-th event sequence e i They are respectively input into the feature pyramid extractor FE for feature extraction, and the pyramid features of each layer are obtained accordingly, where the i-1 event sequence e i-1 The s-th layer feature is recorded as The i-th low-resolution image x i The s-th layer feature is recorded as IF i s , the i-th event sequence e i The s-th layer feature is recorded as EF i s ; Step 2.3.2, the channel attention branch takes the i-th low-resolution image x i The s-th layer feature IF i s Query vector The i-1th event sequence e i-1 The s-th layer features Projected into key-value pair vectors through two different convolutional layers and Thus, the i-th low-resolution image x is calculated using formula (5) i Cross-modal channel attention of the s-th layer feature In formula (5), c represents the key-value pair vector and The channel dimension, softmax represents the activation function; Step 2.3.3, the spatial attention branch takes the i-th low-resolution image x i The s-th layer feature IF i s Query vector The i-th event sequence e i The s-th layer feature EF i s Projected into key-value pair vectors through two different convolutional layers and V i s , and thus use formula (6) to calculate the i-th low-resolution image x i Cross-modal spatial attention of the s-th layer features In formula (6), n represents the key-value pair vector and V i s spatial dimension; Step 2.3.4: Cross-modal channel attention and cross-modal spatial attention Spliced ​​along the channel, and processed by convolution with a convolution kernel of 1×1, and then combined with the i-th low-resolution image x i The s-th layer feature IF i s Add together to get the s-th layer cross-modal fusion feature F of the i-th image i s , and then pass through m layers of decoding layers to obtain the cross-modal fusion feature F i .

5. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the video super-resolution method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the video super-resolution method according to any one of claims 1 to 4 are executed.

Images