Video reconstruction method and device based on event stream, electronic device and storage medium

Abstract

The application relates to an event stream-based video reconstruction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring event stream data of a target dynamic scene and converting the event stream data into an event frame in the form of a multi-channel tensor; inputting the event frame in the form of a multi-channel tensor into a preset convolutional neural network model to output a cross-scale predicted image frame, wherein the preset convolutional neural network model is trained by historical event frames, and the preset convolutional neural network model comprises a preset bidirectional convolutional long short-term memory module, a preset multi-scale spatial enhancement module, a preset spatio-temporal fusion attention module and a preset multi-scale feature aggregation module; and reconstructing video data of the target dynamic scene based on the cross-scale predicted image frame. Thus, high-quality video reconstruction is realized by recording continuous event streams in a dynamic scene, and the problems of image blurring and fog-like artifacts caused by nonlinear time information and non-uniform spatial distribution in a complex dynamic scene are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an event stream-based video reconstruction method and device, electronic equipment and storage medium. BACKGROUND

[0002] Traditional cameras have limitations in complex dynamic scenes, for example, in high dynamic range or fast motion scenes, traditional cameras may have problems such as overexposure, underexposure or motion blur; and event cameras are different from traditional frame-based cameras, when the logarithmic luminance change of the scene exceeds a predetermined threshold, the event camera will record the coordinate position, time and change polarity of the change occurrence with a very high time resolution. The event camera has the characteristics of high time resolution, high dynamic range and low delay, and can provide more abundant dynamic information, which makes the event camera have advantages in various visual tasks, including object detection, high-speed motion estimation, semantic segmentation and video frame interpolation tasks, but the output event data of the event camera is different from the traditional video frame format, so special algorithms need to be developed to process these data.

[0003] In related technologies, existing deep learning methods have made remarkable progress in reconstructing videos from event streams, these deep learning methods usually use recursive fully convolutional networks to generate intensity frames from events. In order to enhance the generalization to real event data, researchers generate synthetic training datasets using statistical analysis of existing datasets. Another solution involves self-supervised learning, i.e. training neural networks using estimated optical flow and event-based photometric consistency assumptions without real reference images.

[0004] However, due to the characteristics of event streams and their inherent photometric consistency assumption conditions, these methods still face limitations such as image blur caused by non-linear time information in complex dynamic scenes, and fog-like artifacts caused by non-uniform spatial distribution in complex dynamic scenes, which need to be solved. SUMMARY

[0005] The present application provides an event stream-based video reconstruction method, device, electronic equipment and storage medium to solve the problems of image blur and fog-like artifacts caused by non-linear time information and non-uniform spatial distribution in complex dynamic scenes, thereby meeting the needs of video reconstruction to higher image quality and more robust development tasks.

[0006] To achieve the above purpose, the first aspect of the present application provides an event stream-based video reconstruction method, comprising the following steps:

[0007] Obtaining event stream data of a target dynamic scene, and converting the event stream data into an event frame in the form of a multi-channel tensor;

[0008] inputting the event frame in a multi-channel tensor form into a preset convolutional neural network model, and outputting a cross-scale predicted image frame, wherein the preset convolutional neural network model is trained by historical event frames, and the preset convolutional neural network model comprises a preset bidirectional convolutional long short-term memory module, a preset multi-scale spatial enhancement module, a preset spatio-temporal fusion attention module, and a preset multi-scale feature aggregation module;

[0009] reconstructing video data of the target dynamic scene based on the cross-scale predicted image frame.

[0010] According to an embodiment of the present application, the event stream data is converted into an event frame in a multi-channel tensor form, comprising:

[0011] creating a full zero matrix of a preset dimension;

[0012] dividing a time range of the event stream data into a plurality of sub-time intervals uniformly based on the preset dimension;

[0013] determining a normalization interval based on a number of the plurality of sub-time intervals, and normalizing a timestamp of each event in the event stream data to the normalization interval to obtain each normalized new event timestamp;

[0014] According to the each normalized new event timestamp, each normalized event is interpolated into the full zero matrix by using a bilinear interpolation strategy with event polarity as a weight, to obtain the event frame in the multi-channel tensor form.

[0015] According to an embodiment of the present application, the event frame in the multi-channel tensor form is inputted into a preset convolutional neural network, and a cross-scale predicted image frame is outputted, comprising:

[0016] preprocessing the event frame in the multi-channel tensor form, and inputting the preprocessed event frame into the preset bidirectional convolutional long short-term memory module to obtain a cross-time dynamic feature;

[0017] inputting the event frame in the multi-channel tensor form and a preset image frame into the preset multi-scale spatial enhancement module to obtain spatial complementary features of different scales;

[0018] inputting the cross-time dynamic feature and the spatial complementary features of different scales into the preset spatio-temporal fusion attention module to obtain fusion features of different scales;

[0019] inputting the fusion features of different scales into the preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales, and obtaining the cross-scale predicted image frame based on the multi-scale enhanced features of different scales.

[0020] According to one embodiment of the present application, the event frame in the form of a multi-channel tensor is preprocessed, including:

[0021] The event frame in the form of a multi-channel tensor is sequentially input to a 3x3 convolution layer, an activation layer and a residual block to obtain the preprocessed event frame.

[0022] According to one embodiment of the present application, the preprocessed event frame is input to the preset bidirectional convolution long short-term memory module to obtain dynamic features across time, including:

[0023] The preprocessed event frame is input to a forward convolution unit in the preset bidirectional convolution long short-term memory module, and features of each time step are processed based on a preset cycle step to generate a forward feature sequence;

[0024] The preprocessed event frame is input to a backward convolution unit in the preset bidirectional convolution long short-term memory module, and the features of each time step are processed based on the preset cycle step to generate a backward feature sequence;

[0025] The forward feature sequence and the backward feature sequence are fused to obtain the dynamic features across time.

[0026] According to one embodiment of the present application, the event frame in the form of a multi-channel tensor and a preset image frame are input to the preset multi-scale spatial enhancement module to obtain spatial complementary features of different scales, including:

[0027] Dynamic features are extracted from the event frame in the form of a multi-channel tensor, and static features are extracted from the preset image frame;

[0028] The dynamic features and the static features are input to the preset multi-scale spatial enhancement module for connection operation to obtain the spatial complementary features of different scales.

[0029] According to one embodiment of the present application, the dynamic features across time and the spatial complementary features of different scales are input to the preset attention module of spatio-temporal fusion to obtain fusion features of different scales, including:

[0030] The dynamic features across time and the spatial complementary features of different scales are subjected to layer regularization and 1x1 convolution operation by the preset attention module of spatio-temporal fusion to obtain a query matrix, a key matrix and a value matrix;

[0031] An attention matrix is calculated based on the query matrix, the key matrix and the value matrix;

[0032] multiply the attention matrix and the value matrix, and add the obtained multiplication result to the spatial complementary features of different scales to obtain a sum result;

[0033] input the sum result into a normalization layer and a multilayer perception machine in sequence to obtain the fusion features of different scales.

[0034] According to an embodiment of the present application, the inputting of the fusion features of different scales into the preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales comprises:

[0035] performing upsampling or downsampling operation on the fusion features of different scales to obtain fusion features of a preset scale;

[0036] inputting the fusion features of the preset scale into the preset multi-scale feature aggregation module to obtain the multi-scale enhanced features of different scales.

[0037] According to an embodiment of the present application, before the inputting of the event frame in the form of a multi-channel tensor into the preset convolutional neural network model, further comprising:

[0038] obtaining event stream data and corresponding clear image sequences of a plurality of historical dynamic scenes, and converting the event stream data of the plurality of historical dynamic scenes into historical event frames in the form of a multi-channel tensor;

[0039] inputting the historical event frames as a training set into a preset convolutional neural network, training the preset convolutional neural network to obtain an initial convolutional neural network model;

[0040] verifying the initial convolutional neural network model by using the corresponding clear image sequences as a verification set until the initial convolutional neural network model meets a preset standard, ending the iterative training of the preset convolutional neural network to obtain the preset convolutional neural network model, otherwise, based on a preset optimization algorithm, continuing the iterative training after adjusting model parameters through back propagation.

[0041] According to the video reconstruction method based on event stream provided in the embodiments of the present application, the features of the event stream data of the target dynamic scene are extracted through a multi-scale multi-output multi-input neural network, a bidirectional convolutional long short-term memory module is proposed, which can simultaneously integrate past and future event information, a multi-scale spatial enhancement module is proposed, which is used to extract multi-scale complementary spatial features from events and from the reconstructed image of the previous stage, and a spatio-temporal fusion attention module is proposed to integrate spatial features at different time scales. Thus, high-quality video reconstruction is realized by recording continuous event streams in a dynamic scene, the image blur and fog-like artifacts caused by nonlinear time information and non-uniform spatial distribution in a complex dynamic scene are solved, thereby meeting the demand of video reconstruction for higher image quality and more robust development tasks.

[0042] To achieve the above object, the second aspect of the present application proposes a video reconstruction device based on event stream, comprising:

[0043] A conversion module is configured to obtain event stream data of a target dynamic scene and convert the event stream data into event frames in the form of multi-channel tensors;

[0044] An obtaining module is configured to input the event frames in the form of multi-channel tensors into a preset convolutional neural network model to output cross-scale predicted image frames, wherein the preset convolutional neural network model is trained by historical event frames, and the preset convolutional neural network model comprises a preset bidirectional convolutional long short-term memory module, a preset multi-scale spatial enhancement module, a preset spatio-temporal fusion attention module and a preset multi-scale feature aggregation module.

[0045] A reconstruction module is configured to reconstruct video data of the target dynamic scene based on the cross-scale predicted image frames.

[0046] According to an embodiment of the present application, the conversion module is specifically configured to:

[0047] Create a full zero matrix of a preset dimension;

[0048] Divide the time range of the event stream data into a plurality of sub-time intervals uniformly based on the preset dimension;

[0049] Determine a normalization interval based on the number of the plurality of sub-time intervals, and normalize the timestamp of each event in the event stream data to the normalization interval to obtain each normalized new event timestamp;

[0050] According to the each normalized new event timestamp, each normalized event is interpolated into the full zero matrix by using a bilinear interpolation strategy with event polarity as the weight, to obtain the event frames in the form of multi-channel tensors.

[0051] According to an embodiment of the present application, the obtaining module comprises:

[0052] The first obtaining unit is configured to pre-process the event frame in the form of the multi-channel tensor and input the pre-processed event frame into the preset bidirectional convolutional long short-term memory module to obtain dynamic features across time;

[0053] The second obtaining unit is configured to input the event frame in the form of the multi-channel tensor and the preset image frame into the preset multi-scale spatial enhancement module to obtain spatial complementary features of different scales;

[0054] The third obtaining unit is configured to input the dynamic features across time and the spatial complementary features of different scales into the preset attention module of spatio-temporal fusion to obtain fusion features of different scales;

[0055] The fourth obtaining unit is configured to input the fusion features of different scales into the preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales, and obtain the predicted image frame across scales based on the multi-scale enhanced features of different scales.

[0056] According to an embodiment of the present application, the obtaining module is specifically configured to:

[0057] The event frame in the form of the multi-channel tensor is sequentially input into a 3*3 convolutional layer, an activation layer and a residual block to obtain the pre-processed event frame.

[0058] According to an embodiment of the present application, the first obtaining unit is specifically configured to:

[0059] The pre-processed event frame is input into a forward convolutional unit in the preset bidirectional convolutional long short-term memory module, features of each time step are processed based on a preset cycle step to generate a forward feature sequence;

[0060] The pre-processed event frame is input into a backward convolutional unit in the preset bidirectional convolutional long short-term memory module, the features of each time step are processed based on the preset cycle step to generate a backward feature sequence;

[0061] The forward feature sequence and the backward feature sequence are fused to obtain the dynamic features across time.

[0062] According to an embodiment of the present application, the second obtaining unit is specifically configured to:

[0063] Dynamic features are extracted from the event frame in the form of the multi-channel tensor, and static features are extracted from the preset image frame;

[0064] The dynamic feature and the static feature are input to the preset multi-scale spatial enhancement module for connection operation to obtain the spatial complementary features of different scales.

[0065] According to an embodiment of the present application, the third obtaining unit is specifically configured to:

[0066] The spatial complementary features of different scales and the dynamic features across time are subjected to layer regularization and 1x1 convolution operation by using the preset spatio-temporal fusion attention module to obtain a query matrix, a key matrix and a value matrix.

[0067] An attention matrix is calculated based on the query matrix, the key matrix and the value matrix.

[0068] The attention matrix and the value matrix are multiplied, and the product obtained is added to the spatial complementary features of different scales to obtain a sum result.

[0069] The sum result is sequentially input to a normalization layer and a multi-layer perception to obtain the fusion features of different scales.

[0070] According to an embodiment of the present application, the fourth obtaining unit is specifically configured to:

[0071] The fusion features of different scales are subjected to up-sampling or down-sampling operation to obtain fusion features of a preset scale.

[0072] The fusion features of the preset scale are input to the preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales.

[0073] According to an embodiment of the present application, before the event frame in the form of the multi-channel tensor is input to the preset convolutional neural network model, the obtaining module is further configured to:

[0074] The historical event frames are input to the preset convolutional neural network as a training set to train the preset convolutional neural network to obtain an initial convolutional neural network model.

[0075] The initial convolutional neural network model is verified by using the corresponding clear image sequence as a verification set until the initial convolutional neural network model meets a preset standard, and the iteration training of the preset convolutional neural network is ended to obtain the preset convolutional neural network model, otherwise, based on a preset optimization algorithm, the iteration training is continued after adjusting the model parameters by back propagation.

[0076] According to the video reconstruction device based on event stream provided by the embodiment of the application, the features of the event stream data of the target dynamic scene are extracted through a multi-scale multi-output multi-input neural network, a bidirectional convolutional long short-term memory module is proposed, which can integrate past and future event information at the same time, a multi-scale spatial enhancement module is proposed, which is used for extracting multi-scale complementary spatial features from events and from the reconstructed image of the previous stage, and a spatio-temporal fusion attention module is proposed to integrate spatial features at different time scales. Thus, high-quality video reconstruction is realized by recording continuous event streams in a dynamic scene, the image blur and fog-like artifact problems caused by nonlinear time information and non-uniform spatial distribution in a complex dynamic scene are solved, and the development task of video reconstruction to higher image quality and more robustness is met.

[0077] To achieve the above object, the third aspect of the embodiment of the application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the video reconstruction method based on event stream as described in the above embodiment.

[0078] To achieve the above object, the fourth aspect of the embodiment of the application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the video reconstruction method based on event stream as described in the above embodiment.

[0079] To achieve the above object, the fifth aspect of the embodiment of the application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the video reconstruction method based on event stream as described in the above embodiment.

[0080] Additional aspects and advantages of the application will be made apparent by the following description and the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0081] The above and / or additional aspects and advantages of the application will become apparent and be readily understood by considering the following detailed description, including the accompanying drawings, in which:

[0082] Figure 1 A flowchart of a video reconstruction method based on event stream according to an embodiment of the application;

[0083] Figure 2 A flowchart of a video reconstruction method using spatio-temporal dynamics of events according to an embodiment of the application;

[0084] Figure 3 A schematic diagram of a preset bidirectional convolutional long short-term memory module according to an embodiment of the application;

[0085] Figure 4 Schematic diagram of a preset multi-scale spatial enhancement module according to an embodiment of the present application;

[0086] Figure 5 Schematic diagram of a preset spatio-temporal fusion attention module according to an embodiment of the present application;

[0087] Figure 6 Schematic diagram of a comparison result of a dataset HQF (High-Quality Frames) and a dataset BS-ERGB (Beam Splitter Event and RGB Dataset) and a current video reconstruction algorithm from event stream according to an embodiment of the present application;

[0088] Figure 7 Block schematic diagram of an event stream-based video reconstruction apparatus provided according to an embodiment of the present application;

[0089] Figure 8 Structural schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0090] Embodiments of the present application are described in detail below with reference to the accompanying drawings. Examples of the embodiments are shown in the drawings, in which the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0091] An event stream-based video reconstruction method, apparatus, electronic device and storage medium according to an embodiment of the present application are described below with reference to the accompanying drawings.

[0092] Figure 1 Flowchart of an event stream-based video reconstruction method according to an embodiment of the present application.

[0093] Before introducing the event stream-based video reconstruction method according to an embodiment of the present application, related technical background is introduced.

[0094] Existing deep learning methods overcome the limitations of traditional optimization methods that rely on scene structure, motion constraints, and artificial integrators or filters, thus avoiding severe loss of details in the reconstruction process. However, due to the characteristics of event streams and their inherent photometric consistency assumption conditions, these methods still face limitations, limiting their performance and application in real-world dynamic scenes. (1) Non-uniform spatial distribution in complex dynamic scenes. In the same scene, the degree of brightness change is different in different areas, resulting in different speeds and quantities of events generated. Fast-moving objects or high-contrast edges will generate a large number of events, thus enriching the reconstruction details of these areas. On the contrary, static or low-contrast areas generate few events, resulting in "blank" areas or loss of details in the reconstructed video. This spatial difference will especially affect the reconstruction of stationary or slowly moving objects, thus reducing the overall quality and consistency of the reconstructed image. (2) Non-linear temporal information in complex dynamic scenes. In dynamic scenes, changes in background, lighting, and object motion will affect the generation and distribution of continuous event data over time, such as irregular time intervals and changing event rates, thus causing frequent changes in the rate and polarity direction of event streams on pixels. Existing deep learning methods cannot cover a wide range of data distributions and only perform well on test data sets similar to the training set data distribution. When the scene is more complex, the performance significantly degrades.

[0095] In summary, existing methods largely rely on pre-trained optical flow models or optical flow estimation within the contrast maximization framework. When there is a significant difference between the event data distribution of the training data set and the data distribution of the real-world test scene, inaccurate optical flow estimation will introduce artifacts in the reconstructed frame, leading to a significant decline in performance.

[0096] Based on the above problems, the embodiments of the present application propose an event stream-based video reconstruction method, which combines Figure 2As shown in the space, a bidirectional convolutional long short-term memory module (BCLSTM) is proposed, which can aggregate past and future information in the event stream, thus modeling dynamics for different sequence lengths and capturing temporal dependencies in the event stream. In space, in order to accurately restore motionless or low-contrast regions, a multi-scale spatial enhancement module (MSSE) is proposed to extract dynamic features from events and static features from previously reconstructed images. These complementary spatial features enable the network to flexibly handle non-uniformly distributed event data. In addition, a spatial-temporal fusion attention module (STFA) is proposed to integrate spatial information of different timestamps and perceive the overall motion and changes in the scene. Finally, in order to effectively combine structural information and details of different scales and make full use of multi-scale features, a multi-scale feature aggregation module (MSFA) is proposed, which can aggregate features of different scales at any given scale, thus facilitating cross-scale information flow within the network. Thus, by recording continuous event streams in dynamic scenes, high-quality video reconstruction is achieved, solving the image blur and fog-like artifacts caused by nonlinear temporal information and non-uniform spatial distribution in complex dynamic scenes, thus meeting the demand for video reconstruction to higher image quality and more robust development tasks.

[0097] As shown in the example, Figure 1 The event stream-based video reconstruction method includes the following steps:

[0098] In step S101, event stream data of a target dynamic scene is obtained, and the event stream data is converted into an event frame in the form of a multi-channel tensor.

[0099] As can be understood, a dynamic scene refers to a scene that changes over time in a video or continuous sequence of images, including moving objects or a changing environment. In computer vision, event stream data typically refers to the information about changes between consecutive frames extracted from a video or image sequence. These changes can be pixel-level or higher-level feature changes, such as object movement or shape changes. In machine learning and deep learning, a tensor refers to a multidimensional array, which can be viewed as a container for numbers. A multichannel tensor means that the tensor has multiple dimensions; for example, in image processing, a color image can be represented as a three-dimensional tensor containing information from the red, green, and blue color channels. An event frame can be understood as a snapshot or frame extracted from event stream data at a specific point in time. These frames contain dynamic information about the scene at that point in time and are usually converted into a format suitable for machine learning models to process.

[0100] Specifically, embodiments of this application can utilize an event camera to perform arbitrary nonlinear motion to capture event stream data in complex dynamic scenes, and convert this data into event frames V in multi-channel tensor form suitable for processing by convolutional neural network structures. ε .

[0101] An event is generated when the brightness change (logarithmic value) at a pixel location exceeds the event trigger threshold C. The polarity of the event indicates whether the brightness increases or decreases, represented by +1 and -1 respectively. The event stream is a collection of these events, with each event containing its location, timestamp, polarity, and index.

[0102] The relationship between the event stream and scene brightness changes is shown in the following equation:

[0103] log(I(x i ,y i ,t i )-I(x i ,y i ,t i -Δt))=p i C;

[0104] Where, p i For the polarity of the event, {x i ,y i ,p i ,t i} represents the set of events (i.e., the event stream) triggered by changes in scene brightness, (x i ,y i ) represents the pixel position where the event was triggered, and t i Let be the trigger time of the event, and i be the i-th triggered event.

[0105] The following will be described in detail how to convert event stream data into an event frame in the form of a multi-channel tensor.

[0106] As a possible implementation, in some embodiments, converting event stream data into an event frame in the form of a multi-channel tensor includes: creating a full zero matrix of a preset dimension; uniformly dividing a time range of the event stream data into a plurality of sub-time intervals based on the preset dimension; determining a normalized interval based on the number of the plurality of sub-time intervals, and normalizing a timestamp of each event in the event stream data to the normalized interval to obtain each normalized new event timestamp; and according to each normalized new event timestamp, using a bilinear interpolation strategy to interpolate each normalized event into the full zero matrix with the event polarity as the weight to obtain the event frame in the form of a multi-channel tensor.

[0107] Wherein, the event stream data is a series of data points recorded over time, each data point containing a timestamp and other related information of the event. Normalization is a method of data preprocessing, which is used to scale the data into a standard interval for processing and comparison.

[0108] Specifically, first, a full zero matrix (i.e., all elements are zero) can be defined for the event stream data, and the dimension is Bin x Width x Height, where Width and Height are the width and height of the event camera resolution, respectively. Then, the start and end time range of the event stream data acquisition is divided into Bin sub-time intervals along the time dimension, and the timestamp of each event in the event stream data is normalized to the normalized interval [0, Bin-1]. Then, using a bilinear interpolation strategy, each event is assigned to the adjacent coordinates according to the pixel position, and the event polarity (+1 or -1) is used as the weight to superimpose to the corresponding position (for example, positive events are added +1 to the corresponding position in the matrix, and negative events are added -1), thereby forming an event frame in the form of a multi-channel tensor, which is displayed in the form of a matrix. Preferably, the Bin in the embodiments of the present application can be set to 5, i.e., the start and end time range of the event stream data acquisition is divided into 5 sub-time intervals along the time dimension, and the normalized interval is [0, 4].

[0109] In step S102, the event frame in the form of a multi-channel tensor is input into a preset convolutional neural network model, and a cross-scale predicted image frame is output, wherein the preset convolutional neural network model is trained by historical event frames, and the preset convolutional neural network model includes a preset bidirectional convolutional long short-term memory module, a preset multi-scale spatial enhancement module, a preset spatio-temporal fusion attention module, and a preset multi-scale feature aggregation module.

[0110] Specifically, after obtaining the event frame in the form of a multi-channel tensor, the event frame in the form of a multi-channel tensor can be input into a preset convolutional neural network model capable of processing and analyzing these data, wherein the preset convolutional neural network model is trained by analyzing historical event frames.

[0111] In addition, the architecture of the preset convolutional neural network model includes several key components that work together to achieve the generation of cross-scale predicted image frames. First, the model contains a preset bidirectional convolutional long short-term memory module BCLSTM, as shown in Figure 3 which can process time series data, dynamically model different sequence lengths, capture long-term dependencies in event streams, and reconstruct target motion information and edge profiles using past and future event streams; secondly, the model also contains a preset multi-scale spatial enhancement module MSSE, as shown in Figure 4 which enhances the model's ability to recognize spatial features by analyzing images at different scales to address the problem of sparse or missing events in certain regions; in addition, in order to integrate the features of the same object close in time to strengthen the features in the target frame, the model integrates a preset spatio-temporal fusion attention module STFA at each scale, as shown in Figure 5 which gives different weights to different temporal and spatial features, allowing the model to focus more on important features and thus improve prediction accuracy; finally, the model also contains a preset multi-scale feature aggregation module MSFA, which integrates feature information from different scales to generate more comprehensive and comprehensive predicted image frames. Through these carefully designed modules, the preset convolutional neural network model can effectively process the input event frames and output high-quality cross-scale predicted image frames.

[0112] As a possible implementation, in some embodiments, inputting the event frame in the form of a multi-channel tensor into a preset convolutional neural network outputs a cross-scale predicted image frame, including: preprocessing the event frame in the form of a multi-channel tensor, and inputting the preprocessed event frame into a preset bidirectional convolutional long short-term memory module to obtain cross-time dynamic features; inputting the event frame in the form of a multi-channel tensor and a preset image frame into a preset multi-scale spatial enhancement module to obtain spatial complementary features at different scales; inputting the cross-time dynamic features and the spatial complementary features at different scales into a preset spatio-temporal fusion attention module to obtain fusion features at different scales; inputting the fusion features at different scales into a preset multi-scale feature aggregation module to obtain multi-scale enhanced features at different scales, and obtaining a cross-scale predicted image frame based on the multi-scale enhanced features at different scales.

[0113] Specifically, in the process of inputting the event frame in the form of a multi-channel tensor into a preset convolutional neural network and outputting a cross-scale predicted image frame, the embodiments of the present application first preprocess the event frame V ε in the form of a multi-channel tensor, and input the preprocessed event frame f V into a preset bidirectional convolutional long short-term memory module. The module can model dynamics for different sequence lengths and capture long-time dependencies in the event stream to output a cross-time dynamic feature F T ; secondly, input the event frame V ε in the form of a multi-channel tensor and the previously reconstructed image frame I L (i.e., a preset image frame) into a preset multi-scale spatial enhancement module to obtain spatial complementary features F S of different scales; then, input the cross-time dynamic feature F T and the spatial complementary features F S of different scales along a time axis into a preset spatio-temporal fusion attention module to aggregate different time features to obtain fusion features (i.e., enhanced spatio-temporal features) F of different scales; and finally, input the fusion features F of different scales into a preset multi-scale feature aggregation module to aggregate features from different scales at any given scale to obtain multi-scale enhanced features F MSFA of different scales, and output a predicted image frame of a corresponding scale through an output layer based on the multi-scale enhanced features F MSFA of different scales.

[0114] Optionally, in some embodiments, the preprocessing of the event frame in the form of a multi-channel tensor includes: sequentially inputting the event frame into a 3x3 convolutional layer, an activation layer and a residual block to obtain the preprocessed event frame.

[0115] It can be understood that the convolutional layer is a layer used to extract features in deep learning, and 3x3 represents the size of the convolutional kernel, which is a commonly used convolutional kernel size in image processing and can extract local features. The activation layer usually refers to a nonlinear activation function added after the convolutional layer, and the purpose is to solve complex problems that cannot be solved by a linear model, such as a ReLU (Rectified Linear Unit) activation function, which can increase the nonlinear capability of the model. The residual block is a structure in a deep neural network, which allows the input to be directly transmitted to the subsequent layers through a skip connection. This structure helps to alleviate the gradient vanishing problem in the training of a deep network, enabling the network to train a deeper model.

[0116] Specifically, in the process of preprocessing the event frame in the form of a multi-channel tensor, the embodiments of the present application can preprocess the event frame V εThe preprocessed event frame f is sequentially input to a 3*3 convolution layer, an activation layer and a residual block, and after processing through these steps, a preprocessed event frame f can be obtained V , which can provide more suitable input data for subsequent deep learning models.

[0117] For ease of understanding, the different stages of the process of obtaining the cross-scale predicted image frame are described in detail below.

[0118] Optionally, in some embodiments, the preprocessed event frame is input to a preset bidirectional convolutional long short-term memory module to obtain a cross-time dynamic feature, including: inputting the preprocessed event frame to a forward convolution unit in the preset bidirectional convolutional long short-term memory module, processing the feature of each time step based on a preset cycle step to generate a forward feature sequence; inputting the preprocessed event frame to a backward convolution unit in the preset bidirectional convolutional long short-term memory module, processing the feature of each time step based on a preset cycle step to generate a backward feature sequence; and fusing the forward feature sequence and the backward feature sequence to obtain a cross-time dynamic feature.

[0119] Specifically, as shown in Figure 3 , the preset bidirectional convolutional long short-term memory module mainly includes a forward convolution unit LSTM and a backward convolution unit LSTM, and the preprocessed event frame f is input to the forward convolution unit and the backward convolution unit of the preset bidirectional convolutional long short-term memory module in sequence. V The forward feature sequence and the backward feature sequence can be obtained by inputting the preset bidirectional convolutional long short-term memory module to the forward convolution unit and the backward convolution unit in sequence and processing the feature of each time step based on a preset cycle step, and the cross-time dynamic feature F can be obtained by fusing the forward feature sequence and the backward feature sequence. T :

[0120] F T = BCLSTM(f V ).

[0121] It should be noted that the preset cycle step is preferably set to 5, that is, the module can simultaneously aggregate event stream data of the past and future 5 time steps in each time sequence processing, thereby more comprehensively modeling the time dependence of the dynamic scene.

[0122] Optionally, in some embodiments, the event frame in the form of a multi-channel tensor and a preset image frame are input to a preset multi-scale spatial enhancement module to obtain spatial complementary features of different scales, including: extracting a dynamic feature from the event frame in the form of a multi-channel tensor and a static feature from the preset image frame; and inputting the dynamic feature and the static feature to the preset multi-scale spatial enhancement module for connection operation to obtain spatial complementary features of different scales.

[0123] Specifically, since the present application is committed to event stream-based video reconstruction, only 5 images are reconstructed each time, so the last reconstructed image frame I L (i.e. the preset image frame) of the last time can be used to extract static features, and dynamic features are extracted from the event frame V ε in the form of a multi-channel tensor, the dynamic features and the static features are taken as inputs of a preset multi-scale spatial enhancement module, the dynamic features and the static features are fused using a connection operation, thereby obtaining spatial complementary features F S of different scales. For example, if the preset multi-scale spatial enhancement module considers 3 scales, 3 spatial complementary features of corresponding scales can be obtained.

[0124] F S = MSSE(Cat(V ε , I L ));

[0125] Where Cat is a connection operation.

[0126] For example, if the preset multi-scale spatial enhancement module considers 3 scales {1, 1 / 2, 1 / 4} (‘1’ represents the original resolution scale, ‘1 / 2’ represents reducing the spatial resolution to 1 / 2 of the original image, and ‘1 / 4’ represents reducing the spatial resolution to 1 / 4 of the original image), when the input needs to be down-sampled, the nearest neighbor interpolation can be performed on the last connection result to obtain the down-sampled result, for example, when down-sampling from the 1 / 2 scale to the 1 / 4 scale, the feature map resolution is halved. Nearest neighbor interpolation is an image processing technique used to generate new pixel values by finding the nearest pixel points and assigning their values to the new pixel points, thereby adjusting the image size. After this process, dynamic and static spatial complementary features under multiple scales, i.e. spatial complementary features F S of different scales, can be obtained.

[0127] Optionally, in some embodiments, the dynamic features across time and the spatial complementary features of different scales are input into a preset spatio-temporal fusion attention module to obtain fusion features of different scales, including: performing layer regularization and 1x1 convolution operation on the dynamic features across time and the spatial complementary features of different scales using the preset spatio-temporal fusion attention module to obtain a query matrix, a key matrix and a value matrix; calculating an attention matrix based on the query matrix, the key matrix and the value matrix; multiplying the attention matrix and the value matrix, and adding the obtained product result to the spatial complementary features of different scales to obtain a sum result; inputting the sum result into a normalization layer and a multi-layer perceptron in sequence to obtain the fusion features of different scales.

[0128] It can be understood that layer regularization is a technique in deep learning for preventing overfitting during model training by adding a regularization term to constrain the complexity of the model. 1x1 convolution is a special convolution operation that can linearly transform features without changing the size of the feature map, and is commonly used to adjust the number of channels. The normalization layer (Layer Normalization, LN for short) is used to adjust the data distribution to have certain statistical properties, such as a mean of 0 and a standard deviation of 1, which helps the stability and convergence speed of the model. Multi-layer perception (Multi-Layer Perceptro, MLP for short) is a simple feedforward neural network that usually contains multiple fully connected layers for processing complex nonlinear relationships.

[0129] Specifically, as shown in Figure 5 this stage can take the dynamic features F T across time and the spatial complementary features D S of different scales as inputs of a preset spatio-temporal fusion attention module, perform layer regularization and 1x1 convolution operation, so as to obtain a query matrix Q∈R hw×c (representing features that need to be focused on), a key matrix K∈R hw×c (representing features that can be focused on) and a value matrix V∈R hw×c (representing the value of the features), where h is the height of the matrix, w is the width of the matrix, and c is the number of channels of the matrix. Then, based on the three matrices, an attention matrix Attention(Q,K,V) is calculated, which reflects the relevance between the features:

[0130]

[0131] where d k is the dimension size of the value matrix, which is set to c here.

[0132] Then, the attention matrix is multiplied with the value matrix to obtain a weighted value matrix, and the weighted value matrix is added to the spatial complementary features of different scales. The result after addition is sequentially subjected to a normalization layer and a multi-layer perception to obtain the fusion features F of different scales:

[0133] F=(LN(Attention(Q,K,V)V+F S ))。

[0134] Optionally, in some embodiments, the fusion features of different scales are input into a preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales, including: performing upsampling or downsampling operation on the fusion features of different scales to obtain fusion features of a preset scale; inputting the fusion features of the preset scale into the preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales.

[0135] Specifically, according to the requirement of the current scale, the fusion features F S of different scales can be upsampled or downsampled to the current scale to obtain fusion features of a preset scale (the preset scale is the same as the current scale), and then the fusion features of the preset scale are input into the preset multi-scale feature aggregation module to obtain multi-scale enhanced features F MSFA of different scales.

[0136] For example, when the preset scale is 1:

[0137]

[0138] When the preset scale is 1 / 2:

[0139]

[0140] Wherein, ↑ is the upsampling operation, and ↓ is the downsampling operation.

[0141] It should be noted that when the preset scale is 1 / 4, the multi-scale enhanced feature F MSFA of the scale of 1 / 4 can first pass through a residual block and then pass through a 3x3 convolution layer to obtain a reconstruction result of the scale of 1 / 4; when the preset scale is 1 / 2 or 1, the output feature of the residual block at the previous scale can be first upsampled, and then the upsampled result is connected with the multi-scale enhanced feature F MSFA of the current scale, sequentially passes through a 1x1 convolution layer, a residual block and a 3x3 convolution layer to obtain a reconstruction result of the current scale.

[0142] Next, how to obtain the preset convolutional neural network model is described in detail.

[0143] As a possible implementation manner, in some embodiments, before inputting the event frame in the form of a multi-channel tensor into the preset convolutional neural network model, further comprising: obtaining event stream data and corresponding clear image sequences of a plurality of historical dynamic scenes, and converting the event stream data of the plurality of historical dynamic scenes into historical event frames in the form of a multi-channel tensor; inputting the historical event frames as a training set into the preset convolutional neural network, training the preset convolutional neural network to obtain an initial convolutional neural network model; verifying the initial convolutional neural network model using the corresponding clear image sequences as a verification set, until the initial convolutional neural network model meets the preset standard, ending the iterative training of the preset convolutional neural network to obtain the preset convolutional neural network model, otherwise, based on the preset optimization algorithm, adjusting the model parameters through back propagation and continuing the iterative training.

[0144] Specifically, the event camera and the traditional camera can be used to capture event stream data and clear images under a plurality of historical complex dynamic scenes, and a data set can be constructed based on the event stream data and the corresponding clear image sequences of the plurality of historical dynamic scenes, for training to obtain the preset convolutional neural network model. Before the event stream data of the historical dynamic scene is used for model training, the event stream data of the plurality of historical dynamic scenes can also be converted into historical event frames in the form of a multi-channel tensor. A series of historical event frames are inputted as a training set into the preset convolutional neural network, and the preset convolutional neural network is trained using a supervised framework, so as to obtain an initial convolutional neural network model; then, a series of corresponding clear image sequences are used as a verification set to verify the initial convolutional neural network model. If the initial convolutional neural network model can meet the preset standard, the iterative training of the preset convolutional neural network can be ended to obtain the preset convolutional neural network model. If the model fails to meet the preset standard, the model parameters can be adjusted through back propagation based on the preset optimization algorithm (such as AdamW optimization algorithm, which is a commonly used optimization algorithm in the field of machine learning and deep learning, used for training neural networks. It combines the adaptive moment estimation (Adam) algorithm and the weight decay technique, aiming to improve the efficiency and performance of model training), and then the iterative training is continued until a model meeting the preset standard (i.e., the preset convolutional neural network model) is obtained.

[0145] In the process of training the preset convolutional neural network using a supervised framework, the full-supervised loss is:

[0146]

[0147] wherein, I k is the lth predicted image frame output by the model; is the real first image; is the L1 loss under multi-scale; is the similarity loss under multi-scale; is the perceptual loss under multi-scale; λ1, λ2 are weight factors, both of which are set to 1 in the embodiments of the present application; L is the number of continuous images of the video in the training stage, which is preferably set to 10 in the embodiments of the present application.

[0148] Further, in the process of adjusting the parameters of the model based on the preset optimization algorithm (such as AdamW optimization algorithm), the first moment estimation and the second moment estimation of the gradient are calculated to design independent adaptive learning rates for different parameters. This method allows the model to more accurately adjust the parameters of the neural network convolutional layer when iteratively updating based on the training data, thereby improving the training efficiency and performance of the model.

[0149] In step S103, the video data of the target dynamic scene is reconstructed based on the cross-scale predicted image frames.

[0150] Specifically, after obtaining the cross-scale predicted image frames, the video data of the target dynamic scene can be reconstructed based on the cross-scale prediction technology. The core of this method is to utilize information of different scales to improve the accuracy of prediction and the quality of reconstructed images. In this way, the dynamic changes in the target scene can be better understood and simulated, thereby providing more abundant and accurate data support for video analysis, computer vision and related fields.

[0151] To further help those skilled in the art to better understand the event stream-based video reconstruction method proposed in the embodiments of the present application, the effectiveness of the present application is demonstrated below in combination with the reconstruction comparison results of the method (ST-E2V (Spatio-Temporal Event-to-Video, Spatio-Temporal Event-to-Video)) of the present application and other methods (such as Figure 6 as shown).

[0152] Among them, the other methods are event stream reconstruction video methods based on deep learning and supervision, including E2VID+ (event camera to video method), SPADE-E2VID (spatially-adaptive denormalization for event-based video reconstruction, a spatially adaptive reconstruction method), SSL-E2VID (self-supervised learning for event-to-video reconstruction, a self-supervised learning method for event stream reconstruction video), ET-Net (event-time neural network, an event transformer reconstruction method), and HyperE2VID (hypernetworks for event-based video reconstruction, a hypernetwork reconstruction method).

[0153] The video reconstruction results of the above six models on the dataset HQF and the dataset BS-ERGB are shown in Table 1:

[0154] Table 1

[0155]

[0156] By measuring the quantitative indicators under the same dataset, including mean square error (MSE), structural similarity (SSIM), and learned perceptual image patch similarity (LPIPS), the specific definitions are as follows:

[0157]

[0158]

[0159] Wherein, μ k is the mean of the predicted image frame I k , μ k,GT is the mean of the real image , is the variance of the predicted image frame I k , is the variance of the real image , σ is the covariance of the image I k and , and c1, c2 are both very small constants (used to avoid the case that the denominator is zero).

[0160] It can be understood that the higher the structural similarity SSIM value is, the lower the mean square error MSE and the learnable perceptual image patch similarity LPIPS value is, and the better the video reconstruction performance is. As shown in the numerical results in Table 1, the embodiments of the present application perform better in reconstructing video from event stream, and also verify the effectiveness of the event stream-based video reconstruction method proposed in the embodiments of the present application.

[0161] According to the event stream-based video reconstruction method proposed in the embodiments of the present application, the features of the event stream data of the target dynamic scene are extracted by a multi-scale multi-output multi-input neural network, a bidirectional convolutional long short-term memory module is proposed, which can integrate past and future event information at the same time, a multi-scale spatial enhancement module is proposed, which is used to extract multi-scale complementary spatial features from events and from the reconstructed image of the previous stage, and a spatio-temporal fusion attention module is proposed to integrate spatial features at different time scales. Thus, high-quality video reconstruction is achieved by recording continuous event stream in a dynamic scene, and the image blur and fog-like artifact problems caused by nonlinear time information and non-uniform spatial distribution in a complex dynamic scene are solved, thereby meeting the demand for the development of video reconstruction to higher image quality and more robustness.

[0162] Secondly, the event stream-based video reconstruction device according to the embodiments of the present application is described with reference to the accompanying drawings.

[0163] Figure 7 is a block schematic diagram of the event stream-based video reconstruction device of an embodiment of the present application.

[0164] As shown in Figure 7 , the event stream-based video reconstruction device 10 comprises a conversion module 100, an obtaining module 200 and a reconstruction module 300.

[0165] The conversion module 100 is configured to obtain event stream data of a target dynamic scene and convert the event stream data into event frames in the form of a multi-channel tensor.

[0166] The obtaining module 200 is configured to input the event frames in the form of a multi-channel tensor into a preset convolutional neural network model to output cross-scale predicted image frames, wherein the preset convolutional neural network model is trained by historical event frames, and the preset convolutional neural network model comprises a preset bidirectional convolutional long short-term memory module, a preset multi-scale spatial enhancement module, a preset spatio-temporal fusion attention module and a preset multi-scale feature aggregation module.

[0167] The reconstruction module 300 is configured to reconstruct video data of the target dynamic scene based on the cross-scale predicted image frames.

[0168] Further, in some embodiments, the conversion module 100 is specifically used for:

[0169] creating a full zero matrix of a preset dimension;

[0170] dividing a time range of the event stream data into a plurality of sub-time intervals based on the preset dimension;

[0171] determining a normalization interval based on a number of the plurality of sub-time intervals, and normalizing a timestamp of each event in the event stream data to the normalization interval to obtain each normalized new event timestamp;

[0172] according to each normalized new event timestamp, interpolating each normalized event into the full zero matrix by using a bilinear interpolation strategy with an event polarity as a weight to obtain an event frame in a multi-channel tensor form.

[0173] Further, in some embodiments, the obtaining module 200 includes:

[0174] a first obtaining unit, configured to pre-process the event frame in the multi-channel tensor form, and input the pre-processed event frame into a preset bidirectional convolutional long short-term memory module to obtain a cross-time dynamic feature;

[0175] a second obtaining unit, configured to input the event frame in the multi-channel tensor form and a preset image frame into a preset multi-scale spatial enhancement module to obtain spatial complementary features of different scales;

[0176] a third obtaining unit, configured to input the cross-time dynamic feature and the spatial complementary features of different scales into a preset spatio-temporal fusion attention module to obtain fusion features of different scales;

[0177] a fourth obtaining unit, configured to input the fusion features of different scales into a preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales, and obtain a cross-scale predicted image frame based on the multi-scale enhanced features of different scales.

[0178] Further, in some embodiments, the obtaining module 200 is specifically used for:

[0179] sequentially inputting the event frame in the multi-channel tensor form into a 3x3 convolutional layer, an activation layer and a residual block to obtain the pre-processed event frame.

[0180] Further, in some embodiments, the first obtaining unit is specifically used for:

[0181] inputting the pre-processed event frame into a forward convolution unit in the preset bidirectional convolutional long short-term memory module, processing a feature of each time step based on a preset cycle step to generate a forward feature sequence;

[0182] input the preprocessed event frame into a backward convolution unit in a preset bidirectional convolution long short-term memory module, process features of each time step based on a preset cycle step, and generate a backward feature sequence;

[0183] fuse the forward feature sequence and the backward feature sequence to obtain dynamic features across time.

[0184] Further, in some embodiments, the second obtaining unit is specifically configured to:

[0185] extract dynamic features from the event frame in the form of a multi-channel tensor and extract static features from a preset image frame;

[0186] input the dynamic features and the static features into a preset multi-scale spatial enhancement module for connection operation to obtain spatial complementary features of different scales.

[0187] Further, in some embodiments, the third obtaining unit is specifically configured to:

[0188] perform layer regularization and 1x1 convolution operation on the dynamic features across time and the spatial complementary features of different scales by using a preset spatio-temporal fusion attention module to obtain a query matrix, a key matrix, and a value matrix;

[0189] calculate an attention matrix based on the query matrix, the key matrix, and the value matrix;

[0190] multiply the attention matrix and the value matrix, add a product result obtained to the spatial complementary features of different scales, and obtain a sum result;

[0191] input the sum result into a normalization layer and a multi-layer perception machine in sequence to obtain fusion features of different scales.

[0192] Further, in some embodiments, the fourth obtaining unit is specifically configured to:

[0193] perform up-sampling or down-sampling operation on the fusion features of different scales to obtain fusion features of a preset scale;

[0194] input the fusion features of the preset scale into a preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales.

[0195] Further, in some embodiments, before inputting the event frame in the form of a multi-channel tensor into the preset convolutional neural network model, the obtaining module is further configured to:

[0196] input historical event frames as a training set into a preset convolutional neural network, train the preset convolutional neural network to obtain an initial convolutional neural network model;

[0197] The initial convolutional neural network model is verified by using a corresponding clear image sequence as a verification set until the initial convolutional neural network model meets a preset standard, and the iteration training of the preset convolutional neural network is ended to obtain the preset convolutional neural network model, or the model parameters are adjusted through back propagation based on a preset optimization algorithm, and the iteration training is continued.

[0198] It should be noted that the foregoing explanation of the embodiment of the event stream based video reconstruction method is also applicable to the embodiment of the event stream based video reconstruction device, which will not be described here again.

[0199] The event stream based video reconstruction device provided by the embodiment of the present application extracts the features of the event stream data of the target dynamic scene through a multi-scale multi-output multi-input neural network, proposes a bidirectional convolutional long short-term memory module that can integrate past and future event information at the same time, proposes a multi-scale spatial enhancement module for extracting multi-scale complementary spatial features from events and from the reconstructed image of the previous stage, and proposes a spatio-temporal fusion attention module for integrating spatial features at different time scales. Thus, high-quality video reconstruction is achieved by recording continuous event streams in a dynamic scene, and the image blur and fog-like artifact problems caused by nonlinear time information and non-uniform spatial distribution in a complex dynamic scene are solved, thereby meeting the demand for the development of video reconstruction to higher image quality and greater robustness.

[0200] Figure 8 The structure schematic diagram of the electronic device provided by the embodiment of the present application is provided. The electronic device can include:

[0201] The memory 801, the processor 802, and the computer program stored in the memory 801 and executable on the processor 802.

[0202] The processor 802 implements the event stream based video reconstruction method provided in the above embodiments when executing the program.

[0203] Further, the electronic device further includes:

[0204] The communication interface 803 is used for communication between the memory 801 and the processor 802.

[0205] The memory 801 is used to store the computer program executable on the processor 802.

[0206] The memory 801 can include a high-speed RAM (Random Access Memory, random access memory) memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0207] If the memory 801, the processor 802 and the communication interface 803 are implemented independently, the communication interface 803, the memory 801 and the processor 802 can be connected with each other through a bus and complete communication between each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 8 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0208] Optionally, in a specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on a chip, the memory 801, the processor 802 and the communication interface 803 can complete communication between each other through an internal interface.

[0209] The processor 802 can be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0210] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the video reconstruction method based on an event stream as above.

[0211] The embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the video reconstruction method based on an event stream as above.

[0212] In addition, the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0213] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are contained in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples without contradiction.

[0214] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present application, and the person skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for video reconstruction based on event stream, characterized in that, The method comprises the following steps: obtaining event stream data of a target dynamic scene and converting the event stream data into an event frame in the form of a multi-channel tensor; inputting the event frame in the form of a multi-channel tensor into a preset convolutional neural network model to output a cross-scale predicted image frame, wherein the preset convolutional neural network model is trained by historical event frames, and the preset convolutional neural network model comprises a preset bidirectional convolutional long short-term memory module, a preset multi-scale spatial enhancement module, a preset spatio-temporal fusion attention module and a preset multi-scale feature aggregation module; reconstructing video data of the target dynamic scene based on the cross-scale predicted image frame.

2. The method of claim 1, wherein, The conversion of the event stream data into an event frame in the form of a multi-channel tensor comprises: creating a full zero matrix of a preset dimension; dividing a time range of the event stream data into a plurality of sub-time intervals uniformly based on the preset dimension; determining a normalization interval based on the number of the plurality of sub-time intervals, and normalizing a timestamp of each event in the event stream data to the normalization interval to obtain each normalized new event timestamp; according to the each normalized new event timestamp, interpolating each normalized event into the full zero matrix by using a bilinear interpolation strategy with event polarity as a weight to obtain the event frame in the form of a multi-channel tensor.

3. The method of claim 1, wherein, The inputting of the event frame in the form of a multi-channel tensor into a preset convolutional neural network to output a cross-scale predicted image frame comprises: preprocessing the event frame in the form of a multi-channel tensor and inputting the preprocessed event frame into the preset bidirectional convolutional long short-term memory module to obtain a cross-time dynamic feature; inputting the event frame in the form of a multi-channel tensor and a preset image frame into the preset multi-scale spatial enhancement module to obtain spatial complementary features of different scales; inputting the cross-time dynamic feature and the spatial complementary features of different scales into the preset spatio-temporal fusion attention module to obtain fusion features of different scales; inputting the fusion features of different scales into the preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales, and obtaining the cross-scale predicted image frame based on the multi-scale enhanced features of different scales.

4. The method of claim 3, wherein, The preprocessing of the event frame in the form of a multi-channel tensor comprises: sequentially inputting the event frame in the form of a multi-channel tensor into a 3*3 convolutional layer, an activation layer and a residual block to obtain the preprocessed event frame.

5. The method of claim 3, wherein, The inputting of the preprocessed event frame into the preset bidirectional convolutional long short-term memory module to obtain a cross-time dynamic feature comprises: inputting the preprocessed event frame into a forward convolutional unit in the preset bidirectional convolutional long short-term memory module, processing a feature of each time step based on a preset cycle step to generate a forward feature sequence; inputting the preprocessed event frame into a backward convolutional unit in the preset bidirectional convolutional long short-term memory module, processing the feature of each time step based on the preset cycle step to generate a backward feature sequence; fuse the forward feature sequence and the backward feature sequence to obtain the dynamic feature across time.

6. The method of claim 3, wherein, The event frame in the form of a multi-channel tensor and a preset image frame are input into the preset multi-scale spatial enhancement module to obtain spatial complementary features of different scales, including: extracting dynamic features from the event frame in the form of a multi-channel tensor and static features from the preset image frame; the dynamic features and the static features are input into the preset multi-scale spatial enhancement module for connection operation to obtain the spatial complementary features of different scales.

7. The method of claim 3, wherein, The dynamic feature across time and the spatial complementary features of different scales are input into the preset spatio-temporal fusion attention module to obtain fusion features of different scales, including: The spatio-temporal fusion attention module is used to perform layer regularization and 1x1 convolution operation on the dynamic feature across time and the spatial complementary features of different scales to obtain a query matrix, a key matrix and a value matrix; An attention matrix is calculated based on the query matrix, the key matrix and the value matrix; The attention matrix and the value matrix are multiplied, and the product result is added to the spatial complementary features of different scales to obtain a sum result; The sum result is input into a normalization layer and a multi-layer perception in sequence to obtain the fusion features of different scales.

8. The method of claim 3, wherein, The fusion features of different scales are input into the preset multi-scale feature aggregation module to obtain multi-scale enhanced features of different scales, including: The fusion features of different scales are upsampled or downsampled to obtain fusion features of a preset scale; The fusion features of the preset scale are input into the preset multi-scale feature aggregation module to obtain the multi-scale enhanced features of different scales.

9. The method of claim 1, wherein, Before the event frame in the form of a multi-channel tensor is input into the preset convolutional neural network model, the method further includes: obtaining event stream data of a plurality of historical dynamic scenes and corresponding clear image sequences, and converting the event stream data of the plurality of historical dynamic scenes into historical event frames in the form of a multi-channel tensor; inputting the historical event frames as a training set into a preset convolutional neural network to train the preset convolutional neural network to obtain an initial convolutional neural network model; verify the initial convolutional neural network model using the corresponding clear image sequences as a verification set until the initial convolutional neural network model meets a preset standard, ending the iterative training of the preset convolutional neural network to obtain the preset convolutional neural network model, otherwise, based on a preset optimization algorithm, adjusting the model parameters through back propagation and continuing the iterative training.

10. A video reconstruction device based on event stream, characterized in that, including: a conversion module configured to obtain event stream data of a target dynamic scene and convert the event stream data into an event frame in the form of a multi-channel tensor; The obtaining module is configured to input the event frame in the form of the multi-channel tensor into a preset convolutional neural network model, and output a cross-scale predicted image frame, wherein the preset convolutional neural network model is trained by historical event frames, and the preset convolutional neural network model comprises a preset bidirectional convolutional long short-term memory module, a preset multi-scale spatial enhancement module, a preset spatio-temporal fusion attention module and a preset multi-scale feature aggregation module. The reconstruction module is configured to reconstruct video data of the target dynamic scene based on the cross-scale predicted image frame.

Images