A video reconstruction method and system based on asynchronous event streams

Abstract

The application provides a video reconstruction method and system based on an asynchronous event stream, comprising: acquiring an asynchronous event stream which needs to be reconstructed into a video, and converting the asynchronous event stream into an intermediate representation; inputting the intermediate representation into a pre-constructed asynchronous event stream image reconstruction system to obtain a gray image sequence; and connecting each gray image in the gray image sequence in time sequence to form a video; in the asynchronous event stream image reconstruction system provided by the application, an event fusion module integrates key information in a period of time, a down-sampling module adopts a cyclic structure to enhance the connection between frames, and a skip connection is deployed with a structure for enhancing important information and suppressing noise; the application integrates key information at each moment, reproduces the light intensity of silent pixel positions, removes artifacts and noise, and can improve the quality of the reconstructed video.

Description

Technical Field

[0001] This invention belongs to the field of image and video reconstruction technology, specifically relating to a video reconstruction method and system based on asynchronous event streams. Background Technology

[0002] Asynchronous event streams, as the output of novel dynamic vision sensors, have a format drastically different from images captured by traditional standard cameras. The design of these novel dynamic vision sensors is inspired by the human eye's greater sensitivity to change, hence the name "event camera." To capture changes in light intensity, the event camera asynchronously records each change as an event. When the intensity change of a pixel exceeds a predefined threshold, an event is output. The predefined threshold determines the frequency of events occurring under similar conditions. Each event includes the pixel location, a timestamp, and a polarity value indicating whether the light intensity is increasing or decreasing. Events are independent of each other, and the collection of all events generated over a period of time forms an asynchronous, sparse event stream. Compared to traditional cameras capturing images at a fixed frame rate, this shooting method brings substantial advantages to event cameras, such as low power consumption, high dynamic range, no motion blur, low latency, and microsecond temporal resolution. In challenging environments, such as high-speed motion and extreme lighting conditions, these characteristics enable them to have a more complete receptive field and collect more comprehensive information. Currently, event cameras have broad application prospects in feature extraction and tracking, 3D reconstruction, and simultaneous localization and mapping.

[0003] An event is a four-dimensional information, which cannot be densely visualized in two-dimensional space without loss of quality. While event streams are not redundant, they still suffer from information loss, lacking information about silent pixel regions (pixel locations where no events occurred) synchronized with the event stream, hindering scene analysis and processing. In contrast, standard cameras are well-developed, outputting images with rich and specific color information, allowing for the immediate observation of various details. More importantly, current visual algorithms are mostly based on image frames. Due to the unique output data of event cameras, traditional visual algorithms are difficult to directly apply to novel event cameras, severely limiting the development of event camera-related applications. Therefore, fully leveraging the advantages of event cameras, especially in challenging environments, to transform asynchronous event stream data—which is difficult to analyze intuitively—into easily understandable high-quality images is an important research direction. Image reconstruction based on asynchronous event streams will build a bridge between event camera vision and traditional computer vision, providing a new paradigm for the application of classic algorithms such as object recognition, object detection, and object tracking in event cameras, expanding the application scope of event cameras, and allowing for the improvement and optimization of existing traditional visual algorithms by utilizing the uniqueness of events, which is of great significance.

[0004] Extensive research has been conducted on converting asynchronous event streams captured by event cameras into standard grayscale images. Most researchers previously believed that specific algorithms were needed to accommodate the asynchronous sparsity of event data. Model-based methods employed joint optimization of optical flow and intensity, temporal filtering, channel regularization, and frame-event integration to build complex and diverse models, reconstructing images using multiple visual parameters and camera motion parameters. However, some models restrict the event camera's environment and motion, such as allowing only rotation, and the image quality generated by traditional methods has significant room for improvement. Learning-based methods can generate grayscale images under any circumstances, achieving state-of-the-art results with more natural and realistic images of higher quality, effectively recovering content. However, these methods cannot directly accept event streams as input; they require intermediate representations to be accepted by deep learning frameworks. Among current methods for reconstructing video based on event streams, the E2VID method, published in the 2019 IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) entitled "High speed and high dynamic range video with an event camera," built an end-to-end U-shaped network framework and achieved good results. However, artifacts and other problems still exist in the generated results. The E2VID+ method presented at the 2020 European Conference on Computer Vision, entitled "Reducing the Sim-to-Real Gap for Event Cameras," adjusted the training strategy based on the E2VID method and obtained better results, but there is still room for improvement. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention proposes a video reconstruction method based on asynchronous event streams, comprising:

[0006] Obtain the asynchronous event stream that needs to be reconstructed into a video, and convert the asynchronous event stream into an intermediate representation;

[0007] The intermediate representation is input into a pre-constructed asynchronous event stream image reconstruction system to obtain a grayscale image sequence;

[0008] A video is constructed by connecting grayscale images in a grayscale image sequence in chronological order.

[0009] The asynchronous event stream image reconstruction system is composed of an event fusion module that integrates key information over a period of time, a downsampling module that uses a loop structure to enhance the connection between frames, a noise-suppressing skip connection, and an upsampling module that reduces channels to restore image size; each intermediate representation is calculated from all events in an event stream.

[0010] Preferably, the construction of the asynchronous event stream image reconstruction system includes:

[0011] An asynchronous event stream image reconstruction system is constructed, comprising an event fusion module, a downsampling module, a residual block, an upsampling module, and a tail, all connected sequentially. The downsampling module includes multiple sequentially connected downsampling structures, and the upsampling module includes multiple sequentially connected upsampling structures, with the number of upsampling structures equal to the number of downsampling structures. The downsampling and upsampling structures are symmetrically arranged around the residual block. Each downsampling structure also transmits its output data through an attention mechanism layer to the symmetrically distributed upsampling structures via skip connections. The event fusion module also transmits its output data to the tail via skip connections. The event fusion module receives and fuses intermediate representations of a preset duration before and after the current time. The downsampling module performs downsampling, the residual block performs residual calculation, the upsampling module performs upsampling, and the tail outputs a grayscale image.

[0012] Based on the COCO dataset and a selected foreground image set, the ESIM method is used to simulate the asynchronous event stream and corresponding grayscale video for training.

[0013] Convert the asynchronous event stream used for training into an intermediate representation used for training;

[0014] Using the intermediate representation used for training as input and each grayscale image in the corresponding grayscale video as output, the structure is trained to obtain an asynchronous event stream image reconstruction system.

[0015] Preferably, the event fusion module includes: a shared convolutional layer, a global pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer;

[0016] The shared convolutional layer is used to process the mediation representations before and after the current time for a preset duration, to obtain the first feature map corresponding to the mediation representations before the current time for a preset duration and the second feature map corresponding to the mediation representations after the current time for a preset duration.

[0017] The global pooling layer is used to reduce the dimensionality of the data after summing the corresponding pixels of the first feature map and the second feature map to obtain the first attention map, and then input the first attention map into the first fully connected layer.

[0018] The first fully connected layer is used to reduce the number of channels in the first attention map to obtain a second attention map, and the second attention map is input into the second fully connected layer and the third fully connected layer respectively;

[0019] The second fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the first feature map;

[0020] The third fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the second feature map;

[0021] The output of the event fusion module is the sum of the product of the first feature map and its corresponding attention map plus the product of the second feature map and its corresponding attention map.

[0022] Preferably, the downsampling structure includes a convolutional layer, an activation layer, and a long short-term memory convolutional layer connected in sequence;

[0023] The convolutional layer is used to input data, and the long short-time memory convolutional layer is used to output data.

[0024] The long short-term memory convolutional layer contains a state graph of current features that enhances the connection between adjacent frames.

[0025] Preferably, the residual block comprises two residual structures connected in sequence;

[0026] Each of the residual structures includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence;

[0027] The input data of the first convolutional layer is also added to the output data of the second convolutional layer through skip connections and then input to the second activation layer;

[0028] The first convolutional layer of the previous residual structure is used to obtain the input data of the residual block, and the second activation layer of the next residual structure is used to output the output data of the residual block.

[0029] Preferably, the upsampling structure includes: a first layer, a second layer, a third layer, and a fourth layer connected in sequence; the first layer is used for input data, and the fourth layer is used for output data;

[0030] The first layer is a bilinear interpolation layer;

[0031] The second layer consists of two parallel convolutional layers;

[0032] The third layer is the attention mechanism layer;

[0033] The fourth layer is the activation layer.

[0034] Preferably, the tail portion is a convolutional layer.

[0035] Preferably, the attention mechanism layer includes a channel domain and a spatial domain;

[0036] The channel domain includes a global pooling layer, two convolutional layers, and an activation layer connected in sequence.

[0037] The spatial domain includes convolutional layers and activation layers connected in sequence;

[0038] The global pooling layer in the channel domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the channel domain is used to output the channel domain attention map.

[0039] The convolutional layer in the spatial domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the spatial domain is used to output the spatial domain attention map.

[0040] The channel domain attention map and the spatial domain attention map are multiplied by the feature map of the input attention mechanism layer and then summed to obtain the feature map output by the attention mechanism layer.

[0041] Preferably, the computational formula for converting the asynchronous event stream into an intermediate representation is as follows:

[0042]

[0043] In the formula, V b∈[0,B-1] Let p represent the value at the b-th channel position in the asynchronous event stream within the mediated representation, where B is the total number of channels in the mediated representation, N is the number of asynchronous events in the mediated representation, and p i Let the polarity of the i-th asynchronous event in the mediation representation be denoted as . Let be the weight of the i-th asynchronous event in the mediation representation;

[0044] The weight of the i-th asynchronous event in the mediation representation The calculation formula is as follows:

[0045]

[0046] In the formula, t i Let t be the timestamp corresponding to the i-th asynchronous event in the mediation representation. min (ε T ) represents the smallest timestamp in the event stream of a preset duration prior to the current moment, t max (ε T ) represents the maximum timestamp in the event stream of a preset duration prior to the current moment, t max (ε T+1 ) represents the maximum timestamp in the event stream after the current moment and for a preset duration, t min (ε T+1 This represents the smallest timestamp in the event stream after the current moment and for a preset duration.

[0047] Based on the same inventive concept, this application also provides a video reconstruction system based on asynchronous event streams, including: a representation conversion module, an image conversion module, and a video reconstruction module;

[0048] The representation conversion module is used to acquire the asynchronous event stream that needs to be reconstructed into a video, and convert the asynchronous event stream into an intermediate representation;

[0049] The image conversion module is used to input the intermediate representation into a pre-constructed asynchronous event stream image reconstruction system to obtain a grayscale image sequence;

[0050] The video reconstruction module is used to connect the grayscale images in the grayscale image sequence in chronological order to form a video.

[0051] The asynchronous event stream image reconstruction system is composed of an event fusion module that integrates key information over a period of time, a downsampling module that uses a loop structure to enhance the connection between frames, a noise-suppressing skip connection, and an upsampling module that reduces channels to restore image size; each intermediate representation is calculated from all events in an event stream.

[0052] Preferably, the image conversion module constructs an asynchronous event stream image reconstruction system, including:

[0053] An asynchronous event stream image reconstruction system is constructed, comprising an event fusion module, a downsampling module, a residual block, an upsampling module, and a tail, all connected sequentially. The downsampling module includes multiple sequentially connected downsampling structures, and the upsampling module includes multiple sequentially connected upsampling structures, with the number of upsampling structures equal to the number of downsampling structures. The downsampling and upsampling structures are symmetrically arranged around the residual block. Each downsampling structure also transmits its output data through an attention mechanism layer to the symmetrically distributed upsampling structures via skip connections. The event fusion module also transmits its output data to the tail via skip connections. The event fusion module receives and fuses intermediate representations of a preset duration before and after the current time. The downsampling module performs downsampling, the residual block performs residual calculation, the upsampling module performs upsampling, and the tail outputs a grayscale image.

[0054] Based on the COCO dataset and a selected foreground image set, the ESIM method is used to simulate the asynchronous event stream and corresponding grayscale video for training.

[0055] Convert the asynchronous event stream used for training into an intermediate representation used for training;

[0056] Using the intermediate representation used for training as input and each grayscale image in the corresponding grayscale video as output, the structure is trained to obtain an asynchronous event stream image reconstruction system.

[0057] Preferably, the event fusion module in the image conversion module includes: a shared convolutional layer, a global pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer;

[0058] The shared convolutional layer is used to process the mediation representations before and after the current time for a preset duration, to obtain a first feature map corresponding to the mediation representations before the current time for a preset duration and a second feature map corresponding to the mediation representations after the current time for a preset duration.

[0059] The global pooling layer is used to reduce the dimensionality of the data after summing the corresponding pixels of the first feature map and the second feature map to obtain the first attention map, and then input the first attention map into the first fully connected layer.

[0060] The first fully connected layer is used to reduce the number of channels in the first attention map to obtain a second attention map, and the second attention map is input into the second fully connected layer and the third fully connected layer respectively;

[0061] The second fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the first feature map;

[0062] The third fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the second feature map;

[0063] The output of the event fusion module is the sum of the product of the first feature map and its corresponding attention map plus the product of the second feature map and its corresponding attention map.

[0064] Preferably, the downsampling structure in the image conversion module includes a convolutional layer, an activation layer, and a long short-term memory convolutional layer connected in sequence;

[0065] The convolutional layer is used to input data, and the long short-time memory convolutional layer is used to output data.

[0066] The long short-term memory convolutional layer contains a state graph of current features that enhances the connection between adjacent frames.

[0067] Preferably, the residual block in the image conversion module comprises two sequentially connected residual structures;

[0068] Each of the residual structures includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence;

[0069] The input data of the first convolutional layer is also added to the output data of the second convolutional layer through skip connections and then input to the second activation layer;

[0070] The first convolutional layer of the previous residual structure is used to obtain the input data of the residual block, and the second activation layer of the next residual structure is used to output the output data of the residual block.

[0071] Preferably, the upsampling structure in the image conversion module includes: a first layer, a second layer, a third layer, and a fourth layer connected in sequence; the first layer is used for input data, and the fourth layer is used for output data;

[0072] The first layer is a bilinear interpolation layer;

[0073] The second layer consists of two parallel convolutional layers;

[0074] The third layer is the attention mechanism layer;

[0075] The fourth layer is the activation layer.

[0076] Preferably, the tail portion of the image conversion module is a convolutional layer.

[0077] Preferably, the attention mechanism layer in the image conversion module includes a channel domain and a spatial domain;

[0078] The channel domain includes a global pooling layer, two convolutional layers, and an activation layer connected in sequence.

[0079] The spatial domain includes convolutional layers and activation layers connected in sequence;

[0080] The global pooling layer in the channel domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the channel domain is used to output the channel domain attention map.

[0081] The convolutional layer in the spatial domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the spatial domain is used to output the spatial domain attention map.

[0082] The channel domain attention map and the spatial domain attention map are multiplied by the feature map of the input attention mechanism layer and then summed to obtain the feature map output by the attention mechanism layer.

[0083] Preferably, the image conversion module or representation conversion module converts the asynchronous event stream into a computational formula for the intermediate representation as follows:

[0084]

[0085] In the formula, V b∈[0,B-1] Let p represent the value at the b-th channel position in the asynchronous event stream within the mediated representation, where B is the total number of channels in the mediated representation, N is the number of asynchronous events in the mediated representation, and p i Let the polarity of the i-th asynchronous event in the mediation representation be denoted as . Let be the weight of the i-th asynchronous event in the mediation representation;

[0086] The weight of the i-th asynchronous event in the mediation representation The calculation formula is as follows:

[0087]

[0088] In the formula, t i Let t be the timestamp corresponding to the i-th asynchronous event in the mediation representation. min (ε T ) represents the smallest timestamp in the event stream of a preset duration prior to the current moment, t max (ε T ) represents the maximum timestamp in the event stream of a preset duration prior to the current moment, t max (ε T+1 ) represents the maximum timestamp in the event stream after the current moment and for a preset duration, t min (ε T+1 This represents the smallest timestamp in the event stream after the current moment and for a preset duration.

[0089] Compared with the closest existing technology, the present invention has the following beneficial effects:

[0090] This invention provides a video reconstruction method and system based on asynchronous event streams, comprising: acquiring an asynchronous event stream to be reconstructed into a video, and converting the asynchronous event stream into an intermediate representation; inputting the intermediate representation into a pre-constructed asynchronous event stream image reconstruction system to obtain a grayscale image sequence; and connecting the grayscale images in the grayscale image sequence in chronological order to form a video. The asynchronous event stream image reconstruction system is composed of an event fusion module that integrates key information over a period of time, a downsampling module that uses a loop structure to enhance the connection between frames, a skip connection that suppresses noise, and an upsampling module that reduces channels to restore image size. Each intermediate representation is calculated from all events in an event stream. In the asynchronous event stream image reconstruction system provided by this invention, the event fusion module integrates key information over a period of time, the downsampling module uses a loop structure to enhance the connection between frames, and the skip connection deploys structures that enhance important information and suppress noise. This invention integrates key information at each moment, reproduces the light intensity at silent pixel locations, removes artifacts and noise, and can improve the quality of the reconstructed video. Attached Figure Description

[0091] Figure 1 A schematic diagram of a video reconstruction method based on asynchronous event streams provided by the present invention;

[0092] Figure 2 This invention provides a schematic diagram of the asynchronous event stream image reconstruction system structure in a video reconstruction method based on asynchronous event streams.

[0093] Figure 3 This is a schematic diagram of the event fusion module structure in the asynchronous event stream image reconstruction system provided by the present invention;

[0094] Figure 4 This is a schematic diagram of the downsampling structure in the asynchronous event stream image reconstruction system provided by the present invention;

[0095] Figure 5 This is a schematic diagram of the upsampling structure in the asynchronous event stream image reconstruction system provided by the present invention;

[0096] Figure 6 This is a schematic diagram of the attention mechanism layer structure in the asynchronous event stream image reconstruction system provided by the present invention;

[0097] Figure 7 This is a schematic diagram illustrating the event stream capture of the mediation representation involved in this invention;

[0098] Figure 8 This is a schematic diagram of a video reconstruction system based on asynchronous event streams provided by the present invention. Detailed Implementation

[0099] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0100] The technical problem to be solved by the present invention is that the images generated by the current event stream to image conversion method have many artifacts and insufficient content details.

[0101] The technical solution of this invention is a video reconstruction method and system based on asynchronous event streams. It adopts deep learning methods, uses reasonable intermediate representations, proposes a feature extraction module that integrates event stream information, and designs an end-to-end system framework to realize the conversion of event streams to video.

[0102] Example 1:

[0103] A schematic diagram of the video reconstruction method based on asynchronous event stream provided by this invention is shown below. Figure 1 As shown, it includes:

[0104] Step 1: Obtain the asynchronous event stream that needs to be reconstructed as video, and convert the asynchronous event stream into an intermediate representation;

[0105] Step 2: Input the mediation representation into the pre-built asynchronous event stream image reconstruction system to obtain a grayscale image sequence;

[0106] Step 3: Connect the grayscale images in the grayscale image sequence in chronological order to form a video;

[0107] The asynchronous event stream image reconstruction system consists of an event fusion module that integrates key information over a period of time, a downsampling module that enhances inter-frame connections using a loop structure, a noise-suppressing skip connection, and an upsampling module that reduces channels to restore image size; each intermediate representation is calculated from all events in an event stream.

[0108] Specifically, the present invention includes the following steps:

[0109] Before step 1, an asynchronous event-stream image reconstruction system needs to be constructed. The system structure is as follows: Figure 2 The system consists of an event fusion module, a downsampling module, a residual block, an upsampling module, and a tail section, with some components having jump connections between them.

[0110] Event fusion module, such as Figure 3 The diagram shows a structure incorporating an attention mechanism, consisting of a shared convolutional layer, three fully connected layers, and a global pooling layer, directly connected to the downsampling module. The shared convolutional layer contains 32 convolutional kernels, each 5×5 in size, with padding of 2 and a stride of 1. It extracts information from the input intermediate event representation data, expanding the channels to obtain a feature map of size 32×112×112 (channels × length × width). During system training and testing, two intermediate event representation data sets are processed separately, resulting in two feature maps x and y of size 32×112×112, denoted by C. The corresponding pixels of the two feature maps x and y sent by the shared convolutional layer are then directly added to obtain a feature map u combining both parts, i.e., u = x + y. Finally, a global average pooling (GAP) layer is used to reduce the dimensionality of feature map u, resulting in an attention map of size 32×1×1. Input this attention graph into a fully connected layer (FCL), reduce the number of channels C to make the feature size 16×1×1, and obtain z = FCL. C→C / 2 (GAP(u)). Then, the process splits into two routes, from global to local, to calculate the attention map A corresponding to the feature maps of the two inputs for each route. x A y Each part includes a fully connected layer, at which point the feature channels are restored to 32×1×1 before reduction, i.e., A. x =FCL C / 2→C (z), A y =FCL C / 2→C (z). Finally, the feature map of each part is multiplied by its respective attention map and then summed to obtain the event fusion feature EFF = A, which combines the two parts. x x+A y The size of y and EFF is the same as that of the input features, both being 32×112×112.

[0111] The downsampling module consists of three sequentially connected, functionally identical multi-layer recurrent convolutional downsampling structures, each as follows: Figure 4 As shown, the module includes one convolutional layer, one activation layer, and one long short-term memory convolutional layer (ConvLSTM, published in the 2015 issue of "Convolutional LSTM network: A machine learning approach for precipitation nowcasting" in the journal *Research Progress in Neural Information Processing Systems*). The entire module has nine layers and three outputs generated by three downsampling structures. The downsampling module reduces the width and height of the feature map while expanding the number of channels. Each long short-term memory convolutional layer contains a state map of the current feature, which enhances the connection between adjacent frames. The downsampling module is directly connected to the residual block and the skip connection.

[0112] The first downsampling structure consists of layers 1, 2, and 3. Layer 1, a convolutional layer, uses 64 5×5 convolutional kernels with a stride of 2 and padding of 2, transforming the input 32×112×112 event fusion feature map (EFF) into a 64×56×56 size. Layer 2 is an activation layer. Layer 3, a long short-term memory convolution, uses 256 3×3 convolutional kernels with a stride of 1 and a convolution size of 1, stacking the 64×56×56 feature map and the 64×56×56 state map as input, resulting in a 64×56×56 input gate feature map, a 64×56×56 forget gate feature map, a 64×56×56 output gate feature map, and a 64×56×56 state gate feature map. The state gate and output gate feature maps are retained, and the output gate data is used as the input for the next layer.

[0113] Similarly, the second downsampling structure consists of layers 4, 5, and 6. Layer 4, a convolutional layer, uses 128 5×5 convolutional kernels with a stride of 2 and padding of 2, transforming the input 64×56×56 feature map into a size of 128×28×28. Layer 5 is the activation layer. Layer 6, a Long Short-Term Memory (LST) convolutional layer, uses 512 3×3 convolutional kernels with a stride of 1 and a convolution value of 1. It stacks the 128×28×28 feature map and the 128×28×28 state map as input, resulting in a 128×28×28 input gate feature map, a 128×28×28 forget gate feature map, a 128×28×28 output gate feature map, and a 128×28×28 state gate feature map. The state gate and output gate feature maps are retained, and the output gate data is used as the input for the next layer.

[0114] Similarly, the third downsampling structure consists of layers 7, 8, and 9. Layer 7, a convolutional layer, uses 256 5×5 convolutional kernels with a stride of 2 and padding of 2, transforming the input 128×28×28 feature map into a size of 256×14×14. Layer 8 is the activation layer. Layer 9, a Long Short-Term Memory (LST) convolutional layer, uses 1024 3×3 convolutional kernels with a stride of 1 and a convolution value of 1. It stacks the 256×14×14 feature map and the 256×14×14 state map as input, resulting in a 256×14×14 input gate feature map, a 256×14×14 forget gate feature map, a 256×14×14 output gate feature map, and a 256×14×14 state gate feature map. The state gate and output gate feature maps are preserved, and the output gate data is used as input to the residual block.

[0115] The residual block contains two sequentially connected residual structures. Each structure includes two 3×3 convolutional layers, one skip connection, and two activation layers. The entire module has eight layers. The first residual structure comprises the first four layers, and the second residual structure comprises the last four layers. All convolutional layers in the residual block have a stride of 1 and are padded with 1s. The input and output feature sizes are both 256×14×14. Layers 1, 3, 5, and 7 are convolutional layers, and layers 2, 4, 6, and 8 are activation layers. The first skip connection directly adds the input of layer 1 to the output feature map of layer 3 as the input of layer 4. The second skip connection directly adds the input of layer 5 to the output feature map of layer 7 as the input of layer 8.

[0116] The upsampling module consists of three sequentially connected, functionally identical upsampling structures that restore features to their original size, reduce the channels expanded during the three downsampling processes, and restore the width and height dimensions. To preserve information lost due to the reduction in the number of channels, each part of the identical structure uses a bilinear interpolation layer connected to two different convolutional layers, followed by an attention mechanism layer SCSE (presented in "Concurrent spatial and channel 'squeeze & excitation' in fully convolutional networks" at the 2018 International Conference on Computational Medical Imaging and Computer-Aided Intervention) that combines channel and spatial domains, and an activation layer before output. The upsampling structure is as follows: Figure 5 As shown.

[0117] Specifically, the first upsampling structure consists of layers 1, 2, 3, and 4. Layer 1 is a bilinear interpolation layer that expands the 256×14×14 features to 256×28×28. Layer 2 consists of two parallel convolutional layers, one with 128 3×3 convolutional kernels and the other with 128 5×5 convolutional kernels, transforming the interpolated feature map into two 128×28×28 feature maps, which are then directly added together as input to the next layer. Layer 3 is the SCSE attention mechanism layer, containing two parallel domain processing layers, with a total of one global pooling layer, three convolutional layers, and two activation layers. The channel domain consists of one global pooling layer, two convolutional layers, and one activation layer sequentially linked together. The global pooling layer reduces the dimensionality of the 128×28×28 feature map to 128×1×1, resulting in a channel attention map. The first convolutional layer contains 64 1×1 kernels, reducing the 128×1×1 attention map to 64×1×1. The second convolutional layer contains 128 1×1 kernels, restoring the channel attention map to 128×1×1. The final activation layer remains unchanged. The spatial domain consists of a convolutional layer and an activation layer directly connected. The convolutional layer contains a single 1×1 kernel, generating a 1×28×28 spatial attention map from the input 128×28×28 feature map. The channel and spatial attention maps are multiplied by the feature map and then summed to obtain the adjusted 128×28×28 feature map. The fourth layer is the activation layer, and its output serves as part of the input for the next layer.

[0118] Similarly, the second upsampling structure consists of layers 5, 6, 7, and 8. Layer 5 of the upsampling module is a bilinear interpolation layer, expanding the 128×28×28 features to 128×56×56. Layer 6 consists of two parallel convolutional layers, one with 64 3×3 kernels and the other with 64 5×5 kernels, transforming the interpolated feature map into two 64×56×56 feature maps, which are then directly summed and used as input to the next layer. Layer 7 is the SCSE attention mechanism layer, containing two parallel domain processing layers. The channel domain consists of one global pooling layer, two convolutional layers, and one activation layer sequentially linked together. The global pooling layer reduces the dimensionality of the 64×56×56 feature map to 64×1×1, resulting in a channel attention map. The first convolutional layer contains 32 1×1 kernels, reducing the 64×1×1 attention map to 32×1×1. The second convolutional layer contains 64 1×1 kernels, restoring the channel attention map to 64×1×1. The final activation layer remains unchanged. The spatial domain consists of a convolutional layer and an activation layer directly connected. The convolutional layer contains one 1×1 kernel, generating a 1×56×56 spatial attention map from the input 64×56×56 feature map. The channel and spatial attention maps are multiplied by the feature map and then summed to obtain the adjusted 64×56×56 feature map. The 8th layer is an activation layer, and its output serves as part of the input for the next layer.

[0119] The third upsampling structure consists of layers 9, 10, 11, and 12. Layer 9 is a bilinear interpolation layer that expands the 64×56×56 features to 64×112×112. Layer 10 consists of two parallel convolutional layers, one with 32 3×3 convolutional kernels and the other with 32 5×5 convolutional kernels, transforming the interpolated feature map into two 32×112×112 feature maps, which are then directly added together as input to the next layer. Layer 11 is the SCSE attention mechanism layer, containing two parallel domain processing layers. The channel domain consists of one global pooling layer, two convolutional layers, and one activation layer sequentially linked together. The global pooling layer reduces the dimensionality of the 32×112×112 feature map to 32×1×1, resulting in a channel attention map. The first convolutional layer contains 16 1×1 kernels, reducing the 32×1×1 attention map to 16×1×1. The second convolutional layer contains 32 1×1 kernels, restoring the channel attention map to 32×1×1. The final activation layer remains unchanged. The spatial domain consists of a convolutional layer and an activation layer directly connected. The convolutional layer contains one 1×1 kernel, generating a 1×112×112 spatial attention map from the input 32×112×112 feature map. The channel and spatial attention maps are multiplied by the feature map respectively and then summed to obtain the adjusted 32×112×112 feature map. The 12th layer is an activation layer, and its output serves as part of the input for the next layer.

[0120] The tail is a convolutional layer containing a 1×1 convolutional kernel, which reduces the 32×112×112 feature map channels to generate a 1×112×112 reconstructed grayscale image.

[0121] The skip connection section connects the downsampling and upsampling parts and performs feature fusion, with a total of four channels.

[0122] In the first channel, the output feature map EFF of the event fusion module is directly added to the output feature map of the 12th layer of the upsampling module, and the fused feature is used as the input at the tail.

[0123] The second channel connects the output feature map of the third layer of the downsampling module and the output feature map of the eighth layer of the upsampling module for fusion. Before fusion, the output feature map of the third layer of the downsampling module needs to pass through an attention mechanism layer SCSE that combines the channel domain and the spatial domain. The structure of the attention mechanism layer is as follows: Figure 6 As shown, the channel domain consists of one global pooling layer, two convolutional layers, and one activation layer connected sequentially. The global pooling layer reduces the dimensionality of the 64×56×56 feature map to 64×1×1, resulting in a channel attention map. The first convolutional layer contains 32 1×1 kernels, reducing the 64×1×1 attention map to 32×1×1. The second convolutional layer contains 64 1×1 kernels, restoring the channel attention map to 64×1×1. The final activation layer remains unchanged. The spatial domain consists of a convolutional layer and an activation layer directly connected. The convolutional layer contains one 1×1 kernel, generating a 1×56×56 spatial attention map from the input 64×56×56 feature map. The attention maps in the channel and spatial domains are multiplied by the feature map and then summed to obtain an adjusted 64×56×56 feature map, which is used as the input to the 9th layer of the upsampling module.

[0124] The third channel connects the output feature map of layer 6 of the downsampling module and the output feature map of layer 4 of the upsampling module for fusion. Before fusion, the output feature map of layer 6 of the downsampling module needs to pass through an attention mechanism layer SCSE that combines the channel and spatial domains. The channel domain consists of one global pooling layer, two convolutional layers, and one activation layer connected sequentially. The global pooling layer reduces the dimensionality of the 128×28×28 feature map to 128×1×1 to obtain the channel attention map. The first convolution has 64 1×1 convolutional kernels, reducing the 128×1×1 attention map to 64×1×1. The second convolution has 128 1×1 convolutional kernels, restoring the channel attention map to 128×1×1. The final activation layer does not change its size. The spatial domain consists of a convolutional layer and an activation layer directly connected. The convolutional layer contains one 1×1 convolutional kernel, obtaining a 1×28×28 spatial attention map from the input 128×28×28 feature map. The attention maps in the channel domain and spatial domain are multiplied by the feature maps respectively and then added together to obtain an adjusted 128×28×28 feature map, which is then used as the input to the 5th layer of the upsampling module.

[0125] The fourth channel connects the output feature map of layer 9 of the downsampling module with the output feature map of layer 6 of the residual block. Before fusion, the output feature map of layer 9 of the downsampling module needs to pass through an attention mechanism layer SCSE that combines the channel domain and the spatial domain. The channel domain consists of one global pooling layer, two convolutional layers, and one activation layer connected sequentially. The global pooling layer reduces the dimensionality of the 256×14×14 feature map to 256×1×1 to obtain the channel attention map. The first convolution has 128 1×1 convolutional kernels, reducing the 256×1×1 attention map to 128×1×1. The second convolution has 256 1×1 convolutional kernels, restoring the channel attention map to 256×1×1. The final activation layer does not change its size. The spatial domain consists of a convolutional layer and an activation layer directly connected. The convolutional layer contains one 1×1 convolutional kernel, obtaining a 1×14×14 spatial attention map from the input 256×14×14 feature map. The attention maps in the channel domain and spatial domain are multiplied by the feature maps respectively and then added together to obtain an adjusted 256×14×14 feature map, which is used as the input of the first layer of the upsampling module.

[0126] Secondly, create a high-quality training dataset.

[0127] Asynchronous event streams and corresponding grayscale videos were simulated using the ESIM method presented in "ESIM: An Open Event Camera Simulator" at the 2018 Conference on Robot Learning (CoRL). The COCO dataset and several foreground image sets were downloaded. A certain number of images were randomly selected from the COCO dataset to serve as backgrounds. The backgrounds simulated random trajectories on the image plane, including slow-motion, medium-speed, and fast-motion movements. After selecting the backgrounds, several images were extracted from the COCO dataset and foreground images to serve as foregrounds. The foregrounds simulated motion similar to real-world scenarios, exhibiting as much diversity as possible and presenting a wide range of scene dynamics, including but not limited to translation, rotation, and dilation. This resulted in event stream sequences with high-quality grayscale images for system learning. Different backgrounds were selected during dataset creation to generate numerous event streams and video sequences of varying lengths and formats as the training dataset.

[0128] Next, the training dataset is converted into a mediator event representation (referred to as a mediator representation) and input into each module of the system built in the first step for thorough training.

[0129] Take the event stream and grayscale video generated by the ESIM method, and add zero-mean noise as preprocessing. Set the mean μ = 0 and the variance σ = 0.1, i.e., the noise... This method splits each grayscale video segment into several image frames, generates images frame by frame, and then concatenates them into a video. For example... Figure 7 As shown, before generating each frame of image, the event stream from the previous frame to the current frame is first extracted. The expression is given, where T represents the number of the extracted event stream sequence, and N represents the number of the extracted event stream sequence. T This represents the number of events in the event stream, where 'e' represents the event, and each event consists of a quadruple (x, y, t, p), where x and y represent the coordinates of the event, t is the timestamp of the event, p is the polarity, and i is the event number. Similarly, the event stream from the current frame to the next frame is retrieved using... This indicates that the two parts are calculated and processed separately. Similarly, ε is truncated when generating the first frame. T-1 and ε T Two short event streams, ε is extracted when generating the subsequent frame. T+1 and ε T+2 Two short event streams. To better reproduce the scene of the current frame, this method assigns higher weight to events that occur closer to the timestamp of the current frame.

[0130] This method processes two event streams into a voxel grid V. The voxel grid (i.e., the intermediate representation) is a 3D data set. Each voxel grid has B = 5 channels, and its length and width are represented by W and H, respectively, based on the length and width captured by the event camera. First, the events in the event stream are divided according to their location. Assuming there are N events at a given location, the numerical calculation for different channels at each location is as follows:

[0131]

[0132] Stacking the values ​​from B channels and then concatenating the data according to the event occurrence positions yields a voxel grid. The event stream from the current frame to the previous frame contains... This represents the weight assigned to each event, calculated as follows:

[0133]

[0134] In the event stream from the current frame to the next frame The calculation form is:

[0135]

[0136] t in the two-part formula min (ε T ) represents the smallest timestamp in the event stream of a preset duration prior to the current moment, t max (ε T ) represents the maximum timestamp in the event stream of a preset duration prior to the current moment, t max (ε T+1 ) represents the maximum timestamp in the event stream after the current moment and for a preset duration, t min (ε T+1 This represents the smallest timestamp in the event stream after the current moment and for a preset duration. The calculation is completed using V... T and V T+1 This represents the two parts of the input to the image reconstruction system. The dataset is randomly cropped to a size of 112×112 before input.

[0137] The system parameters built in the first step are randomly initialized. The state gate data of each downsampled-length short-term memory convolutional layer are all initialized to 0. The activation layer in the SCSE attention mechanism layer uses the sigmoid function, and all other activation layers use the ReLU function. The data in the training dataset is randomly divided into several segments for the network to learn. This method ensures that the length of each small segment of the event stream image frame is L = 40. At each step in the network, L timestamps of associated mediating event representations are input. The batch size is set to 4, the epoch is set to 500, and the learning rate is 0.001. The ADAM optimizer is used to minimize the loss function, and the various parameters in the system are updated for training. During training, to improve the generation effect when the camera is stationary, the system pauses at each step with a probability of 0.05, initializing all input data to 0 to simulate a situation where no events occur. In the paused state, the pause is maintained with a probability of 0.9.

[0138] The training loss function is a weighted sum of perceptual loss LPIPS (published at the IEEE International Conference on Computer Vision and Pattern Recognition in 2018 in "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric") and temporal consistency loss (TC, published at the European Conference on Computer Vision in 2018 in "Learning Blind Video Temporal Consistency"), where k represents the number of the generated grayscale frame. The total loss function at each training step is... The calculation form is:

[0139]

[0140] Among them, LPIPS loss Calculate and generate image and target results The difference in feature maps between deep network structures (i.e., the corresponding video frames in the simulated dataset) is calculated using the Alex network.

[0141] Temporal consistency loss is used to reduce the gap between adjacent frames. The L1 norm of the mask after optical flow alignment of adjacent frames is calculated and can be expressed as: Here This represents the grayscale image generated by the system's network framework, where k represents the image number. This refers to the operation of calculating optical flow using two frames of images and aligning the same optical flow positions. There is a weighting factor that helps mitigate the effects of occlusion, in which The target data is α, which is an adjustable parameter; in implementation, it can be set to 50. The parameter λ in the loss function adjusts the weights of both. TC =2.

[0142] During training, the system is considered complete when the mean of the loss function stops decreasing over a certain period of time, and the parameters of each module are saved.

[0143] After training with a large amount of data, the optimized parameters of the asynchronous event stream image reconstruction system were obtained. Subsequently, the asynchronous event stream image reconstruction system can be directly used to generate images and videos.

[0144] Step 1 includes:

[0145] 1.1 Obtain the asynchronous event stream that needs to be reconstructed as video. The event stream can be obtained using a pre-prepared event stream dataset or an asynchronous event stream output from the camera in real time.

[0146] 1.2 Convert asynchronous event streams into intermediate representations. Event streams can be divided according to time intervals or the number of events. The division parameters can be adjusted based on the expected frame rate of the grayscale video. For example, a fixed number of events N = W × H × C can be used in each event's intermediate representation. th Intercepting the event stream, C th The values ​​are determined based on the actual situation. W and H represent the pixel width and height of the event camera. Then, the captured event stream is calculated using the same method as during training. For each generated image, the timestamp is used to calculate the median representation of the left and right events. The number of channels for each voxel grid is set to B=5. First, the events in the event stream are divided according to their location. Assuming there are N events at a certain location, the numerical calculation for different channels at each location is as follows:

[0147]

[0148] The meanings of other symbols in the formula are the same as during training, where the event stream from the current frame to the previous frame... The calculation form is:

[0149]

[0150] In the event stream from the current frame to the next frame The calculation form is:

[0151]

[0152] t in the two-part formula min (ε T ) represents the smallest timestamp in the event stream of a preset duration prior to the current moment, t max (ε T) represents the maximum timestamp in the event stream of a preset duration prior to the current moment, t max (ε T+1 ) represents the maximum timestamp in the event stream after the current moment and for a preset duration, t min (ε T+1 This represents the smallest timestamp in the event stream after the current moment and for a preset duration. The calculation is completed using V... t and V t+1 This indicates that the system will downsample the mediation event representation three times, so it is necessary to ensure that V t and V t+1 Both the length and width dimensions can be 2 3 Integer division. Therefore, missing pixels need to be padded. For example, event stream data with W=240 and H=180 needs to be padded to W=240 and H=184 in order to be received by the system. Using this as an example, we continue to calculate the data of each module of the system.

[0153] Step 2 includes:

[0154] 2.1 Load the trained asynchronous event stream image reconstruction system and parameters. Initialize the system using the pre-trained system network model parameters, i.e., load the parameters into the system. At the same time, initialize the state gate data of the three downsampled long short-term memory convolutional layers to all zeros.

[0155] 2.2 The intermediate representation is input into the asynchronous event stream image reconstruction system, and then passes through each module of the asynchronous event stream image reconstruction system in sequence to obtain the reconstructed image.

[0156] 2.2.1 Two 5×240×184 intermediate representations first enter the event fusion module. After shared convolution, two feature maps x = Conv(V) are obtained. t ), y = Conv(V t+1 The values ​​are both 32×240×184. The sum of x and y is then processed through global pooling and a fully connected layer to obtain the global attention graph z = FCL. C→C / 2 (GAP(x+y)), with a size of 16×1×1, then calculate the attention graphs A for the two local areas respectively. x =FCL C / 2→C (z), A y =FCL C / 2→C (z), both are 32×1×1. Finally, the feature map of each part is multiplied by its respective attention map and then summed to obtain the event fusion feature EFF = A, which combines the two parts. x x+A y The size of the EFF is 32×240×184.

[0157] 2.2.2 Input the EFF into the downsampling module. After the first convolutional layer, the feature size becomes 64×120×92; after the second activation layer, the size remains unchanged; after the third long short-term memory convolutional layer, the state features and output features are updated, both with a size of 64×120×92, and the output is denoted as DF1; after the fourth convolutional layer, the feature size becomes 128×60×46; after the fifth activation layer, the size remains unchanged; after the sixth long short-term memory convolutional layer, the state features and output features are updated, both with a size of 128×60×46, and the output is denoted as DF2; after the seventh convolutional layer, the feature size becomes 256×30×23; after the fifth activation layer, the size remains unchanged; after the sixth long short-term memory convolutional layer, the state features and output features are updated, both with a size of 256×30×23, and the output is denoted as DF3.

[0158] 2.2.3 Input DF3 into the residual block. The input data sequentially passes through each convolutional and activation layer in the residual block. The input data for layer 4 is the feature obtained by adding DF3 to the output data of layer 3, and the input data for layer 8 is the feature obtained by directly adding the input data of layer 5 to the output data of layer 7. The final output of the residual block is the output of layer 8, denoted as RF. The feature map size for all inputs and outputs in the residual block is 256×30×23.

[0159] 2.2.4 DF1, DF2, and DF3 are fed into the SCSE attention mechanism structure of the skip connection. DF1 passes through the second channel of the skip connection, and an attention map cA of size 64×1×1 is calculated from the channel domain. DF1 The attention diagram sA, which is 1×120×92, was calculated from the spatial domain. DF1 The adjusted feature is denoted as RDF1 = cA DF1 ·DF 1+ sA DF1 • DF1, with a size of 64×120×92; DF2, after skipping to the third channel, yields a attention graph cA of size 128×1×1 calculated from the channel domain portion. DF1 A attention diagram sA of 1×60×46 was obtained from the spatial domain. DF2 The adjusted feature is denoted as RDF2 = cA DF2 ·DF 2+ sA DF2 • DF2, with a size of 128×60×46; DF3, after skipping the fourth channel, yields a attention graph cA of size 256×1×1 calculated from the channel domain portion. DF3 The attention diagram sA, which is 1×240×184, was calculated from the spatial domain. DF3 The adjusted feature is denoted as RDF3 = cA DF3 ·DF 3+ sADF3 ·DF3, size is 256×30×23.

[0160] 2.2.5 The RF features output by the residual block are fused with the adjusted RDF3, RDF2, RDF1, etc. and input into the upsampling layer.

[0161] Specifically as follows:

[0162] First, feature fusion is performed on the fourth channel via skip connections. RF and RDF3 are directly added and input into the upsampled first layer. After bilinear interpolation, the feature becomes 256×60×46. After two parallel convolutions in the upsampled second layer, two 128×60×46 features are obtained. These two features are directly added to obtain UF1, which is then input into the third layer. In the third layer, a 128×1×1 attention map cA in the channel domain is calculated. UF1 Attention diagram sA with a spatial domain of 1×60×46 UF1 Multiplying and summing the attention maps yields the adjusted feature RUF1 = cA. UF1 ·UF 1+ sA UF1 UF1, with a size of 128×60×46; the ReLU activation of the 4th layer does not change the size of the feature, and outputs ReLU(RUF1).

[0163] Next, the feature fusion of the skip connection in the 3rd channel is completed. Relu(RUF1) and RDF2 are directly added and input into the upsampled 5th layer. After bilinear interpolation, the feature becomes 128×120×92. After two parallel convolutions in the upsampled 6th layer, two 64×120×92 features are obtained. The two features are directly added to obtain UF2, which is input into the 7th layer. In the 7th layer, the 64×1×1 attention map cA in the channel domain is calculated. UF2 Attention diagram sA of spatial domain 1×120×92 UF2 Multiplying and summing the attention maps yields the adjusted feature RUF2 = cA. UF2 ·UF 2+ sA UF2 • UF2, with a size of 64×120×92; the ReLU activation of the 8th layer does not change the size of the feature, and the output is ReLU(RUF2).

[0164] Then, the feature fusion of the second channel is completed by skipping the connection. The ReLU(RUF2) and RDF1 are directly added and input into the upsampled fifth layer. After bilinear interpolation, the feature becomes 64×240×184. After two parallel convolutions in the upsampled sixth layer, two 32×240×184 features are obtained. The two features are directly added to obtain UF3, which is input into the seventh layer. In the seventh layer, the 32×1×1 attention map cA in the channel domain is calculated. UF3 Attention diagram sA with spatial domain 1×240×184UF3 Multiplying and summing the attention maps yields the adjusted feature RUF3 = cA. UF3 ·UF 3+ sA UF3 • UF3, with a size of 32×240×184; the ReLU activation of the 8th layer does not change the size of the feature, and outputs ReLU(RUF3), which is used as part of the input at the tail.

[0165] 2.2.6 Add ReLU(RUF3) and EFF together and input them into the tail convolution to obtain the generated grayscale image.

[0166] The tail layer completes the first skip connection, directly adding Relu(RUF3) and EFF. The tail convolutional layer transforms the 32×240×184 feature into a 1×240×184 single-channel grayscale image output. At this point, the image needs to be cropped to remove the padding in 4.3, and the image is reduced to 1×240×180 to complete the restoration.

[0167] 2.3 Concatenate sequentially arranged images into a video.

[0168] Repeat the steps in 2.2 for the event data near each time stamp in the sequence to obtain a sequence of grayscale images. Concatenate the image sequences to output a grayscale video, completing the final step of this method.

[0169] The following is a specific example.

[0170] This invention (SCSE-E2VID) is compared with other event stream image reconstruction methods. The experimental equipment is a desktop computer equipped with an Intel i7-9700 processor with a processor frequency of 3.00GHz and 16GB of RAM. The experimental system is Ubuntu 18.04 and uses the NVIDIA TITAN RTX graphics processor.

[0171] First, the invention was tested using two publicly available event camera datasets. The two datasets are: IJRR (Event-Camera Dataset and Simulator: Event-based data for pose estimation, visual odometry, and SLAM), published in the 2017 International Journal of Robotics Research (pages 142-149); and HQF (High-Quality Dataset), a high-quality dataset presented at the 2020 European Conference on Computer Vision, titled "Reducing the Sim-to-Real Gap for Event Cameras." Segments with good target image quality from IJRR and all scenes from HQF were selected for testing, comparing the performance of different methods on the same data. Quantitatively, mean squared error (MSE), structural similarity (SSIM), and perceptual similarity (LPIPS) were used to quantitatively measure the difference between the generated result and the target value. A smaller MSE indicates a higher SSIM, and a lower LPIPS indicates a better generated result. As can be seen, in both datasets, our method achieves a level consistent with existing methods in terms of mean squared error, and surpasses existing E2VID series methods in both structural and perceptual similarity. The gain achieved on the IJRR dataset is greater than that on HQF. Qualitatively, a direct comparison of our generated images with those of other methods and target values ​​reveals that our method removes some artifacts from the images, enriches details, and effectively completes the reconstruction tasks of grayscale images and videos.

[0172] Table 1. Quantitative evaluation of the effectiveness of this method compared to the E2VID series on the IJRR dataset.

[0173]

[0174] Table 2 Quantitative evaluation of the effectiveness of this method compared to the E2VID series on the HQF dataset.

[0175]

[0176]

[0177] Example 2:

[0178] Based on the same inventive concept, the present invention also provides a video reconstruction system based on asynchronous event streams.

[0179] The system structure is as follows Figure 8 As shown, it includes:

[0180] This indicates the conversion module, image conversion module, and video reconstruction module;

[0181] The representation conversion module is used to acquire the asynchronous event stream that needs to be reconstructed into a video, and convert the asynchronous event stream into an intermediate representation;

[0182] The image conversion module is used to input the intermediate representation into a pre-constructed asynchronous event stream image reconstruction system to obtain a grayscale image sequence;

[0183] The video reconstruction module is used to connect the grayscale images in the grayscale image sequence in chronological order to form a video.

[0184] The asynchronous event stream image reconstruction system is composed of an event fusion module that integrates key information over a period of time, a downsampling module that uses a loop structure to enhance the connection between frames, a noise-suppressing skip connection, and an upsampling module that reduces channels to restore image size; each intermediate representation is calculated from all events in an event stream.

[0185] The image conversion module constructs an asynchronous event-stream image reconstruction system, including:

[0186] An asynchronous event stream image reconstruction system is constructed, comprising an event fusion module, a downsampling module, a residual block, an upsampling module, and a tail, all connected sequentially. The downsampling module includes multiple sequentially connected downsampling structures, and the upsampling module includes multiple sequentially connected upsampling structures, with the number of upsampling structures equal to the number of downsampling structures. The downsampling and upsampling structures are symmetrically arranged around the residual block. Each downsampling structure also transmits its output data through an attention mechanism layer to the symmetrically distributed upsampling structures via skip connections. The event fusion module also transmits its output data to the tail via skip connections. The event fusion module receives and fuses intermediate representations of a preset duration before and after the current time. The downsampling module performs downsampling, the residual block performs residual calculation, the upsampling module performs upsampling, and the tail outputs a grayscale image.

[0187] Based on the COCO dataset and a selected foreground image set, the ESIM method is used to simulate the asynchronous event stream and corresponding grayscale video for training.

[0188] Convert the asynchronous event stream used for training into an intermediate representation used for training;

[0189] Using the intermediate representation used for training as input and each grayscale image in the corresponding grayscale video as output, the structure is trained to obtain an asynchronous event stream image reconstruction system.

[0190] The event fusion module in the image conversion module includes: a shared convolutional layer, a global pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer;

[0191] The shared convolutional layer is used to process the mediation representations before and after the current time for a preset duration, to obtain a first feature map corresponding to the mediation representations before the current time for a preset duration and a second feature map corresponding to the mediation representations after the current time for a preset duration.

[0192] The global pooling layer is used to reduce the dimensionality of the data after summing the corresponding pixels of the first feature map and the second feature map to obtain the first attention map, and then input the first attention map into the first fully connected layer.

[0193] The first fully connected layer is used to reduce the number of channels in the first attention map to obtain a second attention map, and the second attention map is input into the second fully connected layer and the third fully connected layer respectively;

[0194] The second fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the first feature map;

[0195] The third fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the second feature map;

[0196] The output of the event fusion module is the sum of the product of the first feature map and its corresponding attention map plus the product of the second feature map and its corresponding attention map.

[0197] The downsampling structure in the image conversion module includes a convolutional layer, an activation layer, and a long short-term memory convolutional layer connected in sequence.

[0198] The convolutional layer is used to input data, and the long short-time memory convolutional layer is used to output data.

[0199] The long short-term memory convolutional layer contains a state graph of current features that enhances the connection between adjacent frames.

[0200] The residual block in the image conversion module includes two residual structures connected in sequence.

[0201] Each of the residual structures includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence;

[0202] The input data of the first convolutional layer is also added to the output data of the second convolutional layer through skip connections and then input to the second activation layer;

[0203] The first convolutional layer of the previous residual structure is used to obtain the input data of the residual block, and the second activation layer of the next residual structure is used to output the output data of the residual block.

[0204] The upsampling structure in the image conversion module includes a first layer, a second layer, a third layer, and a fourth layer connected in sequence; the first layer is used for input data, and the fourth layer is used for output data.

[0205] The first layer is a bilinear interpolation layer;

[0206] The second layer consists of two parallel convolutional layers;

[0207] The third layer is the attention mechanism layer;

[0208] The fourth layer is the activation layer.

[0209] The tail section of the image conversion module is a convolutional layer.

[0210] The attention mechanism layer in the image conversion module includes a channel domain and a spatial domain;

[0211] The channel domain includes a global pooling layer, two convolutional layers, and an activation layer connected in sequence.

[0212] The spatial domain includes convolutional layers and activation layers connected in sequence;

[0213] The global pooling layer in the channel domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the channel domain is used to output the channel domain attention map.

[0214] The convolutional layer in the spatial domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the spatial domain is used to output the spatial domain attention map.

[0215] The channel domain attention map and the spatial domain attention map are multiplied by the feature map of the input attention mechanism layer and then summed to obtain the feature map output by the attention mechanism layer.

[0216] The image conversion module or representation conversion module converts the asynchronous event stream into a computational formula for an intermediate representation as follows:

[0217]

[0218] In the formula, V b∈[0,B-1] Let p represent the value at the b-th channel position in the asynchronous event stream within the mediated representation, where B is the total number of channels in the mediated representation, N is the number of asynchronous events in the mediated representation, and p i Let represent the polarity of the i-th asynchronous event in the mediation representation. Let be the weight of the i-th asynchronous event in the mediation representation;

[0219] The weight of the i-th asynchronous event in the mediation representation The calculation formula is as follows:

[0220]

[0221] In the formula, t i Let t be the timestamp corresponding to the i-th asynchronous event in the mediation representation. min (ε T ) represents the smallest timestamp in the event stream of a preset duration prior to the current moment, t max (ε T ) represents the maximum timestamp in the event stream of a preset duration prior to the current moment, t max (ε T+1 ) represents the maximum timestamp in the event stream after the current moment and for a preset duration, t min (ε T+1 This represents the smallest timestamp in the event stream after the current moment and for a preset duration.

[0222] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0223] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0224] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0225] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. A video reconstruction method based on asynchronous event streams, characterized in that, include: Obtain the asynchronous event stream that needs to be reconstructed into a video, and convert the asynchronous event stream into an intermediate representation; The intermediate representation is input into a pre-constructed asynchronous event stream image reconstruction system to obtain a grayscale image sequence; A video is constructed by connecting grayscale images in a grayscale image sequence in chronological order. The asynchronous event stream image reconstruction system is composed of an event fusion module that integrates key information over a period of time, a downsampling module that uses a loop structure to enhance the connection between frames, a noise-suppressing skip connection, and an upsampling module that reduces channels to restore image size; each intermediate representation is calculated from all events in an event stream. The asynchronous event stream image reconstruction system comprises an event fusion module, a downsampling module, a residual block, an upsampling module, and a tail, all connected in sequence. The downsampling module includes multiple downsampling structures connected in sequence, and the upsampling module includes multiple upsampling structures connected in sequence, with the number of upsampling structures equal to the number of downsampling structures. The downsampling and upsampling structures are symmetrically arranged around the residual block. Each downsampling structure also transmits its output data through an attention mechanism layer to the symmetrically distributed upsampling structures via skip connections. The event fusion module also transmits its output data to the tail via skip connections. The event fusion module receives intermediate representations of a preset duration before and after the current time and fuses these two intermediate representations. The downsampling module performs downsampling, the residual block performs residual calculation, the upsampling module performs upsampling, and the tail outputs a grayscale image. The event fusion module includes: a shared convolutional layer, a global pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer; The shared convolutional layer is used to process the mediation representations before and after the current time for a preset duration, to obtain the first feature map corresponding to the mediation representations before the current time for a preset duration and the second feature map corresponding to the mediation representations after the current time for a preset duration. The global pooling layer is used to reduce the dimensionality of the data after summing the corresponding pixels of the first feature map and the second feature map to obtain the first attention map, and then input the first attention map into the first fully connected layer. The first fully connected layer is used to reduce the number of channels in the first attention map to obtain a second attention map, and the second attention map is input into the second fully connected layer and the third fully connected layer respectively; The second fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the first feature map; The third fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the second feature map; The output of the event fusion module is the sum of the product of the first feature map and the corresponding attention map plus the product of the second feature map and the corresponding attention map. The attention mechanism layer includes a channel domain and a spatial domain; The channel domain includes a global pooling layer, two convolutional layers, and an activation layer connected in sequence. The spatial domain includes convolutional layers and activation layers connected in sequence; The global pooling layer in the channel domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the channel domain is used to output the channel domain attention map. The convolutional layer in the spatial domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the spatial domain is used to output the spatial domain attention map. The channel domain attention map and the spatial domain attention map are multiplied by the feature map of the input attention mechanism layer and then summed to obtain the feature map output by the attention mechanism layer.

2. The method as described in claim 1, characterized in that, The construction of the asynchronous event stream image reconstruction system includes: Based on the COCO dataset and a selected foreground image set, the ESIM method is used to simulate the asynchronous event stream and corresponding grayscale video for training. Convert the asynchronous event stream used for training into an intermediate representation used for training; Using the intermediate representation used for training as input and each grayscale image in the corresponding grayscale video as output, the structure is trained to obtain an asynchronous event stream image reconstruction system.

3. The method as described in claim 1, characterized in that, The downsampling structure includes a convolutional layer, an activation layer, and a long short-term memory convolutional layer connected in sequence. The convolutional layer is used to input data, and the long short-time memory convolutional layer is used to output data. The long short-term memory convolutional layer contains a state graph of current features that enhances the connection between adjacent frames.

4. The method as described in claim 1, characterized in that, The residual block comprises two residual structures connected in sequence; Each of the residual structures includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence; The input data of the first convolutional layer is also added to the output data of the second convolutional layer through skip connections and then input to the second activation layer; The first convolutional layer of the previous residual structure is used to obtain the input data of the residual block, and the second activation layer of the next residual structure is used to output the output data of the residual block.

5. The method as described in claim 1, characterized in that, The upsampling structure includes a first layer, a second layer, a third layer, and a fourth layer connected in sequence; the first layer is used for input data, and the fourth layer is used for output data. The first layer is a bilinear interpolation layer; The second layer consists of two parallel convolutional layers; The third layer is the attention mechanism layer; The fourth layer is the activation layer.

6. The method as described in claim 1, characterized in that, The tail section is a convolutional layer.

7. The method as described in claim 1 or 2, characterized in that, The computational formula for converting asynchronous event streams into a mediated representation is as follows: In the formula, This represents the value at the b-th channel position in the asynchronous event stream within the mediation representation, where B is the total number of channels in the mediation representation, and N is the number of asynchronous events in the mediation representation. p i The intermediary indicated the first i The polarity of an asynchronous event. The intermediary indicated the first i The weight of each asynchronous event; The intermediary represents the first i Weight of asynchronous events The calculation formula is as follows: In the formula, The intermediary indicated the first i The timestamps corresponding to the asynchronous events This represents the smallest timestamp in the event stream of a preset duration prior to the current moment. This represents the largest timestamp in the event stream of a preset duration prior to the current moment. This represents the largest timestamp in the event stream after the current moment and for a preset duration. It represents the smallest timestamp in the event stream after the current moment and for a preset duration.

8. A video reconstruction system based on asynchronous event streams, characterized in that, include: This indicates the conversion module, image conversion module, and video reconstruction module; The representation conversion module is used to acquire the asynchronous event stream that needs to be reconstructed into a video, and convert the asynchronous event stream into an intermediate representation; The image conversion module is used to input the intermediate representation into a pre-constructed asynchronous event stream image reconstruction system to obtain a grayscale image sequence; The video reconstruction module is used to connect the grayscale images in the grayscale image sequence in chronological order to form a video. The asynchronous event stream image reconstruction system is composed of an event fusion module that integrates key information over a period of time, a downsampling module that uses a loop structure to enhance the connection between frames, a noise-suppressing skip connection, and an upsampling module that reduces channels to restore image size; each intermediate representation is calculated from all events in an event stream. The asynchronous event stream image reconstruction system comprises an event fusion module, a downsampling module, a residual block, an upsampling module, and a tail, all connected in sequence. The downsampling module includes multiple downsampling structures connected in sequence, and the upsampling module includes multiple upsampling structures connected in sequence, with the number of upsampling structures equal to the number of downsampling structures. The downsampling and upsampling structures are symmetrically arranged around the residual block. Each downsampling structure also transmits its output data through an attention mechanism layer to the symmetrically distributed upsampling structures via skip connections. The event fusion module also transmits its output data to the tail via skip connections. The event fusion module receives intermediate representations of a preset duration before and after the current time and fuses these two intermediate representations. The downsampling module performs downsampling, the residual block performs residual calculation, the upsampling module performs upsampling, and the tail outputs a grayscale image. The event fusion module includes: a shared convolutional layer, a global pooling layer, a first fully connected layer, a second fully connected layer, and a third fully connected layer; The shared convolutional layer is used to process the mediation representations before and after the current time for a preset duration, to obtain the first feature map corresponding to the mediation representations before the current time for a preset duration and the second feature map corresponding to the mediation representations after the current time for a preset duration. The global pooling layer is used to reduce the dimensionality of the data after summing the corresponding pixels of the first feature map and the second feature map to obtain the first attention map, and then input the first attention map into the first fully connected layer. The first fully connected layer is used to reduce the number of channels in the first attention map to obtain a second attention map, and the second attention map is input into the second fully connected layer and the third fully connected layer respectively; The second fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the first feature map; The third fully connected layer is used to recover the number of channels in the second attention map and generate the attention map corresponding to the second feature map; The output of the event fusion module is the sum of the product of the first feature map and the corresponding attention map plus the product of the second feature map and the corresponding attention map. The attention mechanism layer includes a channel domain and a spatial domain; The channel domain includes a global pooling layer, two convolutional layers, and an activation layer connected in sequence. The spatial domain includes convolutional layers and activation layers connected in sequence; The global pooling layer in the channel domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the channel domain is used to output the channel domain attention map. The convolutional layer in the spatial domain is used to obtain the feature map of the input attention mechanism layer, and the activation layer in the spatial domain is used to output the spatial domain attention map. The channel domain attention map and the spatial domain attention map are multiplied by the feature map of the input attention mechanism layer and then summed to obtain the feature map output by the attention mechanism layer.

9. The system as described in claim 8, characterized in that, The image conversion module constructs an asynchronous event-stream image reconstruction system, including: Based on the COCO dataset and a selected foreground image set, the ESIM method is used to simulate the asynchronous event stream and corresponding grayscale video for training. Convert the asynchronous event stream used for training into an intermediate representation used for training; Using the intermediate representation used for training as input and each grayscale image in the corresponding grayscale video as output, the structure is trained to obtain an asynchronous event stream image reconstruction system.

Images