Event stream image reconstruction method based on hierarchical uncertainty

Abstract

The invention discloses an event stream image reconstruction method based on hierarchical uncertainty, and the method comprises the steps: S1, inputting event stream data, and carrying out the random sampling, and constructing a multi-mask event frame; s2, inputting the processed event frame into an event flow reconstruction network to obtain a plurality of reconstructed images, the reconstructed images sharing a real image, and establishing an uncertainty model between the reconstructed images and the real image; and S3, adjusting network parameters to train and test a neural network, and screening images needing to be optimized from the uncertainty scores. The accuracy and quality of event stream image reconstruction are improved, and the adaptability and robustness of the model to a dynamic scene are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and particularly to an event stream image reconstruction method based on hierarchical uncertainty. Background Art

[0002] Event cameras are a new vision paradigm inspired by biological vision. Different from traditional cameras that record images at a fixed frame rate, each pixel of an event camera works independently and outputs an event stream asynchronously. Event data has characteristics such as high temporal resolution, high dynamic range, and low power consumption, so it has attracted much attention in the field of computer vision. In recent years, a large number of algorithms for event stream image reconstruction have been proposed, which can be generally divided into the following two categories: event stream image reconstruction based on traditional vision algorithms and event stream image reconstruction based on deep learning.

[0003] Traditional vision algorithms usually adopt a method of accumulating a fixed number of events. The events of a fixed number are counted according to their polarities to obtain a grayscale intensity image corresponding to the scene. The Interactive Visual Maps (IVM) algorithm uses sparse event representations to perform joint approximate inference on multiple visual parameters, including optical flow, grayscale intensity, and camera motion. IVM takes into account parallelism and distributed computing, which benefits it on specific hardware, but its performance is poor on existing hardware architectures. The manifold regularization method models the reconstruction problem as an energy minimization defined on the manifold induced by event timestamps. This method enables the integration of the time series information of events into the reconstruction process without explicitly estimating the optical flow. However, although this method avoids optical flow estimation, its disadvantages are that it still needs to preprocess the timestamp field, the reconstruction quality depends on the camera performance, and it lacks the ability to directly estimate other visual tasks from event data. The fast time compensation method is a method that continuously fuses the image data of an event camera and a traditional camera using a complementary filter to obtain a reinforced image sequence with high temporal resolution and high dynamic range.

[0004] When using deep learning to process event stream image reconstruction, the primary problem is to address the training difficulties caused by the scarcity of paired data between event cameras and natural images. The event camera simulator uses the images of traditional RGB cameras as input and generates approximate event stream data by detecting the points of intensity change in the images. The event stream obtained from this image conversion can effectively simulate the output of a real camera, effectively improving the problem of difficult data acquisition for event cameras. However, there are still differences between the simulated data and the real data. The event enhancement method uses the method of dataset threshold evaluation to compare the similarity between common event camera datasets and simulated datasets, and through data augmentation, the original model shows better metrics in the learning of the new dataset. In addition, researchers have tried to use intermediate information such as optical flow, degrees of freedom, and rotation angle to achieve the conversion between the two data forms. The EKLT (Event-based Kanade-Lucas-Tomasi) method uses the optical flow constraint equation based on the assumption of brightness constancy to associate events with the pixel intensities on natural images, and uses optical flow to estimate the changes of events in time and space.

[0005] With the open source of relevant simulation technologies and related datasets, the work of event stream image reconstruction and video reconstruction has begun to receive extensive attention. The most representative achievement among them is the E2VID (Events-to-Video) framework proposed by ReSeq and its team. This work draws on the advantages of recurrent neural networks, inputs the images before generating the sequence into the U-Net architecture, and obtains the image sequence. Once this model was released, it became the SOTA model in the field of event stream image generation. However, there is still room for improvement in the running speed of E2VID. In order to make the model have both performance and running speed, the team proposed a temporal continuity loss in the FireNet method to help the model achieve real-time deployment in video reconstruction and high-dynamic motion capture. SPADE-E2VID proposed an end-to-end data-driven framework for event stream video reconstruction. This method uses spatial adaptive regularization (SPADE) and recurrent networks to directly reconstruct videos from events, and adopts minimizing the dynamic loss to obtain a more realistic reconstruction effect. SPADE-E2VID has achieved great improvements in both methods and effects, and its open source resources also have high reference value.

[0006] When dealing with event stream image reconstruction in the prior art, it is often difficult to achieve high-precision reconstruction due to the noise in the event data itself. Traditional data-driven methods often fail to effectively cope with the influence of this noise, resulting in low-quality reconstructed images. In addition, when the prior art deals with dynamic scenes, the distinction between dynamic foreground and static background is not handled carefully enough, which may lead to the loss or error of details in the reconstructed image. Moreover, the prior art solutions may lack flexibility, have poor adaptability when dealing with different scenes, and fail to fully utilize the multi-scale information in the scene, which limits their performance in diverse application scenarios. Summary of the Invention

[0007] In view of the above problems, the present invention is proposed to provide an event stream image reconstruction method based on hierarchical uncertainty that overcomes the above problems or at least partially solves the above problems.

[0008] According to one aspect of the present invention, there is provided an event stream image reconstruction method based on hierarchical uncertainty, the reconstruction method comprising:

[0009] Step S1: Input event stream data and randomly sample to construct multi-mask event frames;

[0010] Step S2: Input the processed event frames into an event stream reconstruction network to obtain multiple reconstructed images. The reconstructed images share a ground truth image, and an uncertainty model is established between the reconstructed images and the ground truth image;

[0011] Step S3: Adjust network parameters for the training and testing of the neural network, and screen the images that need to be optimized from the uncertainty scores.

[0012] Optionally, the specific steps of Step S1: Input event stream data and randomly sample to construct multi-mask event frames include:

[0013] Step S11: Given a segment of event stream, where a single event is represented by the pixel coordinates, timestamp, and brightness polarity of the event occurrence;

[0014] Step S12: Use a voxel grid to convert the original event stream data into a tensor-like event frame representation, where the height H, width W, and voxel channels C of the event frame;

[0015] Step S13: Define the trained event stream reconstruction network as f θ , representing the reconstructed image, where θ represents the network parameters;

[0016] Step S14: Use a random binary mask M, where the elements with a value of 0 correspond to the occluded pixel positions, and the value of M is determined by a random probability distribution. Apply M to the original event frame to obtain a locally occluded event frame E'.

[0017] Optionally, establishing an uncertainty model between the reconstructed image and the real image specifically includes:

[0018] Step S21: Define the uncertainty measure of the importance of events at pixels as σ 2 , which is the variance of the importance estimation at the spatial position in multiple occlusion experiments;

[0019] Step S22: Combine the event importance distribution and the uncertainty distribution to obtain a hierarchical representation that comprehensively characterizes the statistical characteristics of the input event stream;

[0020] Step S23: Use the value of the loss function to estimate the confidence or weight of each event stream;

[0021] Step S24: Convert the loss function value into a confidence score, and optimize the selection and combination of event streams based on the uncertainty modeling method of the loss function;

[0022] Step S25: By dynamically adjusting the weight of each event stream, adaptively adapt to different scenarios and data characteristics to obtain more robust and efficient image reconstruction results.

[0023] Optionally, adjusting the network parameters for the training and testing of the neural network specifically includes:

[0024] Step S31: Implement the neural network using the Pytorch architecture and perform training and testing on a single NVIDIA P5000 GPU;

[0025] Step S32: Set the optimizer as the stochastic gradient descent optimizer, with an initial learning rate of 0.001, momentum of 0.9, weight decay of 1e-4, epoch of 90, and the number of sampling times and sampling intervals are set according to specific requirements;

[0026] Step S33: For the event simulation COCO dataset, use the training data generated by event simulation for network training, and the input is a set of event points and the grayscale images corresponding to the event points;

[0027] Step S34: The test data uses the dataset generated by the DAVIS240C camera, including data of dynamic scenes, different speed motion scenes, and indoor and outdoor scenes;

[0028] Step S35: The evaluation metrics use the mean squared error MSE, the structural similarity index SSIM, and the perceptual similarity loss LPIPS of the image to evaluate the quality of the reconstructed image.

[0029] Optionally, the input event stream data specifically includes:

[0030] The dataset is generated by DAVIS, the camera generates event data, and static image data of 24 frames per second is generated;

[0031] Use event simulation to train the network on the COCO dataset. The input is a set of event points and the grayscale images corresponding to the event points.

[0032] Optionally, the random sampling to construct multi-mask event frames specifically includes:

[0033] Randomly sample the number of input event points to construct multiple groups of event frames containing different numbers of event points;

[0034] Given an event stream E = {e i =(x i ,y i ,t i ,p i )|i = 1,..., N}, where e i represents a single event (x i ,y i ) as the pixel coordinates where the event occurs, t i is the timestamp, and p i ∈{1, -1} is the luminance polarity, representing an increase or decrease in pixel brightness respectively;

[0035] Use a voxel grid to transform the original event stream data E into a tensor-like event frame representation E∈R H×W×T , where H and W represent the height and width of the event frame respectively, and T represents the voxel channels of the event frame. Set T = 5;

[0036] Define the trained event stream reconstruction network: f θ : E→I, where I represents the reconstructed image;

[0037] Use a random binary mask M∈[0,1] H×W , where the elements with value 0 in M correspond to the occluded pixel positions, and the value of M ij is determined by a random probability distribution;

[0038] Apply M to the original event frame E to obtain a partial occlusion:

[0039]

[0040] where represents the element-wise multiplication of the event frame and the mask, and Ω = {(x,y)|M(x,y) = 1} is the set of pixel coordinates with value 1 in the mask M;

[0041] can be regarded as an incomplete observation of the original event frame sequence, reflecting the local interference of occlusion on scene perception;

[0042] Apply Input into the image reconstruction network to obtain the corresponding reconstruction result

[0043] Let represent the similarity measure between the unmasked image and the masked image;

[0044] If the local difference between the original reconstructed image I and the masked reconstructed image is large after occluding the (i, j)-th pixel, it indicates that the event information at this pixel is crucial for scene understanding, and occluding it will significantly affect the prediction of the reconstruction network;

[0045] Define the importance R of the event at pixel (i, j) as ij :

[0046]

[0047]

[0048] M where E represents the expectation of pixel-level importance, combining local pixel-level masking operations with global pixel-level similarity evaluation;

[0049] For a continuous random probability distribution M, sample multiple masks and approximate the expectation of the random probability distribution by calculating the variance after sampling;

[0049] Generate K binary masks {M k ∈ {0, 1} H×W | k = 1,..., K} that are the same size as the event frame in space. At this time, the definition of the importance of the event at pixel (i, j) is as follows:

[0050]

[0051] By randomly occluding the event information of different pixels and evaluating the impact of local missing on the overall reconstruction effect, the pixel-level uncertainty distribution R = {R ij ∈ R H×W} of the event frame E is obtained;

[0052] The distribution reveals the spatial saliency structure of the input event stream from a data-driven perspective, and the reconstruction algorithm focuses on key regions with rich semantics.

[0053] Optionally, the uncertainty modeling specifically includes:

[0054] Define the uncertainty measure of the importance of the event at pixel (i, j) as U ij , that is, the variance of the importance estimate R ij at this spatial position in K occlusion experiments:

[0055]

[0056] where U = {U ij ∈R H×W}, by combining the event importance distribution R and the uncertainty distribution U, a hierarchical representation that comprehensively characterizes the statistical properties of the input event stream E is obtained;

[0057] Among them, R reflects the significant differences of events in image reconstruction and provides a reference for the attention allocation of the reconstruction algorithm; while U further reveals the reliability degree of this significant structure itself and introduces an adaptive adjustment mechanism into the reconstruction process;

[0058] Based on the rich priors contained in R and U, an adaptive reconstruction loss function that fuses event importance and confidence perception will be designed to maximize the exploration of the semantic potential of event stream data.

[0059] Optionally, the uncertainty modeling further includes:

[0060] The adaptive reconstruction loss specifically includes:

[0061] Considering the event camera dataset where E n represents the event frame sequence of the nth training sample, and is the corresponding ground truth image;

[0062] For each training sample, calculate its event importance map and uncertainty map Define the adaptive reconstruction loss of the nth sample as:

[0063]

[0064] where, and represent the values of the reconstructed image and the ground truth image of the nth sample at (i, j) respectively, s(·, ·) represents the per-pixel error metric, s(·, ·) is replaced by the MSE loss, and at pixel (i, j) there is

[0065] Compared with the standard average loss, formula (5) introduces an adaptive weight term

[0066]

[0067] where, β is the regularization coefficient, and the weight comprehensively considers the importance and credibility

[0068] An event stream image reconstruction method based on hierarchical uncertainty provided by the present invention, the reconstruction method comprising: Step S1: input event stream data and randomly sample to construct multi-mask event frames; Step S2: input the processed event frames into an event stream reconstruction network to obtain multiple reconstructed images, the reconstructed images share a real image, and an uncertainty model is established between the reconstructed images and the real image; Step S3: adjust network parameters to train and test the neural network, and screen the images that need to be optimized from the uncertainty scores. Improve the accuracy and quality of event stream image reconstruction, and enhance the adaptability and robustness of the model to dynamic scenes.

[0069] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification, and in order to make the above and other objects, features and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention are specifically exemplified below. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0071] Figure 1 It is a flowchart of event stream image reconstruction based on hierarchical uncertainty modeling provided for an embodiment of the present invention;

[0072] Figure 2 It is an overall structure diagram of event stream image reconstruction based on uncertainty modeling and self-paced learning provided for an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0073] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0074] The terms "comprising" and "having" and any variations thereof in the description and claims of the present invention and the drawings thereof are intended to cover non-exclusive inclusion, for example, including a series of steps or units.

[0075] The technical solutions of the present invention will be further described in detail below with reference to the drawings and embodiments.

[0076] An event stream image reconstruction method based on hierarchical uncertainty. The present invention relates to an event stream image reconstruction technology based on hierarchical uncertainty modeling, aiming to improve the accuracy and quality of event stream image reconstruction, and enhance the adaptability and robustness of the model to dynamic scenes. This technology is particularly applicable to the event stream image reconstruction task in the field of computer vision. By constructing an adaptive image reconstruction framework, it can identify and process the hierarchical structure in event stream data, and distinguish the different importance of foreground and background event points.

[0077] The present invention generates event frames with different granularities by randomly sampling event stream data, constructs a multi-input framework and a dynamic loss function to realize the modeling of uncertainty. In addition, the present invention also adopts a self-paced learning strategy to gradually adapt network training to enhance the model's processing ability for complex dynamic scenes, thereby generating reconstructed images with richer details and higher quality. The technical solution of the present invention includes randomly sampling to construct multi-mask event frames, uncertainty modeling and self-paced selection, and adjusting network parameters for the training and testing of neural networks. By this method, the present invention not only improves the accuracy and quality of event stream image reconstruction, but also enhances the adaptability and robustness of the model to dynamic scenes, making the reconstruction results have better semantic interpretability, which has important practical significance for promoting the application of event cameras in the field of computer vision.

[0078] An event stream image reconstruction method based on hierarchical uncertainty modeling, comprising the steps of:

[0079] Step S1: Input event stream data, and randomly sample a certain number of event points to construct multiple groups of event frames containing different numbers of event points;

[0080] Step S2: Input these processed event frames into the event stream reconstruction network to obtain multiple reconstructed images. These reconstructed images share a real image, and an uncertainty model is established between the reconstructed images and the real image;

[0081] Step S3: Screen the images that need to be optimized from the uncertainty scores to avoid poor-reliability reconstructed images from entering the optimization process and causing instability of model parameters.

[0082] The specific steps of randomly sampling to construct multi-mask event frames in Step S1 include:

[0083] Given an event stream, where a single event is represented by the pixel coordinates, timestamp, and brightness polarity of the event occurrence;

[0084] Use a voxel grid to convert the original event stream data into a tensor-like event frame representation, where the height, width, and voxel channels of the event frame are represented as H, W, and C respectively;

[0085] Define the trained event stream reconstruction network as f θ , which represents the reconstructed image, where θ represents the network parameters;

[0086] Use a random binary mask M, where the elements with a value of 0 correspond to the occluded pixel positions. The value of M is determined by a random probability distribution. Apply M to the original event frame to obtain a locally occluded event frame E'.

[0087] The specific steps of uncertainty modeling and self-paced selection in step S2 include:

[0088] Define the uncertainty measure of the importance of events at a pixel as σ 2 , that is, the variance of the importance estimation at this spatial position in multiple occlusion experiments;

[0089] Combine the event importance distribution and the uncertainty distribution to obtain a hierarchical representation that comprehensively characterizes the statistical characteristics of the input event stream. The event importance distribution reflects the significant differences of events in image reconstruction, and the uncertainty distribution reveals the reliability of this significant structure itself;

[0090] Use the value of the loss function to estimate the confidence or weight of each event stream. The higher the confidence score, the greater the contribution of the event stream to the reconstruction quality and the more critical the information it contains;

[0091] Convert the loss function value to a confidence score, optimize the selection and combination of event streams based on the uncertainty modeling method of the loss function, improve the reconstruction quality and reduce the computational overhead;

[0092] By dynamically adjusting the weight of each event stream, adaptively adapt to different scenarios and data characteristics to obtain a more robust and efficient image reconstruction result.

[0093] The specific steps of adjusting network parameters for the training and testing of the neural network in step S3 include:

[0094] Implement the neural network using the Pytorch architecture and perform training and testing on a single NVIDIA P5000 GPU;

[0095] Set the optimizer to the Stochastic Gradient Descent (SGD) optimizer, with an initial learning rate of 0.001, momentum of 0.9, weight decay of 1e-4, epoch of 90, and the number of sampling times and sampling intervals are set according to specific requirements;

[0096] For the event simulation COCO dataset, use the training data generated by event simulation for network training, and the input is a set of event points and the grayscale images corresponding to the event points;

[0097] The test data uses the dataset generated by the DAVIS240C camera, including data of dynamic scenes, different speed motion scenes, and indoor and outdoor scenes;

[0098] The evaluation metrics use the mean squared error (MSE), structural similarity index (SSIM), and perceptual similarity loss (LPIPS) to evaluate the quality of the reconstructed images.

[0099] Step 1: Construct a hierarchical mask module

[0100] 1. Input of training data

[0101] The method proposed in the present invention takes event stream data as input. Specifically, the dataset is generated using DAVIS (Dynamic and Active-Pixel Vision Sensor), a camera that generates event data and static image data at 24 frames per second. The input for training the network using event simulation of the COCO dataset is a set of event points and the grayscale images corresponding to the event points.

[0102] 2. Random sampling to construct multi-mask event frames

[0103] During the process of event stream image reconstruction, the selection of the number of event points has a direct impact on the quality of the reconstruction result. To solve the problems caused by the fixed number of event points in existing methods, the present invention proposes a new training method, that is, randomly sampling the number of input event points to construct multiple groups of event frames containing different numbers of event points.

[0104] Given an event stream E = {e i =(x i ,y i ,t i ,p i )|i = 1,...,N}, where e i represents a single event (x i ,y i ) as the pixel coordinates where the event occurs, t i is the timestamp, and p i ∈{1, -1} is the brightness polarity, representing an increase or decrease in pixel brightness respectively. Furthermore, the original event stream data E is transformed into a tensor-like event frame representation E∈R H ×W×T , where H and W represent the height and width of the event frame respectively, and T represents the voxel channel of the event frame. Here, T = 5 is set.

[0105] Define the trained event stream reconstruction network: f θ : E→I, where I represents the reconstructed image.

[0106] Use a random binary mask \(M\in[0,1]\) H×W , where the elements with value 0 in \(M\) correspond to the occluded pixel positions, and the value of \(M\) ij is determined by a random probability distribution.

[0107] Apply \(M\) to the original event frame \(E\) to obtain a partial occlusion:

[0108]

[0109] where denotes the element-wise multiplication of the event frame and the mask, and \(\Omega=\{(x,y)|M(x,y) = 1\}\) is the set of pixel coordinates with value 1 in the mask \(M\). can be regarded as an incomplete observation of the original event frame sequence, reflecting the local interference of occlusion on scene perception. Input into the image reconstruction network, and the corresponding reconstruction result can be obtained

[0110] Let denote the similarity measure between the unmasked image and the masked image. If the local difference between the original reconstructed image \(I\) and the masked reconstructed image is large after occluding the \((i,j)\)-th pixel, it indicates that the event information at this pixel is crucial for scene understanding, and occluding it will significantly affect the prediction of the reconstruction network. Accordingly, define the importance \(R\) of the event at pixel \((i,j)\) ij as:

[0111]

[0112] where \(E\) M represents the expectation of pixel-level importance. Equation (4-2) combines the local pixel-level masking operation with the global pixel-level similarity evaluation. By observing the influence of local masking on global similarity, infer the degree of uncertainty of different pixel regions on the overall representation of the image, that is, the larger the value of \(R\) ij , the more difficult it is to reconstruct this pixel point, and thus the more important it is.

[0113] For a continuous random probability distribution \(M\), the expectation of this distribution is obtained through an integral form.

[0114] Sample multiple masks, and approximate the expectation of the random probability distribution by calculating the variance after sampling.

[0115] Generate \(K\) binary masks \(\{M\) k \in\{0,1\} H×W |k = 1,\ldots,K\}\) with the same spatial size as the event frame. At this time, the definition of the importance of the event at pixel \((i,j)\) is as follows:

[0116]

[0117] By randomly occluding the event information of different pixels and evaluating the impact of local missingness on the overall reconstruction effect, the pixel-level uncertainty distribution R = {R ij ∈ R H×W} of the event frame E is obtained.

[0118] This distribution reveals the spatial saliency structure of the input event stream from a data-driven perspective, helping the reconstruction algorithm focus on key regions with rich semantics. However, due to the introduction of random occlusion, the event importance estimated by formula (3) inevitably contains statistical noise. To measure the confidence difference of the importance of different pixels, its hierarchical uncertainty will be further modeled.

[0119] Step 2: Uncertainty Modeling

[0120] 1. Uncertainty Modeling

[0121] Although the random occlusion strategy can efficiently evaluate the relative importance of events among different pixels, the importance distribution R obtained by formula (3) may still be affected by biased sampling. For example, in the background region around a moving target, due to the coincidence of occlusion patterns, the importance of individual pixels may be overestimated. To judge the reliability of the importance of each pixel, a hierarchical uncertainty measure is introduced.

[0122] Define the uncertainty measure of the event importance at pixel (i, j) as U ij , that is, the variance of the importance estimate R ij at this spatial position in K occlusion trials:

[0123]

[0124] where U = {U ij ∈ R H×W}. Combining the joint event importance distribution R and the uncertainty distribution U, a hierarchical representation that comprehensively characterizes the statistical properties of the input event stream E is obtained. Among them, R reflects the saliency difference of events in image reconstruction and provides a reference for the attention allocation of the reconstruction algorithm; while U further reveals the reliability of this saliency structure itself and introduces an adaptive adjustment mechanism into the reconstruction process. Based on the rich priors contained in R and U, an adaptive reconstruction loss function that fuses event importance and confidence awareness will be designed in the next step to maximize the exploration of the semantic potential of event stream data.

[0125] 2. Adaptive Reconstruction Loss

[0126] Traditional event image reconstruction methods usually adopt a per-pixel averaging strategy to measure the reconstruction error (such as l 1 / l 2 loss), ignoring the inherent hierarchical semantic structure of event data.

[0127] To better utilize the prior information revealed by the aforementioned event importance distribution R and uncertainty distribution U, an adaptive weighted reconstruction loss function is proposed.

[0128] Consider the event camera dataset where E n represents the sequence of event frames of the nth training sample, and is its corresponding ground truth image. For each training sample, first calculate its event importance map and uncertainty map

[0129]

[0130] Then, define the adaptive reconstruction loss of the nth sample as: and respectively represent the values of the reconstructed image and the ground truth image of the nth sample at (i, j), and s(·, ·) represents the per-pixel error metric. Here, s(·, ·) is replaced by the MSE loss, that is, at pixel (i, j) there is Compared with the standard average loss, formula (5) introduces an adaptive weight term

[0131]

[0132] where β is the regularization coefficient. The weight comprehensively considers the importance and credibility

[0133] of the event at pixel (i, j) in the nth sample. On the one hand, the higher the semantic importance of the event corresponding to pixel (i, j) (i.e., θ the larger), the greater the proportion of the reconstruction error at its position in the total loss, thus driving the reconstruction network f to prioritize the restoration quality of this region. On the other hand, for pixels with more uncertain event importance estimates (i.e., the larger), the weight of their corresponding loss terms should be appropriately reduced.

[0134] The adaptive weight makes the network f θThe training objective changes from minimizing the global average error to an adaptive local optimization that takes into account both event importance and confidence. Formula (5) applies attention proportional to the semantic contribution of different pixels, while reflecting the reliability of this contribution with a differential adjustment amplitude.

[0135] The loss function will guide f θ Focus on the most informative and statistically certain significant patterns in the event stream data, thus achieving more targeted and robust reconstruction.

[0136] Step 3: Adjust the network parameters and perform the training and testing of the neural network

[0137] 1. Parameter setting

[0138] The present invention adopts the Pytorch architecture and is implemented on a single NVIDIA P5000 GPU. The optimizer used is the Stochastic Gradient Descent (SGD) optimizer. For the event simulation COCO dataset, set the initial learning rate to 0.001, the momentum to 0.9, the weight decay to 1e-4, the number of epochs to 90, the number of sampling times K = 5, and the sampling interval to [0.2HW, 0.5HW].

[0139] 2. Input test data

[0140] The test data uses the dataset generated by the DAVIS240C camera. This dataset provides an event stream with microsecond timestamps, IMU measurement data (3-axis gyroscope and accelerometer) collected at a frequency of 1kHz, and 6-DOF ground truth poses obtained from a motion capture system at a frequency of 200Hz after hand-eye calibration. It contains six sub-datasets, namely dynamic_6dof, boxes_6dofposter_6dof, office_zigzag, slider_depth, and calibration.

[0141] Beneficial effects: The present invention not only improves the accuracy and quality of event stream image reconstruction, but also enhances the adaptability and robustness of the model to dynamic scenes, making the reconstruction results have better semantic interpretability, which has important practical significance for promoting the application of event cameras in the field of computer vision.

[0142] The event stream image reconstruction technology based on hierarchical uncertainty modeling improves the accuracy and quality of event stream image reconstruction and enhances the adaptability and robustness of the model to dynamic scenes. This technology is particularly suitable for the event stream image reconstruction task in the field of computer vision. By constructing an adaptive image reconstruction framework, it can identify and process the hierarchical structure in event stream data and distinguish the different importance of foreground and background event points.

[0143] The above specific embodiments have further elaborated in detail the object, technical solution and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for event stream image reconstruction based on hierarchical uncertainty, characterized in that: The reconstruction method comprises: Step S1: input event stream data and randomly sample to construct a multi-mask event frame; Step S2: input the processed event frame into the event stream reconstruction network to obtain multiple reconstructed images, the reconstructed images share a real image, and establish an uncertainty model between the reconstructed image and the real image; Step S3: Adjust the network parameters to train and test the neural network, and select the images that need to be optimized from the uncertainty scores.

2. The event stream image reconstruction method based on hierarchical uncertainty according to claim 1, characterized in that: The step S1: inputting event stream data and constructing a multi-mask event frame by random sampling specifically includes: Step S11: given an event stream, a single event is represented by the pixel coordinates, timestamp and brightness polarity of the event; Step S12: using a voxel grid to convert the original event stream data into a tensor-like event frame representation, where the event frame has a height H, a width W, and a voxel channel C; Step S13: Define the trained event stream reconstruction network as f θ , represents the reconstructed image, where θ represents the network parameters; Step S14: using a random binary mask M, where the elements with a value of 0 correspond to the occluded pixel positions, and the value of M is determined by a random probability distribution, M is applied to the original event frame to obtain a partially occluded event frame E'.

3. The event stream image reconstruction method based on hierarchical uncertainty according to claim 1, characterized in that: The step of establishing an uncertainty model between the reconstructed image and the real image specifically includes: Step S21: define the uncertainty measure of the importance of the event at the pixel as σ2, the variance of the importance estimate at the spatial position in multiple occlusion tests; Step S22: combining the event importance distribution and the uncertainty distribution to obtain a hierarchical representation that comprehensively describes the statistical characteristics of the input event stream; Step S23: using the value of the loss function to estimate the confidence or weight of each event flow; Step S24: converting the loss function value into a confidence score, and optimizing the selection and combination of event streams based on the uncertainty modeling method of the loss function; Step S25: By dynamically adjusting the weight of each event stream, it is adaptively adapted to different scenarios and data characteristics to obtain a more robust and efficient image reconstruction result.

4. The event stream image reconstruction method based on hierarchical uncertainty according to claim 1, characterized in that: The adjusting of network parameters to train and test the neural network specifically includes: Step S31: Implement the neural network using the Pytorch architecture and train and test it on a single NVIDIA P5000 GPU; Step S32: Set the optimizer to a stochastic gradient descent optimizer, with an initial learning rate of 0.001, a momentum of 0.9, a weight decay of 1e-4, an epoch of 90, and the number of sampling times and the sampling interval set according to specific needs; Step S33: For the event simulation COCO dataset, use the training data generated by the event simulation to perform network training, and the input is a set of event points and grayscale images corresponding to the event points; Step S34: The test data uses a data set generated by a DAVIS240C camera, including data of dynamic scenes, scenes of motion at different speeds, and indoor and outdoor scenes; Step S35: The evaluation indicators use mean square error MSE, structural similarity index SSIM and image perceptual similarity loss LPIPS to evaluate the quality of the reconstructed image.

5. The event stream image reconstruction method based on hierarchical uncertainty according to claim 1, characterized in that: The input event stream data specifically includes: The dataset was generated using DAVIS, where the camera produces event data and still image data at 24 frames per second; The network is trained using the event simulation COCO dataset, and the input is a set of event points and the grayscale images corresponding to the event points.

6. The event stream image reconstruction method based on hierarchical uncertainty according to claim 1, characterized in that: The random sampling to construct a multi-mask event frame specifically includes: Randomly sample the number of input event points and construct multiple groups of event frames containing different numbers of event points; Given an event stream E = {e i =(x i ,y i ,t i ,p i )|i=1,...,N}, where e i Represents a single event (x i ,y i ) is the pixel coordinate where the event occurs, t i is the timestamp, p i ∈{1,-1} is the brightness polarity, which means the brightness of the pixel increases or decreases respectively; Use voxel grid to transform the original event stream data E into a tensor-like event frame representation E∈R H×W×T , where H and W represent the height and width of the event frame respectively, T represents the voxel channel of the event frame, and T is set to 5; Define the trained event stream reconstruction network: f θ :E→I, I represents the reconstructed image; Use a random binary mask M∈[0,1] H×W , where the elements with a value of 0 in M ​​correspond to the occluded pixel positions, and M ij The value of is determined by a random probability distribution; Applying M to the original event frame E yields partial occlusion: in represents the element-by-element multiplication of the event frame and the mask, Ω = {(x, y) | M(x, y) = 1} is the set of pixel coordinates whose value is 1 in the mask M; It can be regarded as an incomplete observation of the original event frame sequence, reflecting the local interference of occlusion on scene perception; Will Input into the image reconstruction network to get the corresponding reconstruction result make represents the similarity measure between the unmasked image and the masked image; If the (i,j)th pixel is blocked, the original reconstructed image I and the masked reconstructed image If the local difference of is large, it indicates that the event information at this pixel is crucial to scene understanding, and occluding it will significantly affect the prediction of the reconstruction network; Define the importance R of the event at pixel (i, j) ij : Among them, E M Representing the expectation of pixel-level importance, combining local pixel-level masking operations with global pixel-level similarity assessment; For a continuous random probability distribution M, sample multiple layers of masks and approximate the expectation of the random probability distribution by calculating the variance after sampling; Generate K binary masks {M k ∈{0,1} H×W |k=1,...,K}, the importance of the event at pixel (i,j) is defined as follows: By randomly blocking the event information of different pixels and evaluating the impact of local missing information on the overall reconstruction effect, the pixel-level uncertainty distribution R = {R ij ∈R H×W }; The distribution reveals the spatial salient structure of the input event stream from a data-driven perspective, and the reconstruction algorithm focuses on semantically rich key areas.

7. The event stream image reconstruction method based on hierarchical uncertainty according to claim 3, characterized in that: The uncertainty modeling specifically includes: Define the uncertainty measure of the importance of the event at pixel (i, j) as U ij , that is, the importance estimate R in K occlusion trials ij The variance at this spatial location is: in, U={U ij ∈R H×W }, combine the event importance distribution R and the uncertainty distribution U to obtain a hierarchical representation that comprehensively describes the statistical characteristics of the input event stream E; Among them, R reflects the significant difference of events in image reconstruction and provides a reference for the attention allocation of the reconstruction algorithm; while U further reveals the reliability of this significant structure itself and introduces an adaptive adjustment mechanism for the reconstruction process; Based on the rich priors implied in R and U, an adaptive reconstruction loss function that integrates event importance and confidence perception will be designed to maximize the semantic potential of event stream data.

8. The event stream image reconstruction method based on hierarchical uncertainty according to claim 3, characterized in that: The uncertainty modeling also includes: The adaptive reconstruction loss specifically includes: Consider the event camera dataset Where E n represents the event frame sequence of the nth training sample, is the corresponding real image; For each training sample, calculate its event importance map and uncertainty diagram The adaptive reconstruction loss of the nth sample is defined as: in, and denote the values ​​of the reconstructed image and the real image of the nth sample at (i, j), s(·,·) denotes the pixel-by-pixel error metric, s(·,·) is replaced by the MSE loss, and at pixel (i, j) there is Compared with the standard average loss, formula (5) introduces an adaptive weight term Among them, β is the regularization coefficient, and the weight The importance of the event at pixel (i, j) in the nth sample is considered comprehensively and credibility

Images