An event stream image reconstruction method based on hierarchical uncertainty

Abstract

The application discloses an event stream image reconstruction method based on layered uncertainty, and the reconstruction method comprises the following steps: S1, inputting event stream data and randomly sampling to construct a plurality of mask event frames; S2, inputting the processed event frame into an event stream reconstruction network to obtain a plurality of reconstruction images, the reconstruction images sharing one real image, and establishing an uncertainty model between the reconstruction images and the real image; and S3, adjusting network parameters to train and test the neural network, and screening the images needing to be optimized from the uncertainty scores. The application improves the accuracy and quality of event stream image reconstruction, and enhances the adaptability and robustness of the model to dynamic scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and particularly relates to an event stream image reconstruction method based on hierarchical uncertainty. BACKGROUND

[0002] Event cameras are a new paradigm of vision inspired by biological vision. Unlike traditional cameras that record images at a fixed frame rate, event cameras work independently on each pixel and output event streams asynchronously. Event data has high temporal resolution, high dynamic range, and low power consumption, and therefore is highly concerned in the field of computer vision. In recent years, a large number of algorithms for event stream image reconstruction have been proposed, which can be generally divided into two categories: event stream image reconstruction based on traditional vision algorithms and event stream image reconstruction based on deep learning.

[0003] Traditional vision algorithms usually use a fixed number of event accumulation method to count a fixed number of events according to the polarity, so as to obtain a gray intensity image corresponding to the scene. The intera step S3ting visual maps (IVM) algorithm uses sparse event representation to jointly approximate inference of multiple visual quantities, including optical flow, gray intensity and camera motion. IVM takes into account parallelism and distributed computing, which makes it benefit from specific hardware, but its performance is poor on existing hardware architecture. The manifold regularization method models the reconstruction problem as an energy minimization defined on the event timestamp-induced manifold, which enables the temporal sequence information of events to be integrated into the reconstruction process without explicitly estimating the optical flow. However, this method avoids the estimation of optical flow, but has the disadvantage that it still needs to preprocess the timestamp field, the reconstruction quality depends on the camera performance, and it lacks the ability to estimate other vision tasks directly from event data. The fast temporal compensation method is a method of using a complementary filter to continuously fuse the image data of an event camera and a traditional camera, so as to obtain a sequence of enhanced images with high temporal resolution and high dynamic range.

[0004] The first problem is to solve the training difficulty caused by the scarcity of event camera and natural image paired data. Event camera simulator uses traditional RG step S2 camera images as input, and generates approximate event stream data by detecting intensity change points in the image. This event stream converted from image can effectively simulate the output of real camera, effectively improving the problem of difficult event camera data acquisition. But there are still differences between simulated data and real data. Event enhancement method uses data set threshold evaluation method to compare the similarity of common event camera data set and simulated data set, and through data enhancement, the original model shows better indicators in the learning of new data set. In addition, researchers try to use optical flow, degrees of freedom, rotation angle and other intermediate information to realize the transformation between the two data forms. EKLT (Event-based Kanade-Lu-based Tomasi) method uses optical flow constraint equation based on brightness constancy assumption to associate event with pixel intensity on natural image, and uses optical flow to estimate the change of event in time and space.

[0005] With the opening of related simulation technology and related data sets, event stream image reconstruction and video reconstruction work has begun to receive widespread attention. The most representative achievement is the E2VID (Events-to-Video) framework proposed by Re step S2e step S3q and his team. This work draws on the advantages of recurrent neural networks, inputs the image before generating sequence into the U-Net architecture, and gets the image sequence. This model has become the SOTA model in the field of event stream image generation since it was released, but E2VID still has room for improvement in running speed. In order to make the model have both performance and running speed, the team proposed a temporal continuity loss in FireNet method to help the model achieve real-time deployment in video reconstruction and high dynamic motion capture. SPADE-E2VID proposes an end-to-end data-driven framework for event stream video reconstruction. This method uses spatial adaptive regularization (SPADE) and recurrent network to directly reconstruct video from event, and adopts minimum dynamic loss to obtain more realistic reconstruction effect. SPADE-E2VID has made great improvement in method and effect, and its open source resources also have high reference value.

[0006] Existing technologies often struggle to achieve high-precision reconstruction of event stream images due to noise inherent in the event data itself. Traditional data-driven methods frequently fail to effectively address the impact of this noise, resulting in low-quality reconstructed images. Furthermore, existing technologies do not adequately differentiate between dynamic foreground and static background when handling dynamic scenes, potentially leading to loss of detail or errors in the reconstructed image. Moreover, existing solutions may lack flexibility and adaptability to different scenarios, failing to fully utilize multi-scale information within the scene, thus limiting their performance in diverse application scenarios. Summary of the Invention

[0007] In view of the above problems, the present invention is proposed to provide an event stream image reconstruction method based on hierarchical uncertainty to overcome or at least partially solve the above problems.

[0008] According to one aspect of the present invention, an event stream image reconstruction method based on hierarchical uncertainty is provided, the reconstruction method comprising:

[0009] Step S1: Input event stream data and randomly sample to construct multi-mask event frames;

[0010] Step S2: Input the processed event frame into the event stream reconstruction network to obtain multiple reconstructed images. The reconstructed images share a real image. Establish an uncertainty model between the reconstructed images and the real images.

[0011] Step S3: Adjust network parameters to train and test the neural network, and select images that need to be optimized from the uncertainty scores.

[0012] Optionally, step S1: inputting event stream data and randomly sampling to construct multi-mask event frames specifically includes:

[0013] Step S11: Given a stream of events, where a single event is represented by the pixel coordinates, timestamp, and brightness polarity of the event occurrence;

[0014] Step S12: Use a voxel mesh to transform the original event stream data into a tensor-like event frame representation, where the event frame has a height H, a width W, and voxel channels C;

[0015] Step S13: Define the trained event stream reconstruction network as f θ , represents the reconstructed image, where θ represents the network parameters;

[0016] Step S14: Use a random binary mask M, where elements with a value of 0 correspond to the positions of occluded pixels. The value of M is determined by a random probability distribution. Apply M to the original event frame to obtain the partially occluded event frame E'.

[0017] Optionally, establishing an uncertainty model between the reconstructed image and the real image specifically includes:

[0018] Step S21: Define the uncertainty measure of the importance of an event at a pixel as σ2, and the variance of the importance estimate in spatial location during multiple occlusion trials;

[0019] Step S22: Combine the event importance distribution and uncertainty distribution to obtain a hierarchical representation that comprehensively characterizes the statistical properties of the input event flow;

[0020] Step S23: Use the value of the loss function to estimate the confidence or weight of each event stream;

[0021] Step S24: Convert the loss function values ​​into confidence scores, and optimize the selection and combination of event streams based on the uncertainty modeling method of the loss function;

[0022] Step S25: By dynamically adjusting the weight of each event stream, we can adaptively adapt to different scenarios and data characteristics to obtain more robust and efficient image reconstruction results.

[0023] Optionally, adjusting network parameters for training and testing the neural network specifically includes:

[0024] Step S31: Implement the neural network using the PyTorch architecture and train and test it on a single NVIDIA P5000 GPU;

[0025] Step S32: Set the optimizer to a stochastic gradient descent optimizer with an initial learning rate of 0.001, momentum of 0.9, weight decay of 1e-4, and epoch of 90. The number of samples and the sampling interval are set according to specific requirements.

[0026] Step S33: For the event simulation COCO dataset, use the training data generated by the event simulation to train the network. The input is a set of event points and their corresponding grayscale images.

[0027] Step S34: The test data uses a dataset generated by the DAVIS240C camera, including data on dynamic scenes, scenes moving at different speeds, and indoor and outdoor scenes;

[0028] Step S35: The mean squared error (MSE), structural similarity index (SSIM), and perceptual image similarity loss (LPIPS) are used to evaluate the quality of the reconstructed image.

[0029] Optionally, the input event stream data specifically includes:

[0030] The dataset was generated using DAVIS, with the camera producing event data and generating still image data at 24 frames per second.

[0031] The network is trained using the COCO event simulation dataset, with the input being a set of event points and their corresponding grayscale images.

[0032] Optionally, the random sampling to construct the multi-mask event frame specifically includes:

[0033] The number of input event points is randomly sampled to construct multiple sets of event frames containing different numbers of event points;

[0034] Given an event stream E = {e i =(x i ,y i ,t i ,p i )|i=1,...,N}, where e i Represents a single event (x) i ,y i ) represents the pixel coordinates where the event occurred, and t represents the pixel coordinates where the event occurred. i For timestamps, p i ∈{1,-1} represents the brightness polarity, indicating whether the pixel brightness increases or decreases;

[0035] The raw event stream data E is transformed into a tensor-like event frame representation E∈R using a voxel grid. H×W×T Where H and W represent the height and width of the event frame, respectively, and T represents the voxel channel of the event frame, with T=5;

[0036] Define the trained event stream reconstruction network: f θ :E→I, where I represents the reconstructed image;

[0037] Using a random binary mask M∈[0,1] H×W In M, elements with a value of 0 correspond to the positions of occluded pixels. ij The value of is determined by a random probability distribution;

[0038] Applying M to the original event frame E yields the partial occlusion:

[0039]

[0040] in This represents the element-wise multiplication of the event frame and the mask, where Ω = {(x,y)|M(x,y) = 1} is the set of pixel coordinates with a value of 1 in the mask M;

[0041] It can be regarded as an incomplete observation of the original event frame sequence, reflecting the local interference of occlusion on scene perception;

[0042] Will The image is input into an image reconstruction network to obtain the corresponding reconstruction result.

[0043] make This measures the similarity between unmasked and masked images.

[0044] If the (i,j)th pixel is occluded, the original reconstructed image I and the masked reconstructed image... If the local differences are large, it indicates that the event information at that pixel is crucial for scene understanding, and occlusion will significantly affect the prediction of the reconstruction network.

[0045] Define the importance R of the event at pixel (i,j) ij :

[0046]

[0047] Among them, E M The expectation representing pixel-level importance combines local pixel-level masking operations with global pixel-level similarity evaluation;

[0048] For a continuous random probability distribution M, multiple sampling masks are used, and the expected value of the random probability distribution is approximated by calculating the variance after sampling.

[0049] Generate K binary masks {M} with the same spatial size as the event frame. k ∈{0,1} H×W In the context of |k=1,...,K}, the importance of the event at pixel (i,j) is defined as follows:

[0050]

[0051] By randomly occluding event information of different pixels and evaluating the impact of local missing information on the overall reconstruction effect, the pixel-level uncertainty distribution R = {R_{n+1}_{n+2}_{n+3}_{n+4}_{n+5}_{n+6}_{n+1}_{n+2}_{n+3}_{n+4 ... ij ∈R H×W};

[0052] The distribution reveals the spatial saliency structure of the input event stream from a data-driven perspective, and the reconstruction algorithm focuses on semantically rich key regions.

[0053] Optionally, the uncertainty modeling specifically includes:

[0054] Define U as the uncertainty measure of the importance of the event at pixel (i,j). ij That is, the importance estimate R in K occlusion trials. ij Variance at this spatial location:

[0055]

[0056] in, U={Uij ∈R H×W By combining the event importance distribution R and the uncertainty distribution U, a hierarchical representation that comprehensively characterizes the statistical properties of the input event flow E is obtained.

[0057] R reflects the saliency difference of events in image reconstruction, providing a reference for attention allocation in reconstruction algorithms; while U further reveals the reliability of this saliency structure itself, introducing an adaptive adjustment mechanism for the reconstruction process.

[0058] Based on the rich priors implied by R and U, an adaptive reconstruction loss function that integrates event importance and confidence awareness will be designed to maximize the semantic potential of event stream data.

[0059] Optionally, the uncertainty modeling further includes:

[0060] The adaptive reconstruction loss specifically includes:

[0061] Consider event camera dataset Where E n This represents the sequence of event frames for the nth training sample. Its corresponding real image;

[0062] For each training sample, calculate its event importance graph. and uncertainty diagram Define the adaptive reconstruction loss for the nth sample as:

[0063]

[0064] in, and Let represent the values ​​of the reconstructed image and the ground truth image of the nth sample at (i,j), respectively. s(·,·) represents the pixel-wise error metric, which is replaced by the MSE loss. The value at pixel (i,j) is...

[0065] Compared to the standard average loss, Equation (5) introduces an adaptive weighting term.

[0066]

[0067] Where β is the regularization coefficient and the weight is... The importance of the event at pixel (i,j) in the nth sample was taken into account. and credibility

[0068] This invention provides an event stream image reconstruction method based on hierarchical uncertainty. The reconstruction method includes: Step S1: Inputting event stream data and randomly sampling to construct multi-mask event frames; Step S2: Inputting the processed event frames into an event stream reconstruction network to obtain multiple reconstructed images, which share a single real image, and establishing an uncertainty model between the reconstructed images and the real image; Step S3: Adjusting network parameters to train and test the neural network, and selecting images that need optimization from the uncertainty scores. This improves the accuracy and quality of event stream image reconstruction and enhances the model's adaptability and robustness to dynamic scenes.

[0069] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0070] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 A flowchart of event flow image reconstruction based on hierarchical uncertainty modeling is provided for embodiments of the present invention;

[0072] Figure 2 This is an overall structural diagram of an event flow image reconstruction based on uncertainty modeling and self-stepping learning, provided for an embodiment of the present invention. Detailed Implementation

[0073] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0074] The terms "comprising" and "having," and any variations thereof, in the specification, embodiments, claims, and drawings of this invention are intended to cover non-exclusive inclusion, such as including a series of steps or units.

[0075] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0076] An event stream image reconstruction method based on hierarchical uncertainty. This invention relates to an event stream image reconstruction technique based on hierarchical uncertainty modeling, aiming to improve the accuracy and quality of event stream image reconstruction and enhance the model's adaptability and robustness to dynamic scenes. This technique is particularly suitable for event stream image reconstruction tasks in the field of computer vision. By constructing an adaptive image reconstruction framework, it can identify and process the hierarchical structure in event stream data, distinguishing the different importance of foreground and background event points.

[0077] This invention generates event frames with different granularities by randomly sampling event stream data and constructs a multi-input frame and dynamic loss function to model uncertainty. Furthermore, it employs a self-stepping learning strategy to progressively adapt network training, enhancing the model's ability to handle complex dynamic scenes and generating more detailed and higher-quality reconstructed images. The technical solution of this invention includes random sampling to construct multi-mask event frames, uncertainty modeling and self-stepping selection, and adjusting network parameters for neural network training and testing. Through this method, this invention not only improves the accuracy and quality of event stream image reconstruction but also enhances the model's adaptability and robustness to dynamic scenes, resulting in better semantic interpretability of the reconstruction results. This has significant practical implications for promoting the application of event cameras in the field of computer vision.

[0078] An event stream image reconstruction method based on hierarchical uncertainty modeling includes the following steps:

[0079] Step S1: Input event stream data and randomly sample a certain number of event points to construct multiple sets of event frames containing different numbers of event points;

[0080] Step S2: Input these processed event frames into the event stream reconstruction network to obtain multiple reconstructed images. These reconstructed images share a real image. Establish an uncertainty model between the reconstructed images and the real images.

[0081] Step S3: Filter the images that need to be optimized from the uncertainty scores to avoid reconstructed images with poor reliability from entering the optimization process, which would cause the model parameters to be unstable.

[0082] The specific steps for constructing multi-mask event frames through random sampling in step S1 include:

[0083] Given a stream of events, where a single event is represented by the pixel coordinates, timestamp, and brightness polarity of the event occurrence;

[0084] The raw event stream data is transformed into a tensor-like event frame representation using a voxel grid, where the height, width, and voxel channels of the event frame are represented as H, W, and C, respectively.

[0085] Define the trained event stream reconstruction network as f θ , represents the reconstructed image, where θ represents the network parameters;

[0086] A random binary mask M is used, where elements with a value of 0 correspond to the positions of occluded pixels. The value of M is determined by a random probability distribution. M is applied to the original event frame to obtain the partially occluded event frame E'.

[0087] The specific steps for uncertainty modeling and self-step selection in step S2 include:

[0088] The uncertainty measure of the importance of an event at a pixel is defined as σ², which is the variance of the importance estimate at that spatial location in multiple occlusion trials.

[0089] By combining the event importance distribution and the uncertainty distribution, a hierarchical representation that comprehensively characterizes the statistical properties of the input event stream is obtained. The event importance distribution reflects the saliency differences of events in image reconstruction, while the uncertainty distribution reveals the reliability of this saliency structure itself.

[0090] The confidence or weight of each event stream is estimated using the value of the loss function. The higher the confidence score, the greater the contribution of the event stream to the reconstruction quality and the more critical the information it contains.

[0091] The loss function values ​​are converted into confidence scores, and the uncertainty modeling method based on the loss function is used to optimize the selection and combination of event streams, thereby improving reconstruction quality and reducing computational overhead.

[0092] By dynamically adjusting the weights of each event stream, it adaptively adapts to different scenarios and data characteristics, resulting in more robust and efficient image reconstruction results.

[0093] The specific steps for adjusting network parameters to train and test the neural network in step S3 include:

[0094] The neural network was implemented using the PyTorch architecture and trained and tested on a single NVIDIA P5000 GPU;

[0095] Set the optimizer to a stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.001, momentum of 0.9, weight decay of 1e-4, and epoch of 90. The number of samples and the sampling interval are set according to specific requirements.

[0096] For the event simulation COCO dataset, the network is trained using training data generated by event simulation, with the input being a set of event points and their corresponding grayscale images;

[0097] The test data used a dataset generated by the DAVIS240C camera, including data on dynamic scenes, scenes moving at different speeds, and indoor and outdoor scenes;

[0098] The evaluation metrics used were mean squared error (MSE), structural similarity index (SSIM), and perceptual image similarity loss (LPIPS) to assess the quality of the reconstructed images.

[0099] Step 1: Construct a layered masking module

[0100] 1. Input of training data

[0101] The method proposed in this invention takes event stream data as input. Specifically, the dataset is generated using a DAVIS (Dynamic and Active-Pixel Vision Sensor), which produces event data and generates still image data at 24 frames per second. The network is trained using the event-simulated COCO dataset, with the input being a set of event points and their corresponding grayscale images.

[0102] 2. Construct multi-mask event frames using random sampling

[0103] In the process of event stream image reconstruction, the choice of the number of event points has a direct impact on the quality of the reconstruction result. In order to solve the problem caused by the fixed number of event points in existing methods, this invention proposes a new training method, which is to randomly sample the number of input event points and construct multiple sets of event frames containing different numbers of event points.

[0104] Given an event stream E = {e i =(x i ,y i ,t i ,p i )|i=1,...,N}, where e i Represents a single event (x) i ,y i ) represents the pixel coordinates where the event occurred, and t represents the pixel coordinates where the event occurred. i For timestamps, p i ∈{1,-1} represents the brightness polarity, indicating whether the pixel brightness increases or decreases. Furthermore, a voxel mesh is used to transform the original event stream data E into a tensor-like event frame representation E∈R. H ×W×T Here, H and W represent the height and width of the event frame, respectively, and T represents the voxel channel of the event frame. Here, T = 5 is set.

[0105] Define the trained event stream reconstruction network: f θ :E→I, where I represents the reconstructed image.

[0106] Using a random binary mask M∈[0,1] H×W In M, elements with a value of 0 correspond to the positions of occluded pixels. ij The value of is determined by a random probability distribution.

[0107] Applying M to the original event frame E yields the partial occlusion:

[0108]

[0109] in This represents the element-wise multiplication of the event frame and the mask, where Ω = {(x,y)|M(x,y) = 1} is the set of pixel coordinates with a value of 1 in the mask M. This can be viewed as an incomplete observation of the original event frame sequence, reflecting the local interference of occlusion on scene perception. By inputting the image into an image reconstruction network, the corresponding reconstruction result can be obtained.

[0110] make This represents a similarity measure between the unmasked image and the masked image. If the (i,j)th pixel is occluded, the original reconstructed image I and the masked reconstructed image... Significant local differences indicate that event information at that pixel is crucial for scene understanding, and occlusion will significantly impact the predictions of the reconstruction network. Therefore, the importance R of the event at pixel (i,j) is defined as follows: ij :

[0111]

[0112] Among them, E M Representing the expectation of pixel-level importance, Equation (4-2) combines local pixel-level masking operations with global pixel-level similarity evaluation. By observing the impact of local masking on global similarity, the degree of uncertainty of different pixel regions in the overall image representation can be inferred, i.e., R0. ij The larger the value, the more difficult it is to reconstruct that pixel, and therefore the more important it is.

[0113] For a continuous random probability distribution M, the expected value of the distribution can be obtained by integral.

[0114] The sampling multi-layer mask is used to approximate the expectation of the random probability distribution by calculating the variance after sampling.

[0115] Generate K binary masks {M} with the same spatial size as the event frame. k ∈{0,1} H×W In the context of |k=1,...,K}, the importance of the event at pixel (i,j) is defined as follows:

[0116]

[0117] By randomly occluding event information of different pixels and evaluating the impact of local missing information on the overall reconstruction effect, the pixel-level uncertainty distribution R = {R_{n+1}_{n+2}_{n+3}_{n+4}_{n+5}_{n+6}_{n+1}_{n+2}_{n+3}_{n+4 ... ij ∈R H×W}

[0118] This distribution reveals the spatial saliency structure of the input event stream from a data-driven perspective, helping the reconstruction algorithm focus on semantically rich key regions. However, due to the introduction of random occlusion, the event importance estimated by Equation (3) inevitably contains statistical noise. To measure the confidence difference in the importance of different pixels, its hierarchical uncertainty will be further modeled.

[0119] Step 2: Uncertainty Modeling

[0120] 1. Uncertainty Modeling

[0121] Although the random occlusion strategy can efficiently assess the relative importance of events among different pixels, the importance distribution R obtained by Equation (3) may still be affected by biased sampling. For example, in the background region around a moving target, the importance of individual pixels may be overestimated due to the coincidence of occlusion patterns. To assess the reliability of the importance of each pixel, a hierarchical uncertainty measure is introduced.

[0122] Define U as the uncertainty measure of the importance of the event at pixel (i,j). ij That is, the importance estimate R in K occlusion trials. ij Variance at this spatial location:

[0123]

[0124] in, U={U ij ∈R H×W By combining the event importance distribution R and the uncertainty distribution U, a hierarchical representation that comprehensively characterizes the statistical properties of the input event stream E is obtained. R reflects the saliency differences of events in image reconstruction, providing a reference for attention allocation in the reconstruction algorithm; while U further reveals the reliability of this saliency structure itself, introducing an adaptive adjustment mechanism for the reconstruction process. Based on the rich priors implied by R and U, the next step will be to design an adaptive reconstruction loss function that integrates event importance and confidence awareness, aiming to maximize the semantic potential of event stream data.

[0125] 2. Adaptive Reconstruction Loss

[0126] Traditional event image reconstruction methods typically employ a pixel-wise averaging strategy to measure reconstruction error (such as l1 / l2 loss), ignoring the inherent hierarchical semantic structure of event data.

[0127] To better utilize the prior information revealed by the aforementioned event importance distribution R and uncertainty distribution U, an adaptive weighted reconstruction loss function is proposed.

[0128] Consider event camera dataset Where E n This represents the sequence of event frames for the nth training sample. This corresponds to the real-world image. For each training sample, first calculate its event importance map. and uncertainty diagram Then, based on this, the adaptive reconstruction loss for the nth sample is defined as:

[0129]

[0130] in, and Let represent the values ​​of the reconstructed image and the ground truth image of the nth sample at (i,j), respectively. s(·,·) represents the pixel-wise error metric, where s(·,·) is replaced by the MSE loss, i.e., the error at pixel (i,j). Compared to the standard average loss, Equation (5) introduces an adaptive weighting term.

[0131]

[0132] Where β is the regularization coefficient. Weight The importance of the event at pixel (i,j) in the nth sample was taken into account. and credibility

[0133] On the one hand, the higher the semantic importance of the event corresponding to pixel (i,j) (i.e. The larger the value of f, the greater the proportion of its location reconstruction error in the total loss, thus driving the reconstruction network f to... θ Prioritize the restoration quality of this area. On the other hand, for pixels with more uncertain event importance estimates (i.e., The larger the value, the lower the weight of the corresponding loss term should be. Weaken the influence of unreliable regions and make the reconstruction process avoid the misleading influence of biased priors as much as possible.

[0134] Adaptive weights Make network f θThe training objective has shifted from minimizing the global average error to adaptive local optimization that balances event importance and confidence. Equation (5) applies attention to different pixels commensurate with their semantic contributions, while reflecting the reliability of those contributions with discriminative adjustment magnitudes.

[0135] The loss function will guide f θ By focusing on the most informative and statistically deterministic salient patterns in event stream data, more targeted and robust reconstructions can be achieved.

[0136] Step 3: Adjust network parameters and train and test the neural network.

[0137] 1. Parameter settings

[0138] This invention employs the PyTorch architecture and is implemented on a single NVIDIA P5000 GPU, using a Stochastic Gradient Descent (SGD) optimizer. For the event simulation COCO dataset, the initial learning rate is set to 0.001, momentum to 0.9, weight decay to 1e-4, epoch to 90, sampling count K=5, and sampling interval to [0.2HW, 0.5HW].

[0139] 2. Input test data

[0140] The test data uses a dataset generated by a DAVIS240C camera, which provides an event stream with microsecond timestamps, IMU measurements (3-axis gyroscope and accelerometer) acquired at 1 kHz, and 6-DOF true pose data obtained from a motion capture system at 200 Hz after hand-eye calibration. It contains six sub-datasets: dynamic_6dof, boxes_6dof, poster_6dof, office_zigzag, slider_depth, and calibration.

[0141] Beneficial effects: This invention not only improves the accuracy and quality of event stream image reconstruction, but also enhances the model's adaptability and robustness to dynamic scenes, making the reconstruction results more semantically interpretable. It has important practical significance for promoting the application of event cameras in the field of computer vision.

[0142] This event stream image reconstruction technique, based on hierarchical uncertainty modeling, improves the accuracy and quality of event stream image reconstruction and enhances the model's adaptability and robustness to dynamic scenes. The technique is particularly suitable for event stream image reconstruction tasks in computer vision. By constructing an adaptive image reconstruction framework, it can identify and process the hierarchical structure in event stream data, distinguishing the different importance of foreground and background event points.

[0143] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An event stream image reconstruction method based on hierarchical uncertainty, characterized in that, The reconstruction method includes: Step S1: Input event stream data and randomly sample to construct multi-mask event frames; Step S2: Input the processed event frames into the event stream reconstruction network to obtain multiple reconstructed images. These reconstructed images share a single ground truth image. An uncertainty model is established between the reconstructed images and the ground truth image, including: Step S21: Define the uncertainty measure of the event importance at a pixel as U. ij The variance of importance estimation at corresponding spatial locations in multiple occlusion trials; Step S22: Combine the event importance distribution and uncertainty distribution to obtain a hierarchical representation that comprehensively characterizes the statistical properties of the input event flow; Step S23: Use the value of the loss function to estimate the confidence or weight of each event stream; Step S24: Convert the loss function values ​​into confidence scores, and optimize the selection and combination of event streams based on the uncertainty modeling method of the loss function; Step S25: By dynamically adjusting the weight of each event stream, we can adaptively adapt to different scenarios and data characteristics to obtain more robust and efficient image reconstruction results; The establishment of the uncertainty model specifically includes: Define U as the uncertainty measure of the importance of the event at pixel (i,j). ij That is, the importance estimate R in K occlusion trials ij Variance at this spatial location: in, U={U ij ∈R H×W By combining the event importance distribution R and the uncertainty distribution U, a hierarchical representation that comprehensively characterizes the statistical properties of the input event flow E is obtained. The event frame has a height H and a width W, and the event stream reconstruction network is f. θ , The vector product symbol, the random binary mask M, s(·,·) represents the pixel-wise error measure, and I represents the reconstructed image; R reflects the saliency difference of events in image reconstruction, providing a reference for attention allocation in reconstruction algorithms; while U further reveals the reliability of this saliency structure itself, introducing an adaptive adjustment mechanism for the reconstruction process. Based on the rich priors implied by R and U, an adaptive reconstruction loss function that integrates event importance and confidence awareness is designed to maximize the semantic potential of event stream data. The establishment of the uncertainty model also includes: The adaptive reconstruction loss specifically includes: Consider event camera dataset Where E n This represents the sequence of event frames for the nth training sample. Its corresponding real image; For each training sample, calculate its event importance graph. and uncertainty diagram Define the adaptive reconstruction loss for the nth sample as: in, and Let represent the values ​​of the reconstructed image and the ground truth image of the nth sample at (i,j), respectively. s(·,·) represents the pixel-wise error metric, which is replaced by the MSE loss. The value at pixel (i,j) is... Compared to the standard average loss, Equation (5) introduces an adaptive weighting term. Where β is the regularization coefficient and the weight is... The importance of the event at pixel (i,j) in the nth sample was taken into account. and credibility Step S3: Adjust network parameters to train and test the event flow reconstruction network, and select images that need to be optimized from the uncertainty scores.

2. The event stream image reconstruction method based on hierarchical uncertainty according to claim 1, characterized in that, Step S1: Inputting event stream data and randomly sampling to construct multi-mask event frames specifically includes: Step S11: Given a stream of events, where a single event is represented by the pixel coordinates, timestamp, and brightness polarity of the event occurrence; Step S12: Use a voxel mesh to transform the original event stream data into a tensor-like event frame representation, where the event frame has a height H, a width W, and voxel channels C; Step S13: Define the trained event stream reconstruction network as f θ , represents the reconstructed image, where θ represents the network parameters; Step S14: Use a random binary mask M, where elements with a value of 0 correspond to the positions of occluded pixels. The value of M is determined by a random probability distribution. Apply M to the original event frame to obtain the partially occluded event frame E'.

3. The event stream image reconstruction method based on hierarchical uncertainty according to claim 1, characterized in that, The training and testing of the event flow reconstruction network by adjusting network parameters specifically includes: Step S31: Implement the neural network using the PyTorch architecture and train and test it on a single NVIDIA P5000 GPU; Step S32: Set the optimizer to a stochastic gradient descent optimizer with an initial learning rate of 0.001, momentum of 0.9, weight decay of 1e-4, and epoch of 90. The number of samples and the sampling interval are set according to specific requirements. Step S33: For the event simulation COCO dataset, use the training data generated by the event simulation to train the network. The input is a set of event points and their corresponding grayscale images. Step S34: The test data uses a dataset generated by the DAVIS240C camera, including data on dynamic scenes, scenes moving at different speeds, and indoor and outdoor scenes; Step S35: The mean squared error (MSE), structural similarity index (SSIM), and perceptual image similarity loss (LPIPS) are used to evaluate the quality of the reconstructed image.

4. The event stream image reconstruction method based on hierarchical uncertainty according to claim 1, characterized in that, The input event stream data specifically includes: The dataset was generated using DAVIS, with the camera producing event data and generating still image data at 24 frames per second. The network is trained using the COCO event simulation dataset, with the input being a set of event points and their corresponding grayscale images.

5. The event stream image reconstruction method based on hierarchical uncertainty according to claim 1, characterized in that, The random sampling to construct multi-mask event frames specifically includes: The number of input event points is randomly sampled to construct multiple sets of event frames containing different numbers of event points; Given an event stream Where e i Represents a single event, (x) i ,y i ) represents the pixel coordinates where the event occurred, and t represents the pixel coordinates where the event occurred. i For timestamps, p i ∈{1,-1} represents the brightness polarity, indicating whether the pixel brightness increases or decreases; Use voxel grids to extract the raw event stream data Transform into an event frame representation E∈R of a tensor H×W×T Where H and W represent the height and width of the event frame, respectively, and T represents the voxel channel of the event frame, with T=5; Define the trained event stream reconstruction network: f θ :E→I, where I represents the reconstructed image; Using a random binary mask M∈[0,1] H×W In M, elements with a value of 0 correspond to the positions of occluded pixels. ij The value of is determined by a random probability distribution; Applying M to the original event frame E yields the partial occlusion. in This represents the element-wise multiplication of the event frame and the mask, where Ω = {(x,y)|M(x,y) = 1} is the set of pixel coordinates with a value of 1 in the mask M; It can be regarded as an incomplete observation of the original event frame sequence, reflecting the local interference of occlusion on scene perception; Will The input is fed into the event stream reconstruction network to obtain the corresponding reconstruction results. make This measures the similarity between unmasked and masked images. If the (i,j)th pixel is occluded, the original reconstructed image I and the masked reconstructed image... If the local differences are large, it indicates that the event information at that pixel is crucial for scene understanding, and occlusion will significantly affect the prediction of the reconstruction network. Define the importance R of the event at pixel (i,j) ij : Among them, E M The expectation representing pixel-level importance combines local pixel-level masking operations with global pixel-level similarity evaluation; For a continuous random probability distribution M, multiple sampling masks are used, and the expected value of the random probability distribution is approximated by calculating the variance after sampling. Generate K binary masks {M} with the same spatial size as the event frame. k ∈{0,1} H×W In the context of |k=1,...,K}, the importance of the event at pixel (i,j) is defined as follows: By randomly occluding event information of different pixels and evaluating the impact of local missing information on the overall reconstruction effect, the pixel-level importance distribution R = {R_{n+1}_{n+2}_{n+3}_{n+4}_{n+5}_{n+6} ... ij ∈R H×W }; The distribution reveals the spatial saliency structure of the input event stream from a data-driven perspective, and the reconstruction algorithm focuses on semantically rich key regions.

Images