An event camera image reconstruction method based on brightness constancy

Abstract

The present invention discloses an event camera image reconstruction method based on brightness constancy, which solves the problem of how to reconstruct a clear intensity image without artifacts from event data without prior information, including: dividing the event camera data set Event Camera Dataset into a training set and a test set; obtaining pixel brightness estimation values ​​through a ReconNet network; under the constraint of brightness constancy, obtaining the optical flow estimation value of the pixel through optical flow estimation and jointly calculating the pixel brightness change estimation value with the brightness estimation value; comparing the brightness change estimation value with the brightness change reference value to obtain the error, and back-propagating to obtain an event flow image reconstruction network model based on brightness constancy; using the training set as input, the loss function tends to converge to obtain a trained neural network model; using the test set as input, and verifying the quality effect of the reconstructed image with a mean square error index and a structural similarity index. The present invention effectively removes artifacts without prior information and realizes image reconstruction with high-definition resolution.

Description

Technical Field

[0001] The invention relates to an event camera image reconstruction method based on brightness constancy, and belongs to the technical field of image processing. Background Art

[0002] Event cameras have great application advantages in dynamic scenes. When the logarithmic light intensity change of a pixel exceeds a certain sensitivity threshold at a certain time, the camera sends an event, which contains three information: pixel coordinates, time information and polarity; represented by symbols, such as event e(x, y, t, p); where (x, y) is the pixel coordinate of the event in two-dimensional space, t is the time when the event occurs, and p is the polarity of the event; the polarity of the event represents the brightness change of the scene, that is, increase (Positive) or decrease (Negative).

[0003] Since the events recorded by the event camera are only symbolic representations, the visualization effect is not intuitive and lacks texture information. They are not as informative as optical images, easy to understand, and convenient for analyzing targets. Therefore, it is of great significance to reconstruct scene intensity images from the event data recorded by the event camera.

[0004] In the prior art, the image is reconstructed from the event data recorded by the event camera by integrating the statistical data and regularizing the reconstruction method through prior information; however, due to the difficulty in obtaining prior information and the difference between the event camera and the ideal model, the image reconstructed by integrating the statistical data and regularizing the reconstruction method through prior information has low clarity and resolution, and poor universality for the scene. In addition, the reconstruction method based on neural network does not require prior information of camera motion parameters and scenes, but requires a large amount of training data, slow iterative convergence, and unrealistic artifacts. The quality of the reconstructed image needs to be further improved. Summary of the invention

[0005] The event information recorded by the event camera is only a symbolic representation, the visualization effect is not intuitive, and there is a lack of texture information. It is not as informative as an optical image, easy to understand, and convenient for analyzing the characteristics of the target; and the images reconstructed based on difficult-to-obtain prior information in the prior art are prone to unrealistic ghosting and artifacts. The present invention provides an event camera image reconstruction method based on brightness constancy, designs an event stream image reconstruction network model based on brightness constancy, and trains and verifies the network to reconstruct a clear intensity image from event data containing a large amount of information recorded by the event camera, eliminating artifacts existing in traditional reconstructed images. This solution can effectively remove the influence of artifacts without the need for any prior information, making the image clearer, while increasing the contrast of the reconstructed image, making the edge contour information of the target clearer, and achieving high-definition resolution video reconstruction.

[0006] In order to achieve the above object, the technical solution disclosed in the present invention is as follows:

[0007] The present invention discloses an event camera image reconstruction method based on brightness constancy, which is characterized by comprising:

[0008] Step 1: Get the Event Camera Dataset and divide it into a training set and a test set.

[0009] Step 2: Obtain the brightness estimation value of the pixel through the ReconNet network; Under the constraint of brightness constancy, obtain the optical flow estimation value of the pixel through optical flow estimation, and jointly calculate the optical flow estimation value and the brightness estimation value to obtain the brightness change estimation value of the pixel; Compare the brightness change estimation value with the brightness change reference value of the pixel obtained by integrating the event data to obtain the error, and back-propagate the error through the ReconNet network to obtain the constructed event flow image reconstruction network model based on brightness constancy;

[0010] Step 3: Construct a loss function; input the training set data into the event stream image reconstruction network model based on brightness constancy to perform network learning training; when the loss function converges, the network training is completed and a trained neural network model is obtained;

[0011] Step 4: Input the test set data into the trained neural network model, and verify the quality of the reconstructed image using the mean square error index and / or the structural similarity index.

[0012] In step 1, the event camera dataset Event Camera Dataset is obtained, and the method of dividing it into a training set and a test set includes:

[0013] The DAVIS240C event camera is used to record real scenes to obtain event data and ground truth grayscale full-frame images, and the Event Camera Dataset is obtained.

[0014] The event camera dataset Even Camera Dataset includes dynamic_6dof subset, boxes_6dof subset, poster_6dof subset, office_zigzag subset, slider_depth subset, shapes_6dof subset and calibration subset. The calibration subset is used as the test set, and the remaining subsets are used as training sets.

[0015] In step 2, the brightness constancy constraint is: when the pixel of the object on the pixel plane moves from the pixel coordinate (x, y) to the target pixel coordinate (x+Δx, y+Δy), and the time interval Δt is within the specified threshold, it is considered that the brightness of the pixel has not changed, which is expressed as:

[0016] L(x,y,t)=L(x+Δx,y+Δy,t+Δt);

[0017] Wherein, L(x, y, t) is a nonlinear function of the brightness of the pixel, the pixel coordinates (x, y) and the time t; Δx and Δy are motion vectors within a specified threshold; and Δt is the time interval.

[0018] In step 2, under the constraint of brightness constancy, the method of obtaining the optical flow estimation value of the pixel through optical flow estimation is:

[0019] The brightness constancy constraint is expanded into a first-order linear function through Taylor's formula, and the partial derivatives of the pixel coordinates (x, y) and time t are obtained:

[0020] L x u+L y v+L t =0;

[0021] The motion is jointly estimated by the pixels of the adjacent pixel number N, and the optical flow estimation value is obtained:

[0022]

[0023] Among them, L x , L y , L t are the inverse of the brightness with respect to the pixel coordinates (x, y) and time t, respectively; N is the number of adjacent pixels; u, v are motion vectors; is the optical flow estimate.

[0024] In step 2, the method of jointly calculating the optical flow estimation value and the brightness estimation value to obtain the brightness change estimation value of the pixel includes:

[0025]

[0026] in, is the estimated value of brightness change; is the optical flow estimate The warping function of is the derivative of the brightness estimate.

[0027] In step 2, the brightness change reference value of the pixel obtained by integrating the event data is:

[0028]

[0029] Among them, ΔL * is the brightness change reference value; i is the event index value; θ is the number of all events within the time interval Δt; p i is the polarity of the ith event; C is the event camera threshold.

[0030] In step three, the constructed loss function is:

[0031]

[0032] Among them, Loss is the loss function; S is the number of steps of the cyclic network during training; k is the serial index; is the brightness reconstruction loss; λ1 is the balance coefficient between the temporal consistency loss and the brightness reconstruction loss; It is the loss of temporal consistency.

[0033] In step 4, the method of verifying the quality of the reconstructed image using the mean square error index is:

[0034]

[0035] Where MSE is the mean square error; m is the number of image pixels, j is the index value of the reconstructed image; ω j >0 is the coefficient, y j is the real image pixel intensity value, is the reconstructed image pixel intensity value.

[0036] In step 4, the method of verifying the quality of the reconstructed image using the structural similarity index is:

[0037] SSIM(I r ,I g )=l(I r ,I g )+c(I r ,I g )+s(I r ,I g );

[0038] Among them, SSIM(I r ,I g ) is the structural similarity between the reconstructed image and the real image; l(I r ,I g ) is the brightness similarity between the reconstructed image and the real image; c(I r ,I g ) is the contrast similarity between the reconstructed image and the real image; s(I r ,I g ) is the structural similarity between the reconstructed image and the real image.

[0039] The beneficial effects of the present invention are as follows:

[0040] The present invention constructs an event stream image reconstruction network model based on brightness constancy, and uses the event camera dataset Event Camera Dataset containing event data and ground truth grayscale full-frame images for training to obtain a trained model. By inputting event data into the trained network model and outputting high-resolution images, the reconstruction process of pure event stream to high-resolution intensity images is realized.

[0041] The brightness estimation of the pixel is obtained through the ReconNet network; under the constraint of brightness constancy, the optical flow estimation of the pixel is obtained through optical flow estimation, and the brightness change estimation of the pixel is calculated jointly with the optical flow estimation value; the brightness change estimation value is compared with the brightness change reference value of the pixel obtained by integrating the event data to obtain the error, and the error is back-propagated through the ReconNet network to obtain the constructed event flow image reconstruction network model based on brightness constancy. Based on the image reconstruction network, this model introduces the brightness constancy constraint, compares the calculated brightness estimation value with the brightness change reference value, and optimizes the training process through error back-propagation, so that the grayscale value in the reconstructed image is closer to the actual situation, and is better than the MR and E2VID methods.

[0042] In order to better describe the difference between the reconstructed image and the real image, the loss function is described by brightness reconstruction loss and time consistency loss. The brightness reconstruction loss follows the maximum likelihood method, so that the reconstructed image is closer to the real situation in terms of brightness value. The network relies on cyclic connections to enhance the temporal consistency problem between continuous reconstructions, which can effectively optimize the situation when there is flickering in the scene and artifact problems. The training set data is input into the event stream image reconstruction network model based on brightness constancy for network learning training; when the loss function tends to converge, the network training is completed and the trained neural network model is obtained; the test set data is input into the trained neural network model, and the mean square error index and structural similarity index are used to verify the quality effect of the reconstructed image. The mean square error value is smaller than that of the MR and E2VID methods, and the structural similarity value is greater than that of the MR and E2VID methods. The image reconstruction quality effect is verified. The technical scheme of the present invention is compared with the MR and E2VID reconstruction methods. The average mean square error MSE of the technical scheme of the present invention tested in the calibration subset in the event camera dataset Event Camera Dataset is 12.55%, which is less than 18.35% of E2VID and 33.47% of MR; at the same time, the structural similarity SSIM is 91.76%, which is greater than 87.19% of E2VID and 20.64% of MR, which proves that the technical scheme disclosed in the present invention can effectively reduce the root mean square error between the reconstructed image and the real image, increase the structural similarity between the reconstructed image and the real image, and effectively improve the quality of the reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic diagram of the principle of the event camera image reconstruction method based on brightness constancy;

[0044] Figure 2 This is a schematic diagram of the ReconNet network structure;

[0045] Figure 3 It is a real grayscale full-frame image in the Event Camera Dataset;

[0046] Figure 4 It is the reconstructed image of the event data MR method in the Event Camera Dataset;

[0047] Figure 5 It is the reconstructed image of the event data E2VID method in the Event Camera Dataset;

[0048] Figure 6 It is a reconstructed image of the event data in the Event Camera Dataset using the method of the present invention. DETAILED DESCRIPTION

[0049] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0050] The embodiment of the present invention discloses an event camera image reconstruction method based on brightness constancy. The purpose of event camera image reconstruction is to reconstruct event data into an optical image to facilitate the analysis of target features. The method includes:

[0051] Step 1: Get the Event Camera Dataset and divide it into a training set and a test set.

[0052] Step 2: Obtain the brightness estimation value of the pixel through the ReconNet network; Under the constraint of brightness constancy, obtain the optical flow estimation value of the pixel through optical flow estimation, and jointly calculate the optical flow estimation value and the brightness estimation value to obtain the brightness change estimation value of the pixel; Compare the brightness change estimation value with the brightness change reference value of the pixel obtained by integrating the event data to obtain the error, and back-propagate the error through the ReconNet network to obtain the constructed event flow image reconstruction network model based on brightness constancy;

[0053] Step 3: Construct a loss function; input the training set data into the event stream image reconstruction network model based on brightness constancy to perform network learning training; when the loss function converges, the network training is completed and a trained neural network model is obtained;

[0054] Step 4: Input the test set data into the trained neural network model, and verify the quality of the reconstructed image using the mean square error index and / or the structural similarity index.

[0055] In step 1, the event camera dataset Event Camera Dataset is obtained, and the method of dividing it into a training set and a test set includes:

[0056] like Figure 3 As shown in the figure, the DAVIS240C event camera is used to record real scenes to obtain event data and ground truth grayscale full-frame images, and the Event Camera Dataset is obtained; currently, the dataset is public and available as open source.

[0057] The event camera dataset Even Camera Dataset includes dynamic_6dof subset, boxes_6dof subset, poster_6dof subset, office_zigzag subset, slider_depth subset, shapes_6dof subset and calibration subset. The calibration subset is used as the test set, and the remaining subsets are used as training sets.

[0058] The purpose of constructing a data set is to train the network constructed by the present invention, the input of the network is event data, and the output is an optical image. A training model is obtained by training using a data set containing event data and an optical image, and new event data is input into the model to output the corresponding optical image, thereby realizing the reconstruction process from event data to optical image.

[0059] like Figure 2 As shown in Figure 2, in step 2, the ReconNet network adopts a U-shaped network structure, and the brightness estimation value of the pixel is obtained through the ReconNet network.

[0060] The ReconNet network disclosed in the embodiment of the present invention consists of a head layer H, a cyclic encoder layer ε, a residual block R, a decoder layer D and a final image prediction layer P. Similar to the U-net sampling method, the output number channel N of the head layer b , doubled after each encoder. The encoding layer consists of a 5-core 2D downsampling convolutional layer and a 3-core ConvSLTM. The kernel size of the intermediate residual block is 3, and each decoder consists of a bilinear upsampling layer and a convolutional layer with a kernel size of 5. Except for the final prediction layer, each layer is activated by ReLU and batch normalized. Skip connections are used between the encoder and decoder.

[0061] In step 2, the brightness constancy constraint is: when the pixel of the object on the pixel plane moves from the pixel coordinate (x, y) to the target pixel coordinate (x+Δx, y+Δy), and the time interval Δt is within the specified threshold, it is considered that the brightness of the pixel has not changed, which is expressed as:

[0062] L(x,y,t)=L(x+Δx,y+Δy,t+Δt);

[0063] Wherein, L(x, y, t) is a nonlinear function of the brightness of the pixel, the pixel coordinates (x, y) and the time t; Δx and Δy are motion vectors within a specified threshold; and Δt is the time interval.

[0064] In step 2, under the constraint of brightness constancy, the method of obtaining the optical flow estimation value of the pixel through optical flow estimation is:

[0065] The brightness constancy constraint is expanded into a first-order linear function through Taylor's formula, and the partial derivatives of the pixel coordinates (x, y) and time t are obtained:

[0066] L x u+L y v+L t =0;

[0067] The motion is jointly estimated by the pixels of the adjacent pixel number N, and the optical flow estimation value is obtained:

[0068]

[0069] Among them, L x , L y , L t are the inverse of the brightness with respect to the pixel coordinates (x, y) and time t, respectively; N is the number of adjacent pixels; u, v are motion vectors; is the optical flow estimate.

[0070] Optical flow refers to the flow of light, which is a velocity vector of brightness change caused by the relative motion between the object and the observer. Simply put, the position vector of the same pixel point in two images is called optical flow. When an event camera images a moving object, due to the motion of the object or the camera, the same part of the edge of the observed object is triggered to generate events with different timestamps and pixel positions. The generation of these events causes temporal and spatial dislocation between events, resulting in overlapping target edges and even blurred images. The sparse event stream generated by the event camera can describe the motion through sparse optical flow. In the event sequence, the optical flow can show the change of pixels in the time domain and the correlation and correspondence of events in adjacent time periods, and then estimate the motion information of the object. Therefore, the optical flow estimation method can optimize the problem of blurring in the reconstructed image. Optical flow estimation based on event flow is mainly based on the invariance of light intensity, that is, brightness invariance, in an infinitely short duration. The high temporal resolution of events can overcome the limitations of the method in large displacement estimation, improve the accuracy of optical flow estimation, and event-based optical flow estimation is superior to the existing frame-based optical flow estimation technology. Optical flow estimation is combined with intensity estimation to achieve deblurring of the reconstructed image and remove the influence of artifacts.

[0071] In step 2, the method of jointly calculating the optical flow estimation value and the brightness estimation value to obtain the brightness change estimation value of the pixel includes:

[0072]

[0073] in, is the estimated value of brightness change; is the optical flow estimate The warping function of is the derivative of the brightness estimate.

[0074] In step 2, the brightness change reference value of the pixel obtained by integrating the event data is:

[0075]

[0076] Among them, ΔL * is the brightness change reference value; i is the event index value; θ is the number of all events within the time interval Δt; p i is the polarity of the ith event; C is the event camera threshold.

[0077] To facilitate the understanding of the formula disclosed in step 2, the derivation process of step 2 is as follows:

[0078] The brightness constancy constraint can be expressed as:

[0079] L(x,y,t)=L(x+Δx,y+Δy,t+Δt); (3-1)

[0080] The event is caused by the change of illumination at the edge of the moving object. Under the constraint of brightness constancy, the brightness change value can be expressed by gradient and partial derivative:

[0081]

[0082]

[0083] in, represents the partial derivative of the brightness value L, δ x L represents the partial derivative of the brightness value L in the x-axis direction, δ y L represents the partial derivative of the brightness value L in the y direction. The event is caused by the change of the brightness signal in the spatial gradient and changes with the optical flow estimation value. Move. The dot product indicates that the event is caused by the optical flow vector in Direction projection.

[0084] According to the brightness constancy constraint, optical flow estimation can achieve consistent estimation of brightness constancy with intensity estimation, solving the problem of image blur in reconstructed images. Therefore, the present invention applies optical flow estimation to the image reconstruction process to optimize the quality of reconstructed images by integrating the working principle of the event camera into the deep learning network to build a reconstruction network. On the one hand, a high-resolution intensity image is reconstructed from a continuous event stream through the reconstruction network. On the other hand, motion blur in the event is compensated through optical flow estimation. Finally, the reconstruction effect is optimized through the brightness constancy constraint, realizing the reconstruction process from a simple event stream to an intensity image. The schematic diagram of the principle of the event stream image reconstruction network model based on brightness constancy is shown in the figure. Figure 1 shown.

[0085] Combining formula (3-1), formula (3-2) and formula (3-3), we can know that:

[0086]

[0087] Therefore, based on the brightness constancy constraint, the estimated value of the brightness change can be Using optical flow estimation and brightness estimate express:

[0088]

[0089] Among them, the optical flow estimation value The calculation method is as follows:

[0090] Formula (3-1) is expanded into a first-order linear function using Taylor's formula, and partial derivatives are obtained for x, y, and t respectively:

[0091] L x u+L y v+Lt =0; (3-6)

[0092] Considering the continuity of motion, adjacent pixels will show similarity in motion speed, so the motion can be jointly estimated through adjacent N pixels, that is:

[0093]

[0094] This scheme is based on brightness constancy. By expanding the brightness function into a linear function through the first-order Taylor expansion, the brightness value can be constrained and estimated through the motion of the object, thus simplifying the brightness value solution process.

[0095] According to the working principle of event camera, the imaging unit of event camera is composed of independent pixel arrays, which can respond to brightness changes and transmit change information through sparse asynchronous events. When the brightness change value reaches the threshold C, the event is triggered.

[0096]

[0097] Among them, ΔL * is the brightness change reference value; i is the event index value; θ is the number of all events within the time interval Δt; p i is the polarity of the ith event, +1 indicates an increase in brightness, and -1 indicates a decrease in brightness; where C is the event camera threshold, is the contrast sensitivity, determined by the bias current of the pixel, and is usually a positive constant;

[0098] The brightness change estimate and brightness change reference value ΔL * For comparison, the error is back-propagated through the ReconNet network, such as Figure 1 As shown in Figure 2, the event stream image reconstruction network model based on brightness constancy is finally constructed. Figure 1 It can be seen that the event flow image reconstruction network model based on brightness constancy consists of two parts: the optical flow calculation module and the ReconNet network, which are used for optical flow estimation and image reconstruction respectively. The ReconNet network uses a convolutional neural network similar to the "U" network to recover the brightness estimation value from the event flow image. Optical flow estimation can describe the changes in optical flow during motion by estimating the motion optical flow field, which plays a role in removing motion blur and artifacts. Finally, the brightness constancy of the optical flow estimation value and the intensity estimation value is jointly estimated, and the error is propagated in reverse (the dotted line in the figure represents the back propagation of the error) to improve the reconstruction accuracy and achieve image reconstruction with high-definition resolution.

[0099] In step 3, in order to better describe the difference between the reconstructed image and the real image, the loss function is described by brightness reconstruction loss and temporal consistency loss. The constructed loss function is:

[0100]

[0101] Among them, Loss is the loss function; S is the number of steps of the cyclic network during training; k is the serial index; is the brightness reconstruction loss; λ1 is the balance coefficient between the temporal consistency loss and the brightness reconstruction loss; It is the loss of temporal consistency.

[0102] To facilitate the understanding of the formula of the constructed loss function, the derivation process of step three is as follows:

[0103] The network relies on recurrent connections to enhance the temporal consistency between consecutive reconstructions. When there is flickering in the scene, artifacts may exist at individual times, so a temporal consistency loss is defined. Temporal consistency loss It is defined as the reconstruction error between two consecutive frames, as shown in the formula:

[0104]

[0105] in, is the optical flow estimate The warping function, is the estimated value of the brightness change, and k is the index number.

[0106] Following the maximum likelihood method, the brightness reconstruction loss Defined as the norm of the square of the warped brightness increment:

[0107]

[0108] Therefore, the loss function is:

[0109]

[0110] In step 4, the method of verifying the quality of the reconstructed image using the mean squared error (MSE) is as follows:

[0111]

[0112] Where MSE is the mean square error; m is the number of image pixels, j is the index value of the reconstructed image; ω j >0 is the coefficient, y j is the real image pixel intensity value, is the reconstructed image pixel intensity value.

[0113] In step 4, the method of verifying the quality of the reconstructed image by using structural similarity (SSIM) is as follows:

[0114] SSIM(I r ,I g )=l(I r ,I g )+c(I r ,I g )+s(I r ,I g );

[0115] Among them, SSIM(I r ,I g ) is the structural similarity between the reconstructed image and the real image; l(I r ,I g ) is the brightness similarity between the reconstructed image and the real image; c(I r ,I g ) is the contrast similarity between the reconstructed image and the real image; s(I r ,I g ) is the structural similarity between the reconstructed image and the real image; SSIM (I r ,I g ) is larger, the better the quality of the reconstructed image is.

[0116] Verification results and analysis of the embodiments of the present invention:

[0117] The present invention uses 6 subsets of the event camera dataset Event Camera Dataset, namely the dynamic_6dof subset, boxes_6dof subset, poster_6dof subset, office_zigzag subset, slider_depth subset and shapes_6dof subset, to be sent to the event stream image reconstruction network model based on brightness constancy constructed by the present invention for training. The trained model uses the calibration subset in the Event Camera Dataset to verify the training effect, and is compared with the reconstruction effects of the popular regularization (Manifold Regularisation, MR) method and the E2VID method. The reconstruction effect is quantitatively analyzed by two image quality evaluation indicators, the mean square error MSE and the structural similarity SSIM. The reconstruction results can be seen Figures 4 to 6 ;from Figure 4 , Figure 5 and Figure 6Qualitative comparative analysis shows that the present invention can effectively remove the influence of artifacts without any prior information, making the image clearer, while increasing the contrast of the reconstructed image, making the edge contour information of the target clearer, and realizing image reconstruction with high-definition resolution. The MSE and SSIM comparisons of different image reconstruction methods are shown in Table 1:

[0118] Table 1

[0119]

[0120] It can be seen from Table 1 that in the test of the calibration subset in the Event Camera Dataset, the average mean square error MSE of the technical solution of the present invention is 12.55%, which is less than 18.35% of E2VID and 33.47% of MR; at the same time, the structural similarity SSIM is 91.76%, which is greater than 87.19% of E2VID and 20.64% of MR, which proves that the technical solution disclosed in the present invention can effectively reduce the root mean square error between the reconstructed image and the real image, increase the structural similarity between the reconstructed image and the real image, and effectively improve the quality of the reconstructed image.

Claims

1. An event camera image reconstruction method based on brightness constancy, characterized in that: include: Step 1: Get the Event Camera Dataset and divide it into a training set and a test set. Step 2: Obtain the brightness estimation value of the pixel through the ReconNet network; Under the constraint of brightness constancy, obtain the optical flow estimation value of the pixel through optical flow estimation, and jointly calculate the optical flow estimation value and the brightness estimation value to obtain the brightness change estimation value of the pixel; Compare the brightness change estimation value with the brightness change reference value of the pixel obtained by integrating the event data to obtain the error, and back-propagate the error through the ReconNet network to obtain the constructed event flow image reconstruction network model based on brightness constancy; Step 3: Construct a loss function; input the training set data into the event stream image reconstruction network model based on brightness constancy to perform network learning training; when the loss function converges, the network training is completed and a trained neural network model is obtained; Step 4: Input the test set data into the trained neural network model, and verify the quality of the reconstructed image using the mean square error index and / or the structural similarity index; Among them, in step 2, under the constraint of brightness constancy, the method of obtaining the optical flow estimation value of the pixel through optical flow estimation is: the brightness constrain is expanded into a first-order linear function through Taylor formula, and the partial derivatives of the pixel coordinates (x, y) and time t are obtained respectively: L x u+L y v+L t =0; The motion is jointly estimated by the pixels of the adjacent pixel number N, and the optical flow estimation value is obtained: Among them, L x , L y , L t are the inverse of the brightness with respect to the pixel coordinates (x, y) and time t, respectively; N is the number of adjacent pixels; u, v are motion vectors; is the optical flow estimate.

2. The method according to claim 1, characterized in that In step 1, the event camera dataset EventCamera Dataset is obtained, and the method of dividing it into a training set and a test set includes: The DAVIS240C event camera is used to record real scenes to obtain event data and ground truth grayscale full-frame images, and the Event Camera Dataset is obtained. The event camera dataset Even Camera Dataset includes dynamic_6dof subset, boxes_6dof subset, poster_6dof subset, office_zigzag subset, slider_depth subset, shapes_6dof subset and calibration subset. The calibration subset is used as the test set, and the remaining subsets are used as training sets.

3. The method according to claim 1, characterized in that In step 2, the brightness constancy constraint is: when the pixel of the object on the pixel plane moves from the pixel coordinate (x, y) to the target pixel coordinate (x+Δx, y+Δy), and the time interval Δt is within the specified threshold, it is considered that the brightness of the pixel has not changed, which is expressed as: L(x,y,t)=L(x+Δx,y+Δy,t+Δt); Wherein, L(x, y, t) is a nonlinear function of the brightness of the pixel, the pixel coordinates (x, y) and the time t; Δx and Δy are motion vectors within a specified threshold; and Δt is the time interval.

4. The method according to claim 1, characterized in that In step 2, the method of jointly calculating the optical flow estimation value and the brightness estimation value to obtain the brightness change estimation value of the pixel includes: in, is the estimated value of brightness change; is the optical flow estimate The warping function of is the derivative of the brightness estimate.

5. The method according to claim 1, characterized in that In step 2, the reference value of the brightness change of the pixel obtained by integrating the event data is: Where, ΔL * is the brightness change reference value; i is the event index value; θ is the number of all events within the time interval Δt; p i is the polarity of the ith event; C is the event camera threshold.

6. The method according to claim 1, characterized in that In step three, the constructed loss function is: Among them, Loss is the loss function; S is the number of steps of the cyclic network during training; k is the serial index; is the brightness reconstruction loss; λ1 is the balance coefficient between the temporal consistency loss and the brightness reconstruction loss; It is the loss of temporal consistency.

7. The method according to claim 1, characterized in that In step 4, the method of verifying the quality of the reconstructed image using the mean square error index is: Where MSE is the mean square error; m is the number of image pixels, j is the index value of the reconstructed image; ω j >0 is the coefficient, y j is the real image pixel intensity value, is the reconstructed image pixel intensity value.

8. The method according to claim 1, characterized in that In step 4, the method of verifying the quality of the reconstructed image using the structural similarity index is: SSIM(I r ,I g )=l(I r ,I g )+c(I r ,I g )+s(I r ,I g ); Among them, SSIM(I r ,I g ) is the structural similarity between the reconstructed image and the real image; l(I r ,I g ) is the brightness similarity between the reconstructed image and the real image; c(I r ,I g ) is the contrast similarity between the reconstructed image and the real image; s(I r ,I g ) is the structural similarity between the reconstructed image and the real image.

Images