A dense 3D reconstruction method based on event camera

Abstract

The present invention relates to a dense three-dimensional reconstruction method based on an event camera. The event camera is a dynamic visual sensor. The three-dimensional reconstruction steps are divided into three parts: intensity image reconstruction, motion structure recovery, and dense three-dimensional reconstruction. First, deep learning is used to reconstruct the intensity image from the event. Then, the motion recovery structure based on SFM is used to estimate the camera's intrinsic parameters, posture, and sparse point cloud. Finally, multi-stereo views based on MVS are used to complete the dense reconstruction. In special environments with high dynamic range, lack of illumination, and high-speed motion, traditional cameras will experience blurring, overexposure, and underexposure, which will greatly affect the quality of reconstruction. However, due to the advantages of event cameras, the quality of dense three-dimensional reconstruction completed by the present invention is much higher than that of traditional cameras.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision three-dimensional reconstruction, relates to a dense three-dimensional reconstruction method based on an event camera, and belongs to a three-dimensional reconstruction method of a dynamic vision sensor. Background Art

[0002] An event camera (dynamic vision sensor) is a novel sensor that differs from traditional cameras. Unlike traditional cameras, which capture a complete image, each pixel is asynchronously triggered by an event. An event here refers to a change in the brightness of the illumination falling on each pixel. If the brightness increases or decreases by more than a predetermined threshold, an event is generated. Compared to traditional cameras, event cameras offer the advantages of low latency, extremely low power consumption, high information availability, high dynamic range, and lack of motion blur. Because events are caused by the distinct motion of intensity edges, most 3D reconstructions consist solely of scene edges—i.e., semi-dense reconstructions—which are insufficient for some applications. In light of this, this paper proposes a dense 3D reconstruction method based on event cameras. This method first utilizes deep learning to reconstruct an intensity image from events; then, using structure from motion (SFM)-based structure from motion to estimate the camera's intrinsic parameters, pose, and sparse point cloud; finally, using multi-stereo views based on MVS to achieve dense reconstruction. In the challenging conditions of high dynamic range, poor illumination, and high-speed motion, traditional cameras can suffer from blurring, overexposure, and underexposure, which significantly impacts reconstruction quality. However, due to the advantages of event cameras, the proposed method achieves dense 3D reconstructions of significantly higher quality than traditional cameras. Summary of the Invention

[0003] Technical problems to be solved

[0004] To overcome the shortcomings of existing technologies, this paper proposes a dense 3D reconstruction method based on an event camera. This method leverages the low latency, extremely low power consumption, high information availability, high dynamic range, and lack of motion blur of event cameras to perform dense 3D reconstruction in low-light and fast-moving environments. Experiments were conducted on an industrial computer connected to an event camera.

[0005] Technical Solution

[0006] A dense 3D reconstruction method based on an event camera is characterized in that the event camera is a dynamic visual sensor, and the 3D reconstruction steps are divided into three parts: intensity image reconstruction, motion structure recovery, and dense 3D reconstruction. The specific process is as follows:

[0007] 1. Intensity image reconstruction:

[0008] Step 1.1: Establish a UNet-based intensity image reconstruction neural network, including a head layer H, a main layer B, and a prediction layer P; the main layer B includes three recurrent convolution modules R1, R2, and R3, five grouped convolution modules G1, G2, G3, G4, and G5 with consistent parameters, and three sub-pixel convolution modules U1, U2, and U3;

[0009] Step 1.2: Encode the event to obtain a fixed-size channel tensor for the neural network input. This is then fed into the neural network. After passing through the convolutional layer and RELU activation function in the head layer H, a tensor with an output channel size of 32 is obtained. The convolution kernel size is 3×3.

[0010] Step 1.3: Send the output of the head layer H to three circular convolution modules R1, R2, and R3 for downsampling. After the input passes through each circular convolution block, the number of output channels is doubled, and the height and width of the tensor of each convolution block channel are reduced by two times;

[0011] Each of the cyclic convolution blocks includes a CBR and a ConvLSTM module. After using ConvLSTM, the previous state information is retained, and the state information is used to update the current state in combination with the current input;

[0012] The CBR is a convolution layer with a convolution kernel size of 5×5 + a regularization (BatchNorm) layer + a ReLU activation function;

[0013] The convolution kernel size of the ConvLSTM module is 3×3;

[0014] Step 1.4: The output data of the three circular convolution blocks are input into two group convolution blocks G1 and G2 to group the different feature maps of the input layer and then perform convolution on each group.

[0015] The grouped convolution module first feeds the input channel tensor with N channels into a convolution layer with a convolution kernel size of 1×1 to halve the number of channels. Then, it passes through a grouped convolution layer with 4 tensors per group and a convolution kernel size of 3×3 to obtain an output with N / 2 channels. Finally, this output is fed into a convolution layer with a convolution kernel size of 1×1 to double the number of channels and obtain an output with N channels.

[0016] Step 1.5: Input the outputs of the three recurrent convolution modules into one branch respectively. Each branch consists of a grouped convolution block G3, G4 and G5.

[0017] The grouped convolution module first feeds the input channel tensor with N channels into a convolution layer with a convolution kernel size of 1×1, halving the number of channels. It then passes through a grouped convolution layer with 4 tensors per group and a convolution kernel size of 3×3, obtaining an output with N / 2 channels. Finally, it passes through a convolution layer with a convolution kernel size of 1×1, doubling the number of channels, resulting in an output with N channels.

[0018] Step 1.6: The output with N channels is upsampled by three consecutive sub-pixel convolution modules U1, U2, and U3;

[0019] Each sub-pixel convolution module first performs a sub-pixel splicing operation on the input. That is, the input size during upsampling is a×b×c, and the output after the splicing operation is 2a×2b×c / 4. Then, it passes through a convolution layer with a convolution kernel size of 3×3 to obtain the output of the entire main layer B.

[0020] Among them: after each sub-pixel convolution block, the number of output channels is reduced by two times, and the height and width of the tensor of each convolution block channel are expanded by two times; the input of each sub-pixel convolution module is the output of G3, G4, and G5 of the corresponding circular convolution module after processing by the grouped convolution branch and the output of the previous sub-pixel convolution module;

[0021] Step 1.7: The output of the first recurrent convolution module R1 is fed into a sub-pixel convolution module U0 for upsampling. The upsampling result and the output of the entire main layer B are then fed into the prediction layer. The prediction layer then passes the input through a convolutional layer for convolution, a regularization layer, and finally a sigmoid activation function to obtain the output, which is a predicted value between 0 and 1 for each pixel.

[0022] 2. Restoration of motion structure:

[0023] Step 2.1: Take the intensity image reconstructed by the neural network as the input image set ψ={I i |i=1,2...N I}, use SIFT feature extraction algorithm to detect the features and descriptors of each input image, and mark the feature coordinates as x, the feature descriptor as f, and record all detected features and descriptors as a set

[0024] Step 2.2: Utilize F i As the appearance description of the image, SIFT feature matching is performed by b Each feature in image I is found a The feature descriptor with the highest matching degree searches for the corresponding relationship of features and finds images of the same scene;

[0025] After feature matching, the output is a set of potential matching successful image pairs C = {{I a ,I b}|I a ,I b ∈ψ,a<b} and their related features correspond to M ab ∈F a ×F;

[0026] Step 2.3: Use the epipolar geometric relationship of the potentially matched image pairs to calculate the fundamental matrix F of each image pair, calculate the camera's intrinsic parameters and pose, and if a valid camera's intrinsic parameters and pose map features between more than 30 images, it is considered to have passed the geometric verification. Then, use the RANSAC method to filter outliers on the image pairs that have passed the geometric verification. The final output is the geometrically verified image pair. Corresponding to their relevant characteristics

[0027] Step 2.4: Select the image pairs with the most matching features in the geometrically verified image pairs with a baseline distance greater than 10 cm, perform triangulation and nonlinear optimization, and estimate the camera transformation matrix and the coordinates of the 3D points in space;

[0028] Step 2.5: Add the remaining image pairs as input to the nonlinear optimization in descending order of feature matching quality, and repeat the process of step 2.4 until all image pairs are optimized. The final output is the camera transformation matrix set P = {P c ∈SE(3)|c=1,2...N p} and the 3D coordinate point set X={X k ∈R 3 |k=1,2...N X}, that is, sparse reconstruction is completed;

[0029] 3. Dense 3D Reconstruction

[0030] Step 3.1: Take the transformation matrix output by motion recovery structure, the 3D point coordinates in space, and the corresponding image as input, and use the Patch-Match Stereo method to reconstruct the dense point cloud, and output the dense point cloud and its corresponding image;

[0031] Step 3.2: Take the dense point cloud and its corresponding image as input and use the surface reconstruction method based on binary segmentation to perform rough surface reconstruction on the dense point cloud. The dense point cloud is divided into two categories: internal and external. The surface between the internal and external is the surface of the object. The output is a rough surface reconstruction.

[0032] Step 3.3: Take the rough surface reconstruction as input, optimize the mesh surface details based on photometric consistency, calculate photometric consistency again for the multi-view images, and output a fine surface reconstruction;

[0033] Step 3.4: Perform texture mapping on the fine surface reconstruction and output the final dense 3D reconstruction result.

[0034] The number of groups in the group convolution module in steps 1.4 and 1.5 is η=N / 8.

[0035] The nonlinear optimization formula: Among them, P c is the camera's transformation matrix, X k is the coordinate of the 3D point in space, x j is the feature coordinate in the image, π is the projection function that projects the 3D point in space to the camera plane, ρ j is the weight, which is determined by the number of image-to-feature matches.

[0036] The intensity image reconstructed by the event camera is a grayscale image, and the final dense 3D reconstruction result is gray.

[0037] Beneficial effects

[0038] The present invention proposes a dense 3D reconstruction method based on an event camera. The event camera is a dynamic visual sensor, and the 3D reconstruction steps are divided into three parts: intensity image reconstruction, motion structure recovery, and dense 3D reconstruction. First, deep learning is used to reconstruct the intensity image from the event; then the motion recovery structure based on SFM is used to estimate the camera's intrinsic parameters, posture, and sparse point cloud, and finally, the dense reconstruction is completed using multi-stereo views based on MVS. In special environments with high dynamic range, lack of light, and high-speed motion, traditional cameras will experience blurring, overexposure, and underexposure, which will greatly affect the quality of reconstruction. However, due to the advantages of event cameras, the quality of dense 3D reconstruction completed by the present invention is much higher than that of traditional cameras. The present invention utilizes the characteristics of low latency, extremely low power consumption, high information availability, high dynamic range, and no motion blur of event cameras to perform dense 3D reconstruction in low-light and fast-moving environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Flowchart of dense 3D reconstruction method based on event camera

[0040] Figure 2 Reconstructing the network for intensity images

[0041] Figure 3 Flowchart of structure from motion and dense 3D reconstruction

[0042] Figure 4The original image of the event camera

[0043] Figure 5 The reconstruction result of intensity image

[0044] Figure 6 Dense 3D reconstruction results DETAILED DESCRIPTION

[0045] The present invention will now be further described with reference to the embodiments and accompanying drawings:

[0046] The present invention is mainly divided into three parts: intensity image reconstruction, motion structure recovery, and dense 3D reconstruction.

[0047] Part I: Intensity Image Reconstruction

[0048] Step 1: Establish an intensity image reconstruction neural network based on UNet, which is divided into a head layer H, a main layer B, and a prediction layer P. The main layer is the core of the entire network, completing feature extraction and fusion. The main layer includes three recurrent convolution modules R1, R2, and R3, five grouped convolution modules G1, G2, G3, G4, and G5, and three sub-pixel convolution modules U1, U2, and U3.

[0049] Step 2: Encode the event to obtain a fixed-size channel tensor for the neural network input. This is then fed into the neural network, passing through the convolutional layer and RELU activation function in the head layer H. The convolution kernel size is 3×3, and the output channel size is 32.

[0050] Step 3: The output of the head layer H is fed into three circular convolutional modules R1, R2, and R3 for downsampling. Each circular convolutional block consists of a CBR (convolutional layer + regularization (BatchNorm) layer + ReLU activation function) with a 5×5 kernel and a ConvLSTM module (with a 3×3 kernel). The purpose of using ConvLSTM is to retain previous state information, which is used to update the current state in conjunction with the current input. After the input passes through each circular convolutional block, the number of output channels is doubled, and the height and width of the channel tensor of each convolutional block are reduced by two times.

[0051] Step 4: The output data of the three circular convolution blocks are input into two grouped convolution blocks, namely G1 and G2. Grouped convolution is to group the different feature maps of the input layer, and then use different convolution kernels to convolve each group. The grouped convolution module used in the present invention first sends the channel tensor with the input channel number N into a convolution layer with a convolution kernel size of 1×1, which halves the number of channels. Then, it passes through a grouped convolution layer with 4 tensors per group, where the convolution kernel size is 3×3 and the number of groups η=N / 8, and obtains an output with N / 2 channels. Finally, this output is sent to a convolution layer with a convolution kernel size of 1×1, which doubles the number of channels and obtains an output with N channels. After the circular convolution module performs the downsampling operation, it passes through two grouped convolution modules in order to fully extract the most abstract features.

[0052] Step 5: In addition, the inventors also input the outputs of the three circular convolution modules into a branch respectively, each branch consisting of a grouped convolution block, namely G3, G4, and G5. The grouped convolution module used in the present invention first feeds the channel tensor with N input channels into a convolution layer with a convolution kernel size of 1×1, halving the number of channels. Then, it passes through a grouped convolution layer with 4 tensors per group, where the convolution kernel size is 3×3 and the number of groups η=N / 8, obtaining an output with N / 2 channels. Finally, this output is fed into a convolution layer with a convolution kernel size of 1×1, doubling the number of channels, obtaining an output with N channels. After the circular convolution module performs the downsampling operation, it passes through two grouped convolution modules in order to fully extract the most abstract features.

[0053] Step 6: Next, three consecutive sub-pixel convolution modules U1, U2, and U3 perform upsampling. Each sub-pixel convolution module first performs sub-pixel concatenation on the input. That is, the input size during upsampling is a×b×c, and the output after concatenation is 2a×2b×c / 4. This is then passed through a convolution layer with a kernel size of 3×3. After each sub-pixel convolution block, the number of output channels is reduced by a factor of two, and the height and width of the channel tensor of each convolution block are doubled. The input of each sub-pixel convolution module is the output of the corresponding recurrent convolution module G3, G4, and G5 after processing by the grouped convolution branch, as well as the output of the previous sub-pixel convolution module. Traditional upsampling is achieved through non-learnable methods such as linear interpolation, so we use sub-pixel convolution instead of the original interpolation. This results in the output of the entire main layer B.

[0054] Step 7: The output of the first recurrent convolution module R1 is fed into a sub-pixel convolution module U0 for upsampling. The upsampling result and the output of the entire main layer B are then fed into the prediction layer. The prediction layer first passes the input through a convolution layer for convolution, then through a regularization layer, and finally through a Sigmoid activation function to obtain the output, which is a predicted value between 0 and 1 for each pixel.

[0055] Part II: Structure from Motion Recovery

[0056] Step 1: Take the intensity image reconstructed by the neural network as the input image set ψ={I i |i=1,2...N I For each input image, use the SIFT feature extraction algorithm to detect its features and descriptors, and mark the feature coordinates as x, the feature descriptors as f, and record all detected features and descriptors as a set

[0057] Step 2: Use F i As the appearance description of the image, SIFT feature matching is performed by b Each feature in image I is found a The feature descriptor with the highest matching degree searches for the corresponding relationship of features and finds images that see the same scene. After feature matching, the output is a set of potential matching successful image pairs C = {{I a ,I b}|I a ,I b ∈ψ,a<b} and their related features correspond to M ab ∈F a ×F.

[0058] Step 3: Use the epipolar geometry of the potentially matched image pairs to calculate the fundamental matrix F of each image pair, calculate the camera's intrinsic parameters and pose, if a valid camera intrinsic parameter and pose map more than 30 features between images, they will be considered to have passed the geometric verification, and then use the RANSAC method to filter outliers on the image pairs that have passed the geometric verification, and the final output is the geometrically verified image pair. Corresponding to their relevant characteristics

[0059] Step 4: Select the image pairs with the most matching features in the geometrically verified image pairs with a baseline distance greater than 10 cm for triangulation and nonlinear optimization (Bundle Adjustment) to estimate the camera transformation matrix and the coordinates of the 3D points in space. The nonlinear optimization formula is as follows:

[0060]

[0061] Among them, P c is the camera's transformation matrix, X k is the coordinate of the 3D point in space, x j is the feature coordinate in the image, π is the projection function that projects the 3D point in space to the camera plane, ρ j is the weight, which is determined by the number of image-to-feature matches.

[0062] Step 5: Add the remaining image pairs as input to the nonlinear optimization in descending order of feature matching quality. Repeat the process of step 4 until all image pairs are optimized. The final output is the camera transformation matrix set P = {P c ∈SE(3)|c=1,2...N p} and the 3D coordinate point set X={X k ∈R 3 |k=1,2...N X}, the sparse reconstruction is completed.

[0063] Part III: Dense 3D Reconstruction

[0064] Step 1: Using the transformation matrix output by Structure from Motion, the 3D point coordinates in space, and the corresponding image as input, we use the Patch-Match Stereo method to reconstruct a dense point cloud, obtaining a complete and accurate dense point cloud at a reasonable speed. The output of this step is the dense point cloud and its corresponding image.

[0065] Step 2: Take the dense point cloud from the first step and its corresponding image as input, and use the surface reconstruction method based on binary segmentation to perform rough surface reconstruction on the point cloud. The dense point cloud is divided into two categories: internal and external. The surface between the internal and external is the surface of the object, and the output is a relatively rough surface reconstruction.

[0066] Step 3: Take the rough surface reconstruction from the second step as input, optimize the mesh surface details based on photometric consistency, calculate the photometric consistency of the multi-view images again to achieve the best photometric consistency, and output a fine surface reconstruction.

[0067] Step 4: Texture mapping is performed on the fine surface reconstruction generated in the third step, and the final dense 3D reconstruction result is output. Since the intensity image reconstructed by the event camera is a grayscale image, the final dense 3D reconstruction result is also gray. Specific embodiment:

[0069] like Figure 1 As shown in Figure 2, the dense 3D reconstruction method based on the event camera is as follows:

[0070] Step 1: Establish an intensity image reconstruction neural network based on UNet, whose network structure is as follows Figure 2 As shown in the figure, it is divided into the head layer H, the main layer B and the prediction layer P. The main layer is the central part of the entire network, which completes the feature extraction and fusion. The main layer includes three circular convolution modules R1, R2, R3, five grouped convolution modules G1, G2, G3, G4, G5, and three sub-pixel convolution modules U1, U2, U3.

[0071] Step 2: In order to enable the convolutional recurrent neural network to process the event stream in the text, the event stream is encoded into a three-dimensional tensor with a channel size of 5.

[0072] Step 3: Use the Ecoco dataset for training. The image size is 240×180. During training, the dataset is randomly flipped (-20°, 20°) and cropped to 128×128 to expand the dataset. The loss function formula used is as follows:

[0073] L=SSIM(X1,X2)+d(X1,X2)

[0074] Where X1 and X2 represent two images, SSIM(X1,X2)=L(X1,X2)*C(X1,X2)*S(X1,X2) is the structural similarity index, which measures the similarity between the two images. represents brightness similarity, represents contrast similarity, represents the structure score. and represents the mean of the two images, and represents the standard deviation of the two images, Represents the covariance of the two images, C1 C2 C3 are constants so that the denominator is not zero.

[0075] To learn perceptual similarity, we pass two images through the VGG19 network and calculate the difference in the output value of each layer, where l is the number of neural network layers and H is the number of layers. l W l The height and width of the current layer image, w l is the scaling factor, and Y is the corresponding output of each layer.

[0076] The deep learning framework used is PyTorch, with an epoch of 200 and a batch size of 4. The ADAM optimizer is used with a maximum learning rate of 5×10-4, and dynamic learning rate adjustment is performed for training, and the trained model parameters are saved.

[0077] Step 4: Use the event camera to collect information about a location and record it into a robag. The original image of the event camera is as follows: Figure 4 As shown. Subscribe to the / dvs / events topic in the rosbag recorded by the event camera. Its message type is dvs_msgs, and the content is pixel coordinates, polarity, and timestamp. Read these three contents and save them in a text file in the order of (timestamp, pixel coordinates, polarity) for subsequent processing.

[0078] Step 5: Repeat the encoding process in step 2, input the encoded 5-channel tensor into the trained intensity image reconstruction network, and output a 3-channel tensor after passing through the head layer, main layer, and prediction layer. The tensor is converted into a picture form, which is the result of intensity image reconstruction, as shown in Figure 5 shown.

[0079] Step 6: Next, perform motion structure recovery and dense 3D reconstruction. The specific process is as follows Figure 3 As shown, the reconstructed intensity image is subjected to feature extraction and matching using SIFT, and a set of potential matching successful image pairs C = {{I a ,I b}|I a ,I b ∈ψ,a<b} and their related features correspond to M ab ∈F a ×F.

[0080] Step 7: Use the epipolar geometry of the matched image pairs to calculate the fundamental matrix F of each image pair, calculate the intrinsic parameters and attitude of the camera, and then use the RANSAC method to filter outliers. The final output is the geometrically verified image pair. Corresponding to their relevant characteristics

[0081] Step 8: Select image pairs with sufficient matching features and large enough baseline distance from the geometrically verified image pairs for triangulation and nonlinear optimization (Bundle Adjustment) to estimate the camera transformation matrix and the coordinates of the 3D points in space.

[0082] Step 9: Add the remaining image pairs as input to the nonlinear optimization in the order of feature matching quality from good to bad, and repeat the process of step 7 until all image pairs are optimized. The final output is the camera transformation matrix set P = {P c ∈SE(3)|c=1,2...N p} and the 3D coordinate point set X={X k ∈R 3 |k=1,2...N X}, the sparse reconstruction is completed.

[0083] Step 10: Take the transformation matrix output by motion recovery structure, the 3D point coordinates in space, and the corresponding image as input, use the Patch-Match Stereo method to reconstruct the dense point cloud, and output the dense point cloud and its corresponding image.

[0084] Step 11: Take the dense point cloud from step 10 and its corresponding image as input, and use the surface reconstruction method based on binary segmentation to perform rough surface reconstruction on the point cloud, and output a relatively rough surface reconstruction.

[0085] Step 12: Take the rough surface reconstruction from step 11 as input, optimize the mesh surface details based on photometric consistency, and output a fine surface reconstruction.

[0086] Step 13: Texture mapping is performed on the fine surface reconstruction generated in step 12, and the final dense 3D reconstruction result is output, such as Figure 6 shown.

Claims

1. A dense 3D reconstruction method based on an event camera, characterized by: The event camera is a dynamic visual sensor. The 3D reconstruction process is divided into three parts: intensity image reconstruction, motion structure recovery, and dense 3D reconstruction. The specific process is as follows:

1. Intensity image reconstruction: Step 1.1: Establish a UNet-based intensity image reconstruction neural network, including a head layer H, a main layer B, and a prediction layer P; the main layer B includes three recurrent convolution modules R1, R2, and R3, five grouped convolution modules G1, G2, G3, G4, and G5 with consistent parameters, and three sub-pixel convolution modules U1, U2, and U3; Step 1.2: Encode the event to obtain a fixed-size channel tensor for the neural network input. This is then fed into the neural network. After passing through the convolutional layer and RELU activation function in the head layer H, a tensor with an output channel size of 32 is obtained. The convolution kernel size is 3×3. Step 1.3: Send the output of the head layer H to three circular convolution modules R1, R2, and R3 for downsampling. After the input passes through each circular convolution block, the number of output channels is doubled, and the height and width of the tensor of each convolution block channel are reduced by two times; Each of the cyclic convolution blocks includes a CBR and a ConvLSTM module. After using ConvLSTM, the previous state information is retained, and the state information is used to update the current state in combination with the current input; The CBR is a convolution layer with a convolution kernel size of 5×5 + a regularization (BatchNorm) layer + a ReLU activation function; The convolution kernel size of the ConvLSTM module is 3×3; Step 1.4: The output data of the three circular convolution blocks are input into two group convolution blocks G1 and G2 to group the different feature maps of the input layer and then perform convolution on each group. The grouped convolution module first feeds the input channel tensor with N channels into a convolution layer with a convolution kernel size of 1×1 to halve the number of channels. Then, it passes through a grouped convolution layer with 4 tensors per group and a convolution kernel size of 3×3 to obtain an output with N / 2 channels. Finally, this output is fed into a convolution layer with a convolution kernel size of 1×1 to double the number of channels and obtain an output with N channels. Step 1.5: Input the outputs of the three recurrent convolution modules into one branch respectively. Each branch consists of a grouped convolution block G3, G4 and G5. The grouped convolution module first feeds the input channel tensor with N channels into a convolution layer with a convolution kernel size of 1×1, halving the number of channels. It then passes through a grouped convolution layer with 4 tensors per group and a convolution kernel size of 3×3, obtaining an output with N / 2 channels. Finally, it passes through a convolution layer with a convolution kernel size of 1×1, doubling the number of channels, resulting in an output with N channels. Step 1.6: The output with N channels is upsampled by three consecutive sub-pixel convolution modules U1, U2, and U3; Each sub-pixel convolution module first performs a sub-pixel splicing operation on the input. That is, the input size during upsampling is a×b×c, and the output after the splicing operation is 2a×2b×c / 4. Then, it passes through a convolution layer with a convolution kernel size of 3×3 to obtain the output of the entire main layer B. Among them: after each sub-pixel convolution block, the number of output channels is reduced by two times, and the height and width of the tensor of each convolution block channel are expanded by two times; the input of each sub-pixel convolution module is the output of G3, G4, and G5 of the corresponding circular convolution module after processing by the grouped convolution branch and the output of the previous sub-pixel convolution module; Step 1.7: The output of the first recurrent convolution module R1 is fed into a sub-pixel convolution module U0 for upsampling. The upsampling result and the output of the entire main layer B are then fed into the prediction layer. The prediction layer then passes the input through a convolutional layer for convolution, a regularization layer, and finally a sigmoid activation function to obtain the output, which is a predicted value between 0 and 1 for each pixel.

2. Restoration of motion structure: Step 2.1: Take the intensity image reconstructed by the neural network as the input image set ψ={I i |i=1,2...N I }, use SIFT feature extraction algorithm to detect the features and descriptors of each input image, and mark the feature coordinates as x, the feature descriptor as f, and record all the detected features and descriptors as a set F i ={(x j ,f j )|j=1,2...N Fi }; Step 2.2: Utilize F i As the appearance description of the image, SIFT feature matching is performed by b Each feature in image I is found a The feature descriptor with the highest matching degree searches for the corresponding relationship of features and finds images of the same scene; After feature matching, the output is a set of potential matching successful image pairs C = {{I a ,I b }|I a ,I b ∈ψ,a<b} and their related features correspond to M ab ∈F a ×F; Step 2.3: Use the epipolar geometric relationship of the potentially matched image pairs to calculate the fundamental matrix F of each image pair, calculate the camera's intrinsic parameters and pose, and if a valid camera's intrinsic parameters and pose map features between more than 30 images, it is considered to have passed the geometric verification. Then, use the RANSAC method to filter outliers on the image pairs that have passed the geometric verification. The final output is the geometrically verified image pair. Corresponding to their relevant characteristics Step 2.4: Select the image pairs with the most matching features in the geometrically verified image pairs with a baseline distance greater than 10 cm, perform triangulation and nonlinear optimization, and estimate the camera transformation matrix and the coordinates of the 3D points in space; Step 2.5: Add the remaining image pairs as input to the nonlinear optimization in descending order of feature matching quality, and repeat the process of step 2.4 until all image pairs are optimized. The final output is the camera transformation matrix set P = {P c ∈SE(3)|c=1,2...N p } and the 3D coordinate point set X={X k ∈R 3 |k=1,2...N X }, that is, sparse reconstruction is completed; 3. Dense 3D Reconstruction Step 3.1: Take the transformation matrix output by motion recovery structure, the 3D point coordinates in space, and the corresponding image as input, and use the Patch-Match Stereo method to reconstruct the dense point cloud, and output the dense point cloud and its corresponding image; Step 3.2: Take the dense point cloud and its corresponding image as input and use the surface reconstruction method based on binary segmentation to perform rough surface reconstruction on the dense point cloud. The dense point cloud is divided into two categories: internal and external. The surface between the internal and external is the surface of the object. The output is a rough surface reconstruction. Step 3.3: Take the rough surface reconstruction as input, optimize the mesh surface details based on photometric consistency, calculate photometric consistency again for the multi-view images, and output a fine surface reconstruction; Step 3.4: Perform texture mapping on the fine surface reconstruction and output the final dense 3D reconstruction result.

2. The method for dense 3D reconstruction based on an event camera according to claim 1, characterized in that: The number of groups in the group convolution module in steps 1.4 and 1.5 is η=N / 8.

3. The method for dense 3D reconstruction based on an event camera according to claim 1, characterized in that: The nonlinear optimization formula: Among them, P c is the camera's transformation matrix, X k is the coordinate of the 3D point in space, x j is the feature coordinate in the image, π is the projection function that projects the 3D point in space to the camera plane, ρ j is the weight, which is determined by the number of image-to-feature matches.

4. The method for dense 3D reconstruction based on an event camera according to claim 1, wherein: The intensity image reconstructed by the event camera is a grayscale image, and the final dense 3D reconstruction result is gray.

Images