Single-photon three-dimensional reconstruction method based on multi-stage sparse neural network

Abstract

The invention belongs to the field of electrical digital processing, and particularly relates to a single-photon three-dimensional reconstruction method based on a multi-stage sparse neural network. The method comprises the following steps: 1, making a single photon data set comprising a training set, a verification set and a test set; 2, establishing a multi-stage sparse neural network; 3, inputting the training set into a multi-stage sparse neural network, optimizing network parameters of the multi-stage sparse neural network by utilizing forward and reverse propagation, and constraining the multi-stage sparse neural network by adopting a mixed loss function to obtain the trained and optimized multi-stage sparse neural network; 4, inputting the verification set into the trained and optimized multi-stage sparse neural network, and verifying whether the network performance of the multi-stage sparse neural network reaches the expected performance or not; and 5, inputting the test set into the trained and optimized multi-stage sparse neural network for single-photon three-dimensional reconstruction, and generating a reconstructed depth image. The method is high in feasibility, high in universality, high in reconstruction speed and stable in effect under the condition of low photon counting.

Description

Technical Field

[0001] The invention relates to a single-photon three-dimensional reconstruction method, and in particular to a single-photon three-dimensional reconstruction method based on a multi-stage sparse neural network. Background Art

[0002] As a long-distance measurement technology, LiDAR uses laser beams to measure the distance, shape and surface features of target objects. It usually consists of a laser emitter, a detector, a scanning mirror and a computing device. LiDAR is widely used in autonomous vehicles, environmental modeling, robot navigation, mapping and exploration. The working principle of LiDAR is to emit a short pulse laser beam and then measure the time delay of the laser beam being reflected by the target object. The distance between the target object and the LiDAR can be calculated by multiplying this delay and the speed of light. By constantly changing the direction and angle of the laser beam, the three-dimensional shape and contour of the target object can be obtained. In recent years, the application of traditional LiDAR has been limited due to its low detection sensitivity at long distances, low signal-to-noise ratio and weak echo signals. However, with the continuous progress of single-photon detectors, precision electronic timing technology and quantum science, single-photon radar technology has gradually matured, providing an effective solution for the three-dimensional reconstruction of long-distance targets.

[0003] At present, people have overcome the above problems from three ways: single-photon lidar imaging system equipment, classical algorithms, and machine learning. People have improved the imaging system by using high-speed electronic gating, but this method requires the approximate position of the target to be known in advance, otherwise the target will be lost, which limits its practical application. Shin et al. removed noise by adding a total variation regularization term and adopted a pixel-by-pixel isochronous scanning mode to achieve an average efficiency of one photon per pixel for three-dimensional reconstruction of the target; further, Rapp et al. used the uniformity of the arrival time of noise photons and the aggregation of the arrival time of signal photons to separate signal photons from noise photons by adaptively setting local gating pixel by pixel, and filled the vacant pixels with the mean value of the neighboring pixels. Both methods have the effect of improving the reconstruction accuracy, but as the light flux gradually decreases, the imaging performance of the two will seriously decline. In 2018, Lindell et al. proposed a single-photon lidar three-dimensional reconstruction algorithm based on machine learning, which achieved good imaging results by using noise-free intensity information for three-dimensional reconstruction. However, in practical applications, the feasibility of this method is limited because good intensity information is usually difficult to obtain. Nevertheless, this study successfully introduced machine learning into the field of single-photon imaging and provides useful insights for future research.

[0004] Therefore, it is currently necessary to explore a non-fusion algorithm that has excellent generalization performance in different environments, especially excellent imaging and rapid response under low photon counts and low signal-to-noise ratio conditions, so as to enhance the practical application capabilities of single-photon lidar in three-dimensional reconstruction. Summary of the invention

[0005] The purpose of the present invention is to solve the technical problems of poor imaging and slow response of existing single-photon 3D reconstruction methods under low photon counts and low signal-to-noise ratio conditions, and to provide a single-photon 3D reconstruction method based on a multi-stage sparse neural network.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is:

[0007] A single-photon 3D reconstruction method based on a multi-stage sparse neural network is special in that it includes the following steps:

[0008] Step 1: Create a single photon dataset;

[0009] The single-photon dataset includes training set, validation set and test set;

[0010] Step 2: Build a multi-stage sparse neural network for single-photon 3D reconstruction;

[0011] Step 3: Input the training set into the multi-stage sparse neural network, optimize the network parameters by using the forward and backward propagation of the multi-stage sparse neural network, and use the mixed loss function to constrain the multi-stage sparse neural network to obtain the multi-stage sparse neural network after training and optimization;

[0012] Step 4: input the verification set into the multi-stage sparse neural network after training optimization to verify whether the network performance of the multi-stage sparse neural network reaches the expected performance. If so, execute step 5; if not, return to step 3 until the expected performance is reached.

[0013] Step 5: Input the test set into the trained and optimized multi-stage sparse neural network for single-photon 3D reconstruction to generate a reconstructed depth image.

[0014] Further, in step 2, the multi-stage sparse neural network includes a sparse coding module, a noise removal module, a signal expansion module, a depth estimation module, a sparse decoding module, a joint selection module and a noise prior module;

[0015] The input end of the sparse coding module, the first input end of the joint selection module and the input end of the noise prior module are used as the input end of the multi-stage sparsity neural network to receive the actual photon counting cube Z in the training set, the validation set and the test set;

[0016] The sparse coding module is used to compress the actual photon counting cube Z into a photon counting matrix D through sparse coding, and input the photon counting matrix D into the noise removal module;

[0017] The noise removal module is used to obtain the bins composed of active point coordinates and potential features according to the photon counting matrix D. And the warehouse Input into the signal expansion module and the noise prior module respectively;

[0018] The signal expansion module is based on the warehouse Get the expanded photon count matrix and select Expand Photon Count Matrix Get the target area from the coordinates of all active points in Then the target area Input to the depth estimation module and joint selection module respectively;

[0019] The joint selection module is used to select the photon counting cube Z in the target area. The expanded photon counting matrix

[0020] The depth estimation module is used to stitch the target area With the expanded photon counting matrix After obtaining the stitched photon counting cube Perform depth estimation and obtain estimated photon count cube Then the estimated photon counting cube Input to the sparse decoding module for sparse decoding to obtain the restored photon counting cube That is, the reconstructed depth image;

[0021] The noise prior module is used to convert the bin The three-dimensional noise cube is obtained by residual connection with the actual photon counting cube Z According to the three-dimensional noise cube Get a 2D noise map

[0022] Furthermore, the calculation formula of the noise removal module is:

[0023]

[0024] Where: W(m) is the mth learnable kernel basis; F(i,j,m) and G(i,j) are the mth kernel fusion coefficient and the fusion kernel weight at position (i,j) in the photon counting matrix D, respectively; D e [i,j] is the photon counting matrix D after convolution transformation; V is the number of kernel bases.

[0025] Furthermore, the signal expansion module calculates the expanded photon count matrix The formula is:

[0026]

[0027] Where: * represents the convolution operation; K s is the temporal convolution kernel; K t is the spatial convolution kernel.

[0028] Furthermore, the calculation formula of the joint selection module is:

[0029]

[0030] Where: coord is the coordinate; & is the joint selection.

[0031] Further, the depth estimation module includes a downsampling layer, a first asymmetric feature fusion unit AFF1, a second asymmetric feature fusion unit AFF2 and an upsampling layer;

[0032] The downsampling layer includes a first sparse convolution S1, a second sparse convolution S2, a first sub-manifold convolution L1, a second sub-manifold convolution L2, a third sub-manifold convolution L3, and a fourth sub-manifold convolution L4; the upsampling layer includes a first sparse deconvolution F1, a second sparse deconvolution F2, a fifth sub-manifold convolution L5, a sixth sub-manifold convolution L6, a seventh sub-manifold convolution L7, and an eighth sub-manifold convolution L8;

[0033] The first sub-manifold convolution L1 and the second sub-manifold convolution L2 are connected in sequence, the input end of the first sub-manifold convolution L1 is the input end of the depth estimation module, and the output end of the second sub-manifold convolution L2 is respectively connected to the first input end of the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2, and the input end of the first sparse convolution S1; the output end of the first sparse convolution S1 is connected to the input end of the third sub-manifold convolution L3, and the output end of the third sub-manifold convolution L3 is respectively connected to the second input end of the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2, and the input end of the second sparse convolution S2; the output end of the second sparse convolution S2 is connected to the fourth sub-manifold convolution L4 The input end of the fourth sub-manifold convolution L4 is connected to the third input end of the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2, and the input end of the fifth sub-manifold convolution L5; the fifth sub-manifold convolution L5, the first sparse deconvolution F1, the sixth sub-manifold convolution L6, the second sparse deconvolution F2, the seventh sub-manifold convolution L7 and the eighth sub-manifold convolution L8 are connected in series in sequence; the second input end of the sixth sub-manifold convolution L6 is connected to the output end of the second asymmetric feature fusion unit AFF2, the second input end of the seventh sub-manifold convolution L7 is connected to the output end of the first asymmetric feature fusion unit AFF1, and the output end of the eighth sub-manifold convolution L8 is the output end of the depth estimation module.

[0034] Further, the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2 each include three adjusters, and a cascade C, a 1×1 convolutional layer, and a 3×3 convolutional layer connected in series in sequence;

[0035] The input ends of the three adjusters are respectively used as the first input end, the second input end and the third input end of the first asymmetric feature fusion unit AFF1 or the second asymmetric feature fusion unit AFF2; the output ends of the three adjusters are respectively connected to the three input ends of the first cascade C;

[0036] The output end of the 3×3 convolutional layer is used as the output end of the first asymmetric feature fusion unit AFF1 or the second asymmetric feature fusion unit AFF2.

[0037] Furthermore, the noise prior module uses the following formula based on the three-dimensional noise cube Calculate the two-dimensional noise map

[0038]

[0039] Where: is a two-dimensional noise map; x and y are spatial coordinates; n is the position index in three-dimensional space.

[0040] Furthermore, in step 3, the expression of the mixed loss function is:

[0041]

[0042] Where: Loss is the mixed loss function; RMSE is the root mean square error; SSIM is the structural similarity index; is a two-dimensional noise map is the total variation of ; λ and τ are both weight coefficients.

[0043] Furthermore, the calculation formula of the root mean square error RMSE is:

[0044]

[0045] Where: X and Y are the height and width of the actual photon counting cube Z respectively;

[0046] The calculation formula of the structural similarity index SSIM is:

[0047]

[0048] Where: μ x and μ y Recover photon counting cubes and the mean of the actual photon counting cube Z; σ x and σ y Recover photon counting cubes and the variance of the actual photon counting cube Z; σ xy Recover photon counting cubes and the covariance of the actual photon counting cube Z; c1 and c2 are constants for computational stability.

[0049] 2D Noise Map Total variation The calculation formula is:

[0050]

[0051] Where: TV is the total variation function.

[0052] The beneficial effects of the present invention are:

[0053] 1. The present invention uses a training set to train and optimize a multi-stage sparse neural network. After the training and optimization, the multi-stage sparse neural network only needs to input the single-photon data to be reconstructed, that is, the test set, to reconstruct the final depth image. This method has high feasibility, strong universality, fast reconstruction speed, and stable imaging effect under low photon count conditions.

[0054] 2. The multi-stage sparse neural network of the present invention adopts a lightweight design and utilizes the sparsity of single-photon data, which can significantly reduce storage and computing overhead.

[0055] 3. The present invention adopts a noise removal module and introduces a learnable kernel basis. At each spatial position, linear adaptive fusion is performed through the kernel fusion coefficient to obtain the fusion kernel weight, and finally a bin consisting of active point coordinates and potential features is generated through convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of an embodiment of the present invention;

[0057] Figure 2 Schematic diagram of the single-photon three-dimensional reconstruction structure principle based on a multi-stage sparse neural network in an embodiment of the present invention;

[0058] Figure 3 is a partial network structure diagram of a noise removal module in an embodiment of the present invention;

[0059] Figure 4 is a schematic diagram of the network structure principle of the depth estimation module in an embodiment of the present invention;

[0060] Figure 5 is a schematic diagram of the network structure of the first asymmetric feature fusion unit and the second asymmetric feature fusion unit in an embodiment of the present invention;

[0061] Figure 6 This is a first result graph of the test of the trained multi-stage sparse neural network on the Middlebury 2006 data set in an embodiment of the present invention;

[0062] Figure 7 is a second result graph of the test of the trained multi-stage sparse neural network on the Middlebury 2006 data set in an embodiment of the present invention;

[0063] Figure 8 This is the third result diagram of the test of the trained multi-stage sparse neural network on the Middlebury 2006 data set in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, advantages and features of the present invention clearer, the following is a further detailed description of a single-photon 3D reconstruction method based on a multi-stage sparse neural network proposed by the present invention in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will be clearer according to the following specific implementation methods.

[0065] See also Figure 1 In this embodiment, a single-photon 3D reconstruction method based on a multi-stage sparse neural network specifically includes the following steps:

[0066] Step 1: Create a single-photon data set with low signal-to-noise ratio;

[0067] The single-photon dataset includes a training set, a validation set, and a test set. The training set and the validation set are noisy datasets obtained by simulating the NYU Depth Dataset V2 dataset using a single-photon lidar simulation system, and the test set is a noisy dataset obtained by simulating the Middlebury 2006 dataset using a single-photon lidar simulation system.

[0068] The noise levels of the simulated single-photon data mainly include: 10:2, 5:2, 2:2, 10:10, 5:10, 2:10, 10:50, 5:50, 2:50, and the noise level refers to the number of signal photons: ambient background light.

[0069] The training set contains single-photon data generated at various noise levels. A small portion of the single-photon data is randomly selected as the validation set for verification during the network training process, and the remaining single-photon data is used for actual network training.

[0070] The test set consists of single-photon data with different noise levels generated by simulating the Middlebury 2006 data set through a single-photon lidar simulation system, the corresponding noise-free depth map and photon count histogram. In the test set, single-photon data with different noise levels are stored separately for later data analysis.

[0071] Step 2: Build a multi-stage sparse neural network for single-photon 3D reconstruction;

[0072] See also Figure 2 The multi-stage sparse neural network is mainly composed of a sparse coding module, a noise removal module, a signal expansion module, a depth estimation module, a sparse decoding module, a joint selection module and a noise prior module.

[0073] Among them, the noise removal module and the signal expansion module are the first-stage network; the depth estimation module and the noise prior module are the second-stage network.

[0074] The noisy single-photon data are first compressed into a photon counting matrix D by a sparse coding module to contaminate the actual photon counting cube Z, which facilitates the extraction of active sites.

[0075] like Figure 3As shown in Figure 1, given a photon count matrix D, the noise removal module learns a set of learnable kernel bases W shared in all spatial locations and all images to capture common spatial patterns. In addition, the noise removal module generates different kernel base fusion coefficients F, and the learnable kernel base W is linearly fused with the kernel base fusion coefficient F to obtain the fusion weight G, which is as follows:

[0076]

[0077] Where: W(m) is the mth learnable kernel basis (kernel basis); F(i,j,m) and G(i,j) are the mth kernel basis fusion coefficient and the fusion weight at position (i,j) in the photon counting matrix D respectively; V is the number of kernel basis.

[0078] In addition, the photon counting matrix D undergoes a 1×1 convolution transformation to obtain the convolution transformed photon counting matrix D e , and finally the convolution transformed photon counting matrix D e Perform adaptive convolution with the fusion coefficient G to obtain a bin consisting of active point coordinates and potential features The formula is as follows:

[0079]

[0080] Because the warehouse Concentrate near the target depth, so it is necessary to Sent to the signal expansion module to activate the coordinates of the active points and potential features related to the target depth, and obtain the expanded photon count matrix The formula is as follows:

[0081]

[0082] Where: * represents the convolution operation; K s is the temporal convolution kernel, the size of the convolution kernel is (3,1,1); K t is the spatial convolution kernel, and the size of the convolution kernel is (1,3,3).

[0083] On this basis, abandoning the expanded photon counting matrix The potential features in the extract all active point coordinates as a more accurate target area Then the target area Input to the depth estimation module and joint selection module respectively.

[0084] The joint selection module selects the photon counting cube Z from the actual photon counting cube Z in the target area. The expanded photon counting matrix The calculation formula is:

[0085]

[0086] Where: coord is the coordinate; & is the joint selection.

[0087] The depth estimation module is used to stitch the target area With the expanded photon counting matrix After obtaining the stitched photon counting cube Perform depth estimation and obtain estimated photon count cube Then the estimated photon counting cube Input to the sparse decoding module for sparse decoding to obtain the restored photon counting cube That is the reconstructed depth image.

[0088] See also Figure 4 The depth estimation module includes a downsampling layer, an upsampling layer and two asymmetric feature fusion units (Asymmetric Feature Fusion, AFF). The overall structure of the depth estimation module is a U-shaped encoding-decoding symmetric structure; the encoding part is a downsampling layer composed of two sparse convolutions and four sub-manifold convolutions, which is used to extract target features; the decoding part is an upsampling layer composed of two sparse deconvolutions and four sub-manifold convolutions, which is used to restore target information. Figure 4 In the figure, → represents submanifold convolution, ↓ represents sparse convolution, and ↑ represents sparse deconvolution; N, 8×N, and 16×N represent features of different scales, respectively. AFF1 and AFF2 represent the first asymmetric feature fusion unit and the second asymmetric feature fusion unit, respectively.

[0089] See also Figure 5 The first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2 each include three adjusters (Resize), and a cascade C, a 1×1Conv (convolution) layer, and a 3×3Conv (convolution) layer connected in series in sequence; the input ends of the three adjusters are respectively used as the first input end, the second input end, and the third input end of the first asymmetric feature fusion unit AFF1 or the second asymmetric feature fusion unit AFF2; the output ends of the three adjusters are respectively connected to the three input ends of the first cascade C; the output end of the 3×3Conv (convolution) layer is used as the output end of the first asymmetric feature fusion unit AFF1 or the second asymmetric feature fusion unit AFF2.

[0090] It is defined that the two sparse convolutions in the downsampling layer are the first sparse convolution S1 and the second sparse convolution S2, and the four sub-manifold convolutions are the first sub-manifold convolution L1, the second sub-manifold convolution L2, the third sub-manifold convolution L3 and the fourth sub-manifold convolution L4; the two sparse deconvolutions in the upsampling layer are the first sparse deconvolution F1 and the second sparse deconvolution F2, and the four sub-manifold convolutions are the fifth sub-manifold convolution L5, the sixth sub-manifold convolution L6, the seventh sub-manifold convolution L7 and the eighth sub-manifold convolution L8.

[0091] The first sub-manifold convolution L1 and the second sub-manifold convolution L2 are connected in sequence, the input end of the first sub-manifold convolution L1 is the input end of the depth estimation module, the output end of the second sub-manifold convolution L2 is respectively connected to the input end (i.e., the first input end) of the first adjuster in the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2, and the input end of the first sparse convolution S1; the output end of the first sparse convolution S1 is connected to the input end of the third sub-manifold convolution L3, the output end of the third sub-manifold convolution L3 is respectively connected to the input end (i.e., the second input end) of the second adjuster in the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2, and the input end of the second sparse convolution S2; the output end of the second sparse convolution S2 is connected to the fourth sub-manifold convolution L4 The input end of the fourth sub-manifold convolution L4 is connected to the input end (i.e., the third input end) of the third adjuster in the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2, and the input end of the fifth sub-manifold convolution L5; the fifth sub-manifold convolution L5, the first sparse deconvolution F1, the sixth sub-manifold convolution L6, the second sparse deconvolution F2, the seventh sub-manifold convolution L7 and the eighth sub-manifold convolution L8 are connected in series in sequence; the second input end of the sixth sub-manifold convolution L6 is connected to the output end of the 3×3 convolution layer in the second asymmetric feature fusion unit AFF2, the second input end of the seventh sub-manifold convolution L7 is connected to the output end of the 3×3 convolution layer in the first asymmetric feature fusion unit AFF1, and the output end of the eighth sub-manifold convolution L8 is the output end of the depth estimation module.

[0092] The detection time of background noise photons is uniformly distributed on the pulse repetition interval (0, Tr]. This uniform distribution is independent of the reflected signal photons, spatial coordinates, and scene intensity, and can be regarded as a statistical prior for denoising. Therefore, in this embodiment, a noise prior module is also designed to further improve the reconstruction performance.

[0093] The noise prior module compares the input measurement value, the actual photon counting cube Z, with the bin Using residual connection, we can get a three-dimensional noise cube Since the above uniform distribution is independent of the reflected signal photons, spatial coordinates, and scene intensity, this means that each spatial coordinate should have the same average value. In addition, ideally, the three-dimensional noise cube All non-zero detections in are caused by background noise photons. Therefore, by The two-dimensional noise map can be generated by averaging the position index of each non-zero bin of the spatial coordinate (x, y) The specific calculation formula is:

[0094]

[0095] Where: n is the position index in three-dimensional space.

[0096] Finally, in the 2D noise map The total variation is calculated on θ and used as one of the loss functions to help train the network.

[0097] Step 3: Use the training set to train a multi-stage sparse neural network;

[0098] By inputting the training set data into the multi-stage sparse neural network, the forward and backward propagation of the multi-stage sparse neural network is used to optimize the overall network parameters. By using the method of using a mixed loss function to constrain the training of the multi-stage sparse neural network, the multi-stage sparse neural network can better fit the single-photon data in the training set, and obtain the multi-stage sparse neural network after training and optimization.

[0099] Specifically, the hybrid loss function mainly includes the following three parts:

[0100] The first is to restore the photon counting cube The root mean square error between the actual photon counting cube Z is:

[0101]

[0102] Where: X and Y are the height and width of the actual photon counting cube Z, respectively.

[0103] The second is to restore the photon counting cube The structural similarity index of the actual photon counting cube Z is as follows:

[0104]

[0105] Where: μ x and μ y Recover photon counting cubes and the mean of the actual photon counting cube Z; σ x and σ y Recover photon counting cubes and the variance of the actual photon counting cube Z; σ xy Recover photon counting cubes and the covariance of the actual photon counting cube Z; c1 and c2 are constants for computational stability.

[0106] The third one is a two-dimensional noise map Total variation The calculation formula is:

[0107]

[0108] Where: TV is the total variation function.

[0109] Therefore, the final hybrid loss function expression is:

[0110]

[0111] Wherein: Loss is the mixed loss function; λ and τ are weight coefficients, which are set to 0.43 and 0.29 respectively in this embodiment.

[0112] Step 4: Input the test set into the trained multi-stage sparse neural network to achieve low signal-to-noise ratio single-photon 3D reconstruction and generate a reconstructed depth image; see Figure 6 , Figure 7 and Figure 8 It can be seen that the imaging effect of the method of the present invention is stable on the test set, and a good depth image can still be restored under low photon count conditions.

Claims

1. A single-photon 3D reconstruction method based on a multi-stage sparse neural network, characterized in that: The following steps are involved: Step 1: Create a single photon dataset; The single photon data set includes a training set, a validation set and a test set; Step 2: Build a multi-stage sparse neural network for single-photon 3D reconstruction; Step 3: Input the training set into the multi-stage sparse neural network, optimize the network parameters by using the forward and backward propagation of the multi-stage sparse neural network, and use the mixed loss function to constrain the multi-stage sparse neural network to obtain the multi-stage sparse neural network after training and optimization; Step 4: input the verification set into the multi-stage sparse neural network after training optimization to verify whether the network performance of the multi-stage sparse neural network reaches the expected performance. If so, execute step 5; if not, return to step 3 until the expected performance is reached. Step 5: Input the test set into the trained and optimized multi-stage sparse neural network for single-photon 3D reconstruction to generate a reconstructed depth image.

2. The single-photon 3D reconstruction method based on a multi-stage sparse neural network according to claim 1, characterized in that: In step 2, the multi-stage sparse neural network includes a sparse coding module, a noise removal module, a signal expansion module, a depth estimation module, a sparse decoding module, a joint selection module and a noise prior module; The input end of the sparse coding module, the first input end of the joint selection module and the input end of the noise prior module are used as the input end of the multi-stage sparse neural network to receive the actual photon counting cube Z in the training set, the validation set and the test set; The sparse coding module is used to compress the actual photon counting cube Z into a photon counting matrix D through sparse coding, and input the photon counting matrix D into the noise removal module; The noise removal module is used to obtain a bin consisting of active point coordinates and potential features according to the photon counting matrix D. And the warehouse Input into the signal expansion module and the noise prior module respectively; The signal expansion module is based on the warehouse Get the expanded photon count matrix and select Expand Photon Count Matrix Get the target area from the coordinates of all active points in Then the target area Input to the depth estimation module and joint selection module respectively; The joint selection module is used to select the target area from the actual photon counting cube Z. The expanded photon counting matrix The depth estimation module is used to stitch the target area With the expanded photon counting matrix After obtaining the stitched photon counting cube Perform depth estimation and obtain estimated photon count cube Then the estimated photon counting cube Input to the sparse decoding module for sparse decoding to obtain the restored photon counting cube That is, the reconstructed depth image; The noise prior module is used to The three-dimensional noise cube is obtained by residual connection with the actual photon counting cube Z According to the three-dimensional noise cube Get a 2D noise map 3. The single-photon 3D reconstruction method based on a multi-stage sparse neural network according to claim 2, characterized in that: The calculation formula of the noise removal module is: Where: W(m) is the mth learnable kernel basis; F(i,j,m) and G(i,j) are the mth kernel fusion coefficient and the fusion kernel weight at position (i,j) in the photon counting matrix D, respectively; D e [i,j] is the photon counting matrix D after convolution transformation; V is the number of kernel bases.

4. The single-photon 3D reconstruction method based on a multi-stage sparse neural network according to claim 3, characterized in that: The signal expansion module calculates the expanded photon count matrix The formula is: Where: * represents the convolution operation; K s is the temporal convolution kernel; K t is the spatial convolution kernel.

5. The single-photon 3D reconstruction method based on a multi-stage sparse neural network according to claim 4, characterized in that: The calculation formula of the joint selection module is: Where: coord is the coordinate; & is the joint selection.

6. The single-photon 3D reconstruction method based on a multi-stage sparse neural network according to claim 5, characterized in that: The depth estimation module includes a downsampling layer, a first asymmetric feature fusion unit AFF1, a second asymmetric feature fusion unit AFF2 and an upsampling layer; The downsampling layer includes a first sparse convolution S1, a second sparse convolution S2, a first sub-manifold convolution L1, a second sub-manifold convolution L2, a third sub-manifold convolution L3 and a fourth sub-manifold convolution L4; the upsampling layer includes a first sparse deconvolution F1, a second sparse deconvolution F2, a fifth sub-manifold convolution L5, a sixth sub-manifold convolution L6, a seventh sub-manifold convolution L7 and an eighth sub-manifold convolution L8; The first sub-manifold convolution L1 and the second sub-manifold convolution L2 are connected in sequence, the input end of the first sub-manifold convolution L1 is the input end of the depth estimation module, and the output end of the second sub-manifold convolution L2 is respectively connected to the first input end of the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2, and the input end of the first sparse convolution S1; the output end of the first sparse convolution S1 is connected to the input end of the third sub-manifold convolution L3, and the output end of the third sub-manifold convolution L3 is respectively connected to the second input end of the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2, and the input end of the second sparse convolution S2; the output end of the second sparse convolution S2 is connected to the fourth sub-manifold convolution L 4, the output end of the fourth sub-manifold convolution L4 is respectively connected to the third input end of the first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2, and the input end of the fifth sub-manifold convolution L5; the fifth sub-manifold convolution L5, the first sparse deconvolution F1, the sixth sub-manifold convolution L6, the second sparse deconvolution F2, the seventh sub-manifold convolution L7 and the eighth sub-manifold convolution L8 are connected in series in sequence; the second input end of the sixth sub-manifold convolution L6 is connected to the output end of the second asymmetric feature fusion unit AFF2, the second input end of the seventh sub-manifold convolution L7 is connected to the output end of the first asymmetric feature fusion unit AFF1, and the output end of the eighth sub-manifold convolution L8 is the output end of the depth estimation module.

7. The single-photon 3D reconstruction method based on a multi-stage sparse neural network according to claim 6, characterized in that: The first asymmetric feature fusion unit AFF1 and the second asymmetric feature fusion unit AFF2 each include three adjusters, and a cascade C, a 1×1 convolutional layer, and a 3×3 convolutional layer connected in series in sequence; The input ends of the three adjusters are respectively used as the first input end, the second input end and the third input end of the first asymmetric feature fusion unit AFF1 or the second asymmetric feature fusion unit AFF2; the output ends of the three adjusters are respectively connected to the three input ends of the first cascade C; The output end of the 3×3 convolutional layer serves as the output end of the first asymmetric feature fusion unit AFF1 or the second asymmetric feature fusion unit AFF2.

8. The single-photon 3D reconstruction method based on a multi-stage sparse neural network according to claim 6, characterized in that: The noise prior module uses the following formula based on the three-dimensional noise cube Calculate the two-dimensional noise map Where: is a two-dimensional noise map; x and y are spatial coordinates; n is the position index in three-dimensional space.

9. The single-photon 3D reconstruction method based on a multi-stage sparse neural network according to any one of claims 2 to 7, characterized in that: In step 3, the expression of the mixed loss function is: Where: Loss is the mixed loss function; RMSE is the root mean square error; SSIM is the structural similarity index; is a two-dimensional noise map is the total variation of ; λ and τ are both weight coefficients.

10. The single-photon 3D reconstruction method based on a multi-stage sparse neural network according to claim 9, characterized in that: The calculation formula of the root mean square error RMSE is: Where: X and Y are the height and width of the actual photon counting cube Z respectively; The calculation formula of the structural similarity index SSIM is: Where: μ x and μ y Recover photon counting cubes and the mean of the actual photon counting cube Z; σ x and σ y Recover photon counting cubes and the variance of the actual photon counting cube Z; σ xy Recover photon counting cubes and the covariance of the actual photon counting cube Z; c1 and c2 are constants for computational stability; The two-dimensional noise map Total variation The calculation formula is: Where: TV is the total variation function.

Images