A depth image reconstruction method based on single-photon detector

Abstract

The present invention relates to a depth image reconstruction method based on a single-photon detector, and belongs to the field of computational photography and image processing technology. First, a downsampled 3D convolution is input, and the output of the initial downsampled convolution is sent to an encoder for feature extraction. After each encoding layer, the extracted features are processed by a residual block, and these processed features are connected to the input of the corresponding decoding layer. In the last decoding layer, the input consists of the concatenated features processed by the residual block of the encoding layer and the original features. Finally, an upsampling convolution layer is used to output a denoised photon efficiency measurement histogram. The present invention reduces the redundancy of information in the time dimension, can handle the problem of efficient depth reconstruction of photons under extremely low signal-to-noise ratio conditions, and especially when processing signal-to-noise level inputs that have not been learned during the training process, it can also maintain a certain depth reconstruction performance, proving that the present invention has strong robustness.

Description

Technical Field

[0001] The present invention relates to a depth image reconstruction method based on a single-photon detector, and belongs to the technical field of computational imaging and image processing. Background Art

[0002] Active 3D imaging is a technology that uses signals sent and received by sensors to obtain the 3D structure and information of target objects. It has wide applications in many fields such as robotics, vehicle navigation and remote sensing.

[0003] Typical 3D imaging sensors require a sufficient number of photons to mitigate the effects of background noise. In order to reduce reliance on photon counting, some studies in recent years have tended to use SPAD sensors with higher temporal resolution and photon detection sensitivity for three-dimensional imaging. Active 3D imaging systems based on SPAD arrays typically detect only one photon per pixel on average, significantly improving photon efficiency. However, in some cases, such as remote sensing of long-distance dynamic scenes, non-line-of-sight imaging, and microscopic imaging of delicate biological samples, limitations in light flux and integration time hinder the collection of effective signal photons. High background noise and weak signal photons severely reduce the quality of single-photon data, affecting subsequent tasks.

[0004] To improve imaging quality in low-light environments, some researchers have explored single-photon 3D imaging in low-light environments. These methods can be categorized into two categories: traditional computation-based methods and data-driven methods based on deep learning. Traditional methods utilize the first photon received at each pixel location to recover depth and intensity information. Subsequently, methods have emerged that use array-specific algorithms to convert time-boxed photon detections into highly accurate scene depth and reflectivity. These frameworks achieve high photon efficiency by leveraging the scene's structural information in both the horizontal and vertical domains, effectively filtering out irrelevant detections from the SPAD array, such as background light and dark counts. Compared to traditional histogram techniques, these methods are more effective in low-light conditions. However, their performance degrades significantly with further reductions in photon counts and signal-to-source ratio (SBR).

[0005] With the development of deep learning technology, deep learning-based methods have shown good results in the field of single-photon imaging. Some methods combine technologies such as convolutional neural networks and non-local networks to achieve end-to-end depth reconstruction. Currently, researchers have integrated high-resolution intensity map information into the depth reconstruction process to alleviate the challenge of obtaining depth maps from low-resolution single-photon arrival time information. However, in practical applications, it is difficult to obtain intensity images that match single-photon data. In addition, neural networks that are not specifically designed to process sparse single-photon data have difficulty in achieving good performance. These problems will lead to poor performance. Summary of the Invention

[0006] The present invention aims to overcome the shortcomings and drawbacks of existing technologies by creatively proposing a method for depth image reconstruction based on single-photon detectors. This method samples a 3D UNet in a temporal dimension for photon-efficient depth image reconstruction, further enhancing the fidelity of depth reconstruction under low signal-to-noise ratio conditions and improving network efficiency.

[0007] The innovative features of the present invention include:

[0008] First, the encoder adopted by the present invention includes 3D convolution with time downsampling to compress time dimension information and reduce redundancy, thereby improving the efficiency and performance of feature processing in the network.

[0009] Secondly, the present invention adds a feature enhancement block in each encoder layer to further enhance the signal photon features with stronger time correlation and weaken the background noise features.

[0010] The present invention adopts a gating mechanism to emphasize the signal photon characteristics with long-range temporal correlation.

[0011] Experimental results show that the proposed method outperforms the state-of-the-art methods in depth reconstruction performance under low-photon technology conditions.

[0012] Furthermore, our network is able to reconstruct high-quality depth maps in the presence of noise levels not learned during training, demonstrating the robustness of our approach, which is crucial in practical applications.

[0013] The present invention is implemented by adopting the following technical solutions.

[0014] A depth image reconstruction method based on a single-photon detector comprises the following steps:

[0015] Step 1: Measure the input photon efficiency and perform 3D convolution to downsample the time dimension.

[0016] Step 2: The output of the time-dimension downsampled 3D convolution is used as the input of the encoder, and the encoder is used to extract features to obtain features at different time-dimension scales. The encoder includes downsampling convolution and feature enhancement blocks.

[0017] Step 3: Process the extracted features through the residual block and connect the processed features with the input of the corresponding decoder, except for the last decoding layer. In the last decoder, the input consists of the concatenated features processed by the residual block of the encoding layer and the original features;

[0018] Step 4: Use upsampling convolution to output a denoised photon efficiency measurement histogram.

[0019] Beneficial effects

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] 1. This paper designs a temporally downsampled 3D Unified Network (UNet) to compress temporal information and improve network efficiency. Photon efficiency measurements exhibit sparsity in the temporal dimension, which means that processing the input data as a whole is inefficient and can lead to suboptimal results. Therefore, this paper reduces the redundancy of temporal information by using downsampled 3D convolutions and a UNet-like encoder-decoder structure at the network header.

[0022] 2. This invention designs a feature enhancement block. Photon efficiency measurements exhibit long-range correlations in the temporal dimension, making the contribution of signal photon features more prominent in deeper feature extraction. To exploit this characteristic, the feature enhancement module designed in this invention utilizes input and output gates to selectively retain important signal photon features while discarding noise photon features.

[0023] 3. The network designed in the present invention can handle the problem of efficient depth reconstruction of photons under extremely low signal-to-noise ratio conditions. In particular, when processing signal-to-noise level inputs that have not been learned during the training process, it can also maintain a certain depth reconstruction performance, proving that the present invention has strong robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall flow chart of the method of the present invention;

[0025] Figure 2 It is a structural diagram of the deep neural network of the method of the present invention. DETAILED DESCRIPTION

[0026] The method of the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0027] Example

[0028] like Figure 1 As shown, a depth image reconstruction method based on a single-photon detector includes the following steps:

[0029] Step 1: The input photon efficiency measurement is first subjected to a time dimension downsampling 3D convolution to reduce the redundancy of the time dimension. Specifically, in this embodiment, the number of channels output by the time dimension downsampling 3D convolution can be set to 4.

[0030] The process is expressed as:

[0031]

[0032] in, is the input photon efficiency measurement, The output of the 3D convolution is downsampled in the time dimension, Represents a 3D convolution with downsampling in the time dimension.

[0033] Step 2: Use the output of the 3D convolution after time dimension downsampling as the input of the encoder, and use the encoder to extract features to obtain the feature information of the input data at different time dimension scales.

[0034] Specifically, let the number of channels of the output feature be , the output channels of the initial down-sampling convolution are doubled after each encoder (the final number of output channels of the encoder is The encoder consists of a downsampling convolution (the convolution kernel size is 3 and the stride is 2) and a feature enhancement block. Through the encoder, features at different time dimension scales are obtained. , the process is:

[0035]

[0036] in, Indicates the Layer encoder.

[0037] Unlike traditional UNets, which directly feed feature information extracted from each encoder level into the decoder via skip connections, the network in this paper further processes and extracts features at different time scales using residual blocks. Each residual block consists of at least two 3D convolutions, two batch normalization layers, two activation function layers, and a skip connection. After each encoder, the extracted features are processed using residual blocks, with four in the last layer and one in each of the remaining layers.

[0038] Specifically, after the feature is input into the residual block, it is first processed by 3D convolution.

[0039] Then, batch normalization and activation functions are applied to enhance the representation power and convergence speed of the network.

[0040] Afterwards, the features are again processed by another 3D convolution followed by a batch normalization layer.

[0041] Subsequently, element-wise addition is performed between the output and the input via a skip connection, and the result is passed through an activation function to obtain the final output.

[0042] From the input features To output features The whole process is expressed as:

[0043]

[0044] in, Respectively represent the intermediate vectors in the process of residual block processing from input features to output features, represents convolution, represents the batch normalization operation, Represents the ReLU activation function.

[0045] Finally, feature enhancement is used to enhance the signal photon characteristics and weaken the background noise characteristics.

[0046] Signal photons exhibit long-range correlations in the temporal dimension. In contrast, noise photons exhibit random characteristics due to their random generation process. Our analysis found that in the feature maps extracted from the encoder, the deeper the layer, the higher the proportion of signal photon features. Inspired by this, we employ an LSTM-like structure, combined with a gating mechanism, to enhance significant features in the input and attenuate less significant ones.

[0047] Specifically, first according to the input features Generate input gate , feature unit and output gate The process is expressed as:

[0048]

[0049] in, Represents the sigmoid activation function.

[0050] Input Gate Determines which features in the input feature map are emphasized, and it uses the sigmoid activation function to generate values ​​between 0 and 1 to control the importance of the input features. Similarly, the output gate Determines the importance of the final output feature map, and it also uses the sigmoid activation function to control the importance of the output feature. Feature unit Used to store and propagate the features of the current layer, it uses the hyperbolic tangent (tanh) activation function to improve the convergence speed of the network.

[0051] Output features of feature enhancement block Expressed as:

[0052] Here, × represents element-by-element matrix multiplication.

[0053] Step 3: After being processed, the output of each layer of the encoder is concatenated with the output of the previous layer of the corresponding layer decoder in the channel dimension as the input of the current layer decoder.

[0054] For the first layer decoder, it corresponds to the last layer encoder, and its input is the result of the output features of the last layer encoder being processed by 4 layers of residual blocks.

[0055] The whole process is expressed as:

[0056]

[0057] Among them, Cat represents the channel dimension splicing operation, It is The output of a decoder.

[0058] In step 4, the upsampling convolutional layer is used to output the denoised histogram . is defined as:

[0059]

[0060] in, represents the upsampling convolutional layer, Represents the output of the last layer of decoder.

[0061] The 2D depth map can be obtained from Extract the index with the maximum value to generate.

[0062] It should be noted that the above explanation of the embodiment of the photon efficient depth reconstruction method based on a single-photon detector is also applicable to the photon efficient depth reconstruction system based on a single-photon detector of this embodiment.

[0063] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A depth image reconstruction method based on a single-photon detector, characterized in that: The following steps are involved: Step 1: Measure the input photon efficiency and perform 3D convolution with time dimension downsampling; Step 2: Take the output of the time-dimension downsampled 3D convolution as the input of the encoder, and use the encoder to extract features to obtain features at different time-dimension scales. The encoder includes downsampling convolution and feature enhancement blocks. Let the number of channels of the output feature be N. b , the output channels of the initial downsampling convolution are doubled after each encoder, so the final number of output channels of the encoder is Through the encoder, features at different time dimensions are obtained The process is: Among them, E i represents the i-th layer encoder; Features at different time scales are extracted through residual block processing. Each residual block consists of at least two 3D convolutions, two batch normalization layers, two activation function layers, and a skip connection. After each encoder, the extracted features are processed through residual blocks, with the last layer having four and the other layers having one. When the feature is input into the residual block, it is first processed by 3D convolution; Then, batch normalization and activation functions are applied to enhance the representation ability and convergence speed of the network; After that, the features are again processed by another 3D convolution, followed by a batch normalization layer; Subsequently, element-wise addition is performed between the output and the input via a skip connection, and the result is passed through an activation function to obtain the final output; From input feature F to output feature F ′ The whole process is expressed as: F1=Conv3D(F) F2=ReLU(BN(F1)) F3=Conv3D(F2) <h2 style=";text-align:left;direction:ltr">F<h2 style=";text-align:left;direction:ltr"> ′ <h2 style=";text-align:left;direction:ltr"> =ReLU(BN(F3)+F) Among them, F1, F2, and F3 represent the intermediate vectors in the process of residual block processing from input features to output features, Conv represents convolution, BN represents batch normalization operation, and ReLU represents ReLU activation function; Step 3: Process the extracted features through the residual block and concatenate the processed features with the input of the corresponding decoder, except for the last decoder; in the last decoder, the input consists of the concatenated features processed by the residual block of the encoding layer and the original features; Step 4: Use upsampling convolution to output the denoised photon efficiency measurement histogram; use upsampling convolution layer to output the denoised histogram Defined as: Among them, upConv represents the upsampling convolution layer, Represents the output of the last layer decoder; the 2D depth map is obtained from Extract the index with the maximum value to generate.

2. The method for depth image reconstruction based on a single-photon detector according to claim 1, wherein: In step 1, the time dimension downsampling 3D convolution is expressed as: Among them, H ′ is the input photon efficiency measurement, It is the output of the time dimension downsampling 3D convolution, and downConv represents the time dimension downsampling 3D convolution.

3. The method for depth image reconstruction based on a single-photon detector according to claim 1, wherein: In step 2, the number of channels of the 3D convolution output in the time dimension is downsampled to 4.

4. The method for depth image reconstruction based on a single-photon detector according to claim 1, wherein: In step 2, the downsampling convolution kernel size in the encoder is 3 and the stride is 2.

5. The method for depth image reconstruction based on a single-photon detector according to claim 1, wherein: In step 3, the output of each layer of the encoder is processed and concatenated with the output of the previous layer of the corresponding layer decoder in the channel dimension as the input of the current layer decoder; For the first layer decoder, it corresponds to the last layer encoder, and its input is the result of the output features of the last layer encoder being processed by 4 layers of residual blocks.

Images