Method for Mama single photon 3D reconstruction network based on two-way wavelet feature enhancement

Abstract

The invention discloses a Mama single photon 3D reconstruction neural network method based on two-way wavelet feature enhancement, and the method achieves the accurate extraction and recovery of effective depth features in an original single photon point cloud through introducing a state space modeling mechanism with a dynamic regulation and control capability. Comprising the steps that a dual-path feature reconstruction module combined with wavelet transformation is designed, the reconstruction process is divided into a low-frequency path and a high-frequency path, the low-frequency path is used for representing a global structure and spatial continuity, and the high-frequency path is used for bearing edge information and local texture details. According to the method, low-frequency and high-frequency features are extracted and strengthened respectively, collaborative modeling of local edge details and a global geometric structure is achieved, the definition and structural integrity of a depth map are effectively improved, and compared with a traditional single-photon three-dimensional reconstruction method, the method has the advantages of being higher in anti-noise capacity and higher in robustness. And more real and coherent space depth information can be generated.

Description

Technical Field

[0001] This invention relates to the fields of photoelectric detection and three-dimensional imaging technology, specifically to a method for single-photon 3D point cloud reconstruction based on a dual-path wavelet feature-enhanced Mamba network, which is suitable for high-precision three-dimensional reconstruction in complex environments such as low-light imaging, long-distance imaging, and non-line-of-sight imaging. Background Technology

[0002] LiDAR (Light Detection and Ranging) is an imaging method that uses the system's own illumination to acquire target scene information. It is widely used in satellite imaging, laser ranging, and remote sensing, with operating range being a core performance indicator. Traditional linear APD detectors have low sensitivity and cannot meet the requirements of ultra-long-range detection. Single-photon avalanche diodes (SPADs), due to their advantages such as single-photon response, high signal gain, fast response, and small size, have become key devices for long-range detection, and SPAD-based single-photon lidar has become an important technological path for long-range detection. However, in single-photon imaging tasks, complex environmental factors such as atmospheric scattering and attenuation during long-distance transmission result in a very small number of target reflected photons received by the detector (on the photon or sub-photon scale), leading to an extremely low signal-to-background ratio (SBR). Low SBR conditions severely inhibit the reliability of target information extraction, causing problems such as large ranging errors, blurred images, and loss of some structural details in the reconstruction results.

[0003] Existing single-photon reconstruction methods have limitations: probabilistic models combined with scene structure have limited denoising effects; stable hierarchical Bayesian algorithms have high complexity; deep learning methods require additional sensors to obtain reflectivity information; and methods that employ network structures integrating nonlocal attention modules (NLA), lightweight Unet-based networks that fuse multidimensional spatial and temporal features to extract information, or focus on the accurate restoration of edge structures in depth maps and introduce ADAG modules into their Unet-based network architectures to enhance edge feature extraction capabilities, struggle to balance edge details and global structure reconstruction at low SBR (signal-to-background ratio), resulting in insufficient noise resistance and detail fidelity. Summary of the Invention

[0004] This invention aims to solve the problems of spatial structure distortion and loss of local texture details in the reconstruction of depth maps under low SBR conditions in single-photon 3D imaging, and to achieve single-photon point cloud reconstruction with high noise resistance and high detail fidelity. It provides a method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0006] This invention discloses a method for Mamba single-photon 3D reconstruction networks based on dual-path wavelet feature enhancement, the method comprising the following steps:

[0007] Step S1: Create a training dataset, using the NYUv2 dataset as a basis to simulate and generate single-photon detection data; the training dataset contains 35,840 training data and 1,070 validation data, each data is 1×64×64×1024 in size, and covers 9 different SBR scenarios;

[0008] Step S2: Construct a Mamba reconstruction network based on dual-path wavelet feature enhancement. The network includes an encoding module, a long-range dependency modeling module, and a decoding module. The encoding module is a residual convolutional shallow feature extraction module. The long-range dependency modeling module is a cascaded dual-path feature reconstruction module with wavelet decomposition. The decoding module is a residual convolutional feature integration module.

[0009] Step S3: Train the Mamba reconstruction network using the training dataset;

[0010] Step S4: Use the trained Mamba reconstruction network to perform reconstruction tests on the test set and real point cloud data to obtain the depth map reconstruction results of single-photon point clouds.

[0011] Preferably, in step S1, the process of simulating and generating single-photon detection data includes: extracting the depth map and grayscale map from the RGB-D image pair of NYUv2, interpolating and completing the depth map, converting it into a ToF map according to the laser time-of-flight principle, quantizing it into 1024 time bins, generating the signal photon number by combining the illumination intensity and distance attenuation model, adding background noise based on the grayscale map, and obtaining a sparse three-dimensional time detection tensor through non-homogeneous Poisson sampling and point spread function convolution.

[0012] Preferably, in step S2, the dual-path feature reconstruction module includes a low-frequency path and a high-frequency path; the low-frequency path enhances the global structural continuity through spatial and channel attention; the high-frequency path captures edge details and local texture changes through a lightweight self-attention module.

[0013] Preferably, the long-distance dependency modeling module adopts a layer-by-layer stacked DFRM structure, with each layer of DFRM structure independently completing a round of high-frequency-low-frequency feature separation and structural enhancement;

[0014] The DFRM structure does not require an external compensation path. It achieves bottom-up feature propagation through frequency decoupling and attention feature enhancement modules. The output features of each layer inherit the structural information of the previous layer and enhance new high and low frequency information.

[0015] Preferably, in step S3, the loss function used in the training process is:

[0016] L total =MAE range +λ1(1-SSIM range )+λ2(1-Pixels range )+λ3(1-TV range );

[0017] Among them, λ1=0.2, λ2=0.025, λ3=0.3; MAE range SSIM is the mean absolute error. range Pixels is a structural similarity index. range For pixel loss, TV range This represents the total variation loss.

[0018] Preferably, the MAE range The calculation formula is:

[0019]

[0020] Where Z is the target image, Y is the network-reconstructed image, and n is the number of pixels in the image.

[0021] Preferably, the SSIM range The calculation formula is:

[0022]

[0023] Where the mean of Z and Y is μ z μ y The variances of Z and Y are respectively The product of σ zy Let the covariance be c1, where c1 and c2 are both constants.

[0024] Preferably, the Pixels range The calculation formula is:

[0025]

[0026] Where W is the width of the reconstructed image and H is the height.

[0027] Preferably, the TV range The calculation formula is:

[0028]

[0029] Where Y represents the image reconstructed by the network, and i and j represent the position of a certain pixel value.

[0030] Preferably, in step S2, the input data size of the Mamba reconstruction network is B×H×W×T, where B is the batch size, H and W are the input sizes, and T is the time dimension. The network uses 2D convolution, and the time dimension T is used as the channel of the two-dimensional convolution to form a reasonable and efficient method for training neural networks. Furthermore, it has been verified with real datasets to ensure its excellent single-photon point cloud data reconstruction performance.

[0031] In step S4, the real point cloud data acquisition device includes a fiber pulsed laser with a repetition frequency of 20kHz, a wavelength of 1064nm, a pulse width of 1ns, and a peak power of 500mW, and is used with an imaging lens with a focal length of 8mm, an effective aperture of 5mm, and a field of view of 23°.

[0032] Beneficial effects: 1) Excellent noise resistance: It can still output high-quality depth maps stably under low SBR (low photon) conditions; 2) Balance of detail and structure: The dual-path feature modules enhance global structure and edge details respectively, avoiding the loss of details or structural distortion of traditional methods; 3) Computational efficiency: The DFRM structure reduces computational redundancy and feature redundancy, improving training and inference efficiency; 4) Strong generalization: In both simulated and real-world scenarios, the edge details and global structure consistency of the reconstruction results are better than existing methods, and are closer to the shape of real objects. Attached Figure Description

[0033] Figure 1 This is a flowchart of the method of the present invention.

[0034] Figure 2 This is a diagram of the neural network model structure of the method of the present invention.

[0035] Figure 3 This is a structural diagram of the dual-path feature attention enhancement module of the method of the present invention.

[0036] Figure 4 It is the mean square error index of the method of the present invention compared with other methods.

[0037] Figure 5 This is a diagram showing the model training results of the method of this invention.

[0038] Figure 6 These are the real-world point cloud data test results of the method of this invention. Detailed Implementation

[0039] The following will refer to the accompanying drawings in the embodiments of the present invention. Figures 1-6The technical solutions in the embodiments of this invention are clearly and completely described herein. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. In this application, Shin, Peng, Zang, and Zhao are all abbreviations of authors recorded in prior art literature. Figure 2 Neural network model structure diagram Figure 3 The English phrases and abbreviations involved in the dual-path feature attention enhancement module structure diagram are all existing technologies and can be interpreted according to their meanings.

[0040] Example 1:

[0041] A Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement, such as Figure 1 As shown, the steps of this method are as follows:

[0042] Step S1: Training Dataset Creation. The dataset required for network training is generated based on NYU v2 (NYU Depth Datasetv2) for simulation. During the simulation, RGB-D (distance image) image pairs from NYU v2 are used to extract depth maps and grayscale images as distance and albedo information for the real scene, respectively. After interpolation to fill missing values, the depth map is converted into a Time-of-Flight (ToF) image according to the laser time-of-flight principle and quantized into 1024 time bins (time resolution of 80 ps). The signal photon count is generated by combining the illumination intensity and distance attenuation models, and background noise simulation based on the grayscale image is added. The photon detection process is implemented through non-homogeneous Poisson sampling. The signal is convolved in the time dimension according to the point spread function (PSF), ultimately obtaining a sparse three-dimensional time-sensing tensor to simulate the single-photon data returned by a real SPAD. The data obtained from simulating SPAD detection is used in this indoor dataset, which includes multiple indoor scenes. Using multiple scenes from the NYU v2 dataset allows the trained model to have better generalization ability. It contains 35,840 data points as the training set and 1,070 data points as the validation set. Each data point is 1×64×64×1024 in size, and nine different SBR scenarios are set. High-quality depth information maps can be obtained from single-photon point cloud data reconstruction under different SBR conditions.

[0043] Step S2: To address the degradation of spatial structure and local texture details in the reconstructed distance image under strong background noise interference from single-photon 3D imaging, this invention designs a Mamba network based on dual-path wavelet feature enhancement. The overall network structure includes three parts: a shallow feature extraction encoding module, a long-distance dependency modeling module, and a depth map reconstruction decoding module. The shallow feature extraction module is a residual convolution, the long-distance dependency modeling module is a cascaded dual-path feature reconstruction module with wavelet decomposition, and the depth map reconstruction again utilizes residual convolution to integrate the features of the depth information extracted in the second part as the final decoding module. As a preferred embodiment, in step S2, the input data size of the Mamba reconstruction network is B×H×W×T, where B is the batch size, H and W are the input dimensions, and T is the time dimension. The network uses 2D convolution, with the time dimension T as the channel of the 2D convolution, forming a reasonable and efficient method for training neural networks. Furthermore, its excellent single-photon point cloud data reconstruction performance has been verified using real datasets.

[0044] Step S3: Train the initial target model based on the target dataset: Train the Mamba reconstruction network using the training dataset;

[0045] Step S4: Experiment with the test set using the trained Mamba network weights to obtain the reconstruction results of single-photon point cloud data, and simultaneously capture real datasets for network testing; as a preferred option, in step S4, the real point cloud data acquisition device includes a fiber pulsed laser with a repetition frequency of 20kHz, a wavelength of 1064nm, a pulse width of 1ns, and a peak power of 500mW, coupled with an imaging lens with a focal length of 8mm, an effective aperture of 5mm, and a field of view of 23°.

[0046] This invention discloses a Mamba single-photon 3D reconstruction neural network based on dual-path wavelet feature enhancement. The neural network model structure diagram is shown below. Figure 2 As shown, this method introduces a state-space modeling mechanism with dynamic adjustment capabilities to accurately extract and recover effective depth features from the original single-photon point cloud. Specifically, a dual-path feature reconstruction module combining wavelet transform is designed, dividing the reconstruction process into low-frequency and high-frequency paths: the low-frequency path is used to characterize the global structure and spatial continuity, while the high-frequency path carries edge information and local texture details. By extracting and enhancing low-frequency and high-frequency features respectively, collaborative modeling of local edge details and global geometric structure is achieved, effectively improving the clarity and structural integrity of the depth map. Compared with traditional single-photon 3D reconstruction methods, this invention exhibits stronger noise resistance in measured data and can generate more realistic and coherent spatial depth information.

[0047] This training uses a dataset generated based on NYU v2 for simulation. RGB-D (depth image) pairs from NYU v2 are used to extract depth and grayscale images, which serve as distance and albedo information for the real scene, respectively. During the simulation, the depth map is interpolated to fill in missing values, then converted to a Time-of-Flight (ToF) image according to the laser time-of-flight principle, and quantized into 1024 time bins (time resolution 80 ps). The signal photon count is generated by combining illumination intensity and distance attenuation models, and background noise based on the grayscale image is simulated. The photon detection process is implemented through non-homogeneous Poisson sampling, and the signal is convolved with a point spread function (PSF) in the time dimension, ultimately obtaining a sparse 3D time-probe tensor to simulate single-photon data returned by a real SPAD. This dataset is an indoor dataset containing multiple indoor scenes; using multiple scenes from NYU v2 can improve the generalization ability of the trained model. The dataset contains 35,840 training data points and 1,070 validation data points, each data point being 1×64×64×1024, and nine different SBR scenes are set. Finally, a single-photon camera was used to collect real point cloud data to test the neural network and verify its performance. Details are as follows:

[0048] First, the single-photon point cloud data is input into the encoder for encoding.

[0049] The encoder's output tensor is then processed by a long-range dependency modeling module to extract long-range correlations between point clouds. The long-range dependency modeling part is a cascaded dual-path feature reconstruction module with wavelet decomposition. The long-range dependency modeling module designed in this application adopts a layer-by-layer stacked DFRM structure (the tensor after the encoding module enters the LCFE module, which is composed of cascaded DFRM modules; the DFRM module is composed of an improved windmill state space module and a dual-path feature reconstruction module composed of wavelet transform). Each layer of the DFRM structure has the ability to independently complete a round of high-frequency-low-frequency feature separation and structural enhancement. In this process, the low-frequency path enhances the integrity and consistency of the global structure of continuous and smooth regions in the image through the spatial and channel attention paths in the dual-path enhanced feature attention module, effectively avoiding spatial misinterpretations caused by photon noise. The high-frequency path focuses on capturing fine-grained edge features and local texture details in the image, utilizing a complete dual-path feature attention enhancement module, such as... Figure 3As shown, after multi-level stacking, the network not only achieves a progressive expansion of the receptive field, but also constructs multi-scale, multi-frequency information fusion paths at each level. Unlike traditional methods based on skip connections or explicit attention fusion, the multi-DFRM structure proposed in this application does not require the introduction of external compensation paths. Instead, it relies on the frequency decoupling of the modules themselves, dynamic modeling of attention feature enhancement modules, and information progression strategies to build a compact and stable feature propagation path from the bottom up. The output features of each layer not only carry the structural point cloud information relationship of the previous layer, but also obtain new high-frequency and low-frequency information enhancements through independent frequency response modules, realizing deep collaboration between global structural modeling and local texture detail depiction. This highly integrated and expressive stacked architecture not only significantly improves the overall reconstruction performance of the model, but also effectively reduces computational redundancy and feature redundancy problems in the model training and inference process. Finally, the network completes the reconstruction of the depth map through the decoding module. This module is responsible for restoring the extracted deep feature maps into a complete depth map, and while maintaining the spatial structure, removing redundant noise and irrelevant details as much as possible to improve reconstruction accuracy. The overall network, through the synergistic effect of the encoding module, long-distance dependency modeling module, and decoding module, can effectively improve the reconstruction of highly reliable depth information under sparse point cloud conditions.

[0050] Finally, the weights obtained from neural network training are used to reconstruct real-world point cloud data.

[0051] The loss function (which can be modular) is used to evaluate the error between the network's predictions and the ground truth labels, and guides the network in backpropagation optimization. A hybrid loss function combining the mean absolute error (MAE) between the network's output 2D image and the target image, and structural similarity (SSIM) is chosen for evaluation. MAE measures the numerical error of the image, representing the average error magnitude between the output and target images. This metric is robust to outliers, and its calculation formula is:

[0052]

[0053] Where Y represents the image reconstructed by the network, Z represents the target image, and n represents the number of pixels in the image. To improve the structural similarity between the reconstructed image and the target image and enhance the network's generalization ability, this application introduces a structural similarity index (SSIM) as a constraint. SSIM is calculated based on a comparison of the brightness, contrast, and structure of the reconstructed image and the target image. In this network, SSIM can be expressed as:

[0054]

[0055] The mean values ​​of Z and Y are μ z μy The variances of Z and Y are and The product of these terms, their covariance is σ. zy c1 and c2 are constants. The SSIM value ranges from 0 to 1; the closer the value is to 1, the higher the similarity between the two images. 1-SSIM range It is part of the loss function. This application's technical solution introduces Pixels. range Pixel loss constrains the difference between predicted values ​​and labels, Pixels range It can be represented as:

[0056]

[0057] Where W is the width of the reconstructed image and H is the height. Pixel loss helps maintain stability during training, especially in image reconstruction tasks. Because it directly measures the error in image reconstruction, the model is easier to optimize, 1-Pixels. range As a component of the loss function:

[0058]

[0059] In the distance information reconstruction process, total variation loss is introduced to smooth the distance information and reduce noise or unnatural "blocky" effects, where Y represents the image reconstructed by the network, and i and j represent the position of a certain pixel value. 1-TV range As part of the constraints in the loss function, the complete loss function is as follows:

[0060] L total =MAE range +λ1(1-SSIM range )+λ2(1-Pixels range )+λ3(1-TV range );

[0061] λ1, λ2, and λ3 exist as hyperparameters in the loss function, with sizes of 0.2, 0.025, and 0.3, respectively.

[0062] This invention presents a Mamba single-photon 3D reconstruction neural network based on dual-path wavelet feature enhancement. After 15 rounds of training, the results are as follows: Figure 4 As shown.

[0063] This invention selects and demonstrates four different signal-to-noise ratios for the "Art" scene to compare the reconstruction effects of different methods. Shin's reconstruction method (the existing technology Shin et al. use information from the scene structure to reconstruct the corresponding probability model and remove background noise pixel by pixel) has many details and edge blurring issues. Even at high signal-to-noise ratios, the reconstruction effect still differs greatly from the true value. At low signal-to-noise ratios, it is no longer possible to reconstruct the approximate outline. Peng, Zang, and Zhao used neural networks to predict 3D histograms (specifically: Peng et al. constructed a network structure integrating nonlocal attention modules (NLA) to mine the long-range correlation of SPAD data in the spatiotemporal dimension, achieving the ability to reconstruct high-quality depth maps even under extremely low photon flux; Zang et al. proposed a Unet-based lightweight network that integrates multi-dimensional spatial and temporal features to extract information, and optimized the balance between speed and accuracy of depth estimation through adaptive sampling mechanism and spatiotemporal joint modeling; Zhao et al. focused on the accurate restoration of edge structures in depth maps, introducing an ADAG module into their Unet-based network architecture to enhance edge feature extraction capabilities, thereby improving the restoration quality of depth map boundaries in complex scenes), and then used 3D histograms for estimation to obtain 2D depth maps.

[0064] 3D point cloud data itself contains rich spatial structure and details, especially the edge information of the target. While depth maps can be reconstructed relatively accurately under high signal-to-noise ratio (SNR) conditions, differences still exist in some precise details of the brush strokes. Compared to the reconstruction effect under high SNR conditions, the method proposed in this application has superior performance in terms of detail and edges. The method proposed by Peng et al., under the total variation (TV) constraint method, suffers from blurred details and unsatisfactory edge reconstruction results due to noise. The method proposed by Zang et al. exhibits many singular highlights in the low SNR reconstruction structure, indicating inaccurate depth estimation. The method proposed by Zhao et al. uses the ADAG module to enhance edge details, achieving a smoother edge effect, but its performance in detail is weaker than that of Peng et al., with more severe blurring in brush stroke details.

[0065] The inventive aspect of this application lies in its approach to capture the correlation between point clouds across different time dimensions, as well as the correlation between adjacent point clouds. This technical solution employs two-dimensional convolution, discarding the original channel dimension and boldly utilizing the time dimension as the channel. This effectively filters out noise, captures the correlation between adjacent point clouds and point clouds across different time dimensions, and aids in reconstruction. Figure 4As shown in the red box in the qualitative comparison, at the brush area, due to the capture of long-range correlation of point clouds and the feature enhancement of high-frequency components, the fine structures such as the brush can be reconstructed quite well. No singularities or discontinuous depth information are generated at the brush holder and background areas. Four different signal-to-noise ratios in the "Reindeer" scene were also selected and displayed to compare the reconstruction effects of different methods.

[0066] To verify the performance of the algorithm in a real-world scenario, this application's technical solution used a rag doll as the main component in a real-world experiment. A fiber pulsed laser was used, emitting pulses with a repetition frequency of 20 kHz and a wavelength of 1064 nm, a pulse width of 1 ns, a spot emission angle of 25°, and a peak pulse power of 500 mW. The imaging lens used had a focal length of 8 mm, an effective aperture of 5 mm, and a field of view of 23°. The results after reconstruction from real experimental data are shown below. In the real-world experiment, this application's technical solution compared the performance of the Wave-Mamba network with the neural networks used in the existing technologies of Peng and Zhao in the task of reconstructing depth maps from single-photon point cloud data. Experimental results show that the Mamba network outperforms the methods used by Peng and Zhao in terms of detail reconstruction, edge preservation, and overall structural consistency. Specifically, the improvements include: First, in terms of detail reconstruction, the Wave-Mamba network can more accurately capture the smoother lines of the "rag doll" surface. Regarding the doll's limbs, the reconstruction of the "hands" and the "horns" on the head is more complete, with no missing pixels. The reconstructed depth map is clearer in high-frequency detail areas, while the methods of Peng et al. and Zhao et al. tend to be blurry or lack detail in these areas. Furthermore, in terms of overall structural consistency, the Wave-Mamba network can model global features, ensuring that the reconstructed depth map maintains consistency with the shape of the real object, guaranteeing the coherence of the overall structure of the reconstructed depth map. In contrast, the methods of Peng et al. and Zhao et al. have shortcomings in low-frequency global structure modeling; the "rag doll" may have some missing areas in its overall structure, and some structures may have singular points. In real-world experiments, the Wave-Mamba network demonstrates stronger detail representation capabilities, superior edge preservation, and more stable overall structural reconstruction capabilities, making the final reconstructed depth map closer to the shape of the real object, further validating its superiority in single-photon point cloud data reconstruction tasks.

[0067] In summary, the model proposed in this invention exhibits excellent convergence and detection performance across all metrics, demonstrating its practical application value and technical advantages in the task of reconstructing distance information from single-photon point cloud data. Specifically:

[0068] To address the common problems of high sparsity and significant noise interference in the reconstruction of single-photon point cloud data, this paper proposes a depth reconstruction method based on state-space modeling, which can efficiently reconstruct high-quality distance images from input 3D point cloud data. The proposed method constructs a two-dimensional state-space modeling structure (2D-SSM), effectively capturing long-distance spatial dependencies, structural relationships between cross-regional targets, and local and global contextual information in the point cloud, thereby improving the stability and accuracy of depth estimation. Furthermore, this paper introduces the inherent multi-scale analysis and noise suppression capabilities of wavelet transform, decomposing features into low-frequency and high-frequency components in the frequency domain, effectively separating global structure from edge details, and reducing the impact of noise on distance reconstruction at its source. Based on this, a dual-path attention feature enhancement module, compatible with wavelet characteristics, is designed to strengthen the overall scene structure modeling capability in the low-frequency channel and the ability to express edge details and local textures in the high-frequency channel. This module introduces a structure that integrates spatial and channel attention for low-frequency modeling, and connects a lightweight and efficient attention module similar to the self-attention module in the Transformer core component for high-frequency enhancement. This not only enhances the model's ability to perceive structural abrupt changes and weak boundaries, but also demonstrates excellent distance map reconstruction capabilities under low photon number conditions in simulation scenarios with varying signal-to-noise ratios. Ablation experiments further prove the key role of each module in improving overall network performance. Furthermore, when applied to real-world indoor scene data, this module still reconstructs depth results that conform to human visual perception. Compared with various existing deep learning methods, the proposed method exhibits superior overall performance in edge reconstruction quality, overall structure preservation, and noise resistance, demonstrating its practical application potential and broad applicability in single-photon point cloud distance reconstruction tasks.

[0069] The meanings of the English abbreviations used in this application are as follows:

[0070]

[0071]

[0072] Finally, it should be noted that the present invention is not limited to the above embodiments, and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement, characterized in that... The method includes the following steps: Step S1: Create a training dataset and simulate the generation of single-photon detection data based on the NYUv2 dataset; Step S2: Construct a Mamba reconstruction network based on dual-path wavelet feature enhancement. The network includes an encoding module, a long-range dependency modeling module, and a decoding module. The encoding module is a residual convolutional shallow feature extraction module. The long-range dependency modeling module is a cascaded dual-path feature reconstruction module with wavelet decomposition. The decoding module is a residual convolutional feature integration module. Step S3: Train the Mamba reconstruction network using the training dataset; Step S4: Use the trained Mamba reconstruction network to perform reconstruction tests on the test set and real point cloud data to obtain the depth map reconstruction results of single-photon point clouds.

2. The method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement according to claim 1, characterized in that, In step S1, the process of simulating and generating single-photon detection data includes: extracting the depth map and grayscale map from the RGB-D image pair of NYUv2, interpolating and completing the depth map, converting it into a ToF map according to the laser time-of-flight principle, quantizing it into 1024 time bins, generating the signal photon number by combining the illumination intensity and distance attenuation model, adding background noise based on the grayscale map, and obtaining a sparse three-dimensional time detection tensor through non-homogeneous Poisson sampling and point spread function convolution.

3. The method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement according to claim 1, characterized in that, In step S2, the dual-path feature reconstruction module includes a low-frequency path and a high-frequency path; the low-frequency path enhances the global structural continuity through spatial and channel attention; the high-frequency path captures edge details and local texture changes through a lightweight self-attention module.

4. The method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement according to claim 3, characterized in that, The long-distance dependency modeling module adopts a layer-by-layer stacked DFRM structure, with each layer of DFRM structure independently completing a round of high-frequency-low-frequency feature separation and structural enhancement. The DFRM structure does not require an external compensation path. It achieves bottom-up feature propagation through frequency decoupling and attention feature enhancement modules. The output features of each layer inherit the structural information of the previous layer and enhance new high and low frequency information.

5. The method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement according to claim 1, characterized in that, In step S3, the loss function used in the training process is: L total =MAE range +λ1(1-SSIM range )+λ2(1-Pixels range )+λ3(1-TV range ); Among them, λ1=0.2, λ2=0.025, λ3=0.3; MAE range SSIM is the mean absolute error. range Pixels is a structural similarity index. range For pixel loss, TV range This represents the total variation loss.

6. The method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement according to claim 5, characterized in that, The MAE range The calculation formula is: Where Z is the target image, Y is the network-reconstructed image, and n is the number of pixels in the image.

7. The method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement according to claim 5, characterized in that, The SSIM range The calculation formula is: Where the mean of Z and Y is μ z μ y The variances of Z and Y are respectively The product of σ zy Let the covariance be c1, where c1 and c2 are both constants.

8. The method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement according to claim 5, characterized in that, The Pixels range The calculation formula is: Where W is the width of the reconstructed image and H is the height.

9. The method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement according to claim 5, characterized in that, The TV range The calculation formula is: Where Y represents the image reconstructed by the network, and i and j represent the position of a certain pixel value.

10. The method for Mamba single-photon 3D reconstruction network based on dual-path wavelet feature enhancement according to claim 1, characterized in that, In step S2, the input data size of the Mamba reconstruction network is B×H×W×T, where B is the batch size, H and W are the input sizes, and T is the time dimension. The network uses 2D convolution, and the time dimension T is used as the channel of the two-dimensional convolution. In step S4, the real point cloud data acquisition device includes a fiber pulsed laser with a repetition frequency of 20kHz, a wavelength of 1064nm, a pulse width of 1ns, and a peak power of 500mW, and is used with an imaging lens with a focal length of 8mm, an effective aperture of 5mm, and a field of view of 23°.

Images