Lightweight spad neural network reconstruction method with depth-intensity joint optimization

Abstract

The application discloses a kind of depth-intensity joint optimization light weight SPAD neural network reconstruction methods, belong to single-photon imaging and depth learning technical field, including: based on wavelet transform to the original histogram data of SPAD array is carried out multistage wavelet decomposition, obtain compressed time-frequency domain data;Utilize multiscale superpixel to carry out non-maximum suppression, and obtain estimated depth map by voting mechanism;Local histogram is extracted from compressed time-frequency domain data, and sum along time dimension obtains intensity map;The estimated depth map is expanded, and expanded depth map is obtained;The double-branch complementary depth reconstruction network including depth branch, edge branch and perception interaction module is constructed;Real depth map and real edge gradient map are used to guide network, and training is carried out using joint loss function, to realize light weight SPAD neural network reconstruction.The method significantly improves the depth reconstruction accuracy under low illumination while greatly reducing the amount of calculation.

Description

Technical Field

[0001] This invention belongs to the field of single-photon imaging and deep learning technology, specifically relating to a lightweight SPAD neural network reconstruction method with joint depth-intensity optimization. Background Technology

[0002] LiDAR (Light Detection and Ranging) systems use single-photon avalanche diodes (SPADs) as their core detectors. Their working principle can be summarized into four key stages: emission, reflection, reception, and processing. The system first emits nanosecond-level pulsed laser beams (typical wavelengths of 905 nm or 1550 nm). When the laser encounters a target object, it generates a reflected signal. Leveraging its single-photon sensitivity, the SPAD detector can capture extremely weak photon echoes and convert these individual photon events into detectable electrical signals through the avalanche effect.

[0003] Currently, in terms of signal processing, the system generates a time histogram by accumulating reflection data from multiple laser pulses. The horizontal axis of this histogram represents time bins, each corresponding to a specific distance interval; the vertical axis records photon counts. The reflected signal from the target object forms a significant peak in the corresponding time bin, while ambient light noise exhibits a uniform distribution. By identifying the peak positions in this time histogram, the system can calculate the target distance with picosecond-level time resolution, achieving a typical ranging accuracy down to the centimeter level.

[0004] With the development of deep learning technology, deep reconstruction neural networks are increasingly being applied to LiDAR systems for target detection. However, large SPAD arrays make it extremely difficult to deploy deep reconstruction networks on edge devices, hindering their application. Existing methods directly use histogram data as network input. When reconstructing a single 256×256 depth map, the amount of data input to the network can reach 256×256×1024×8 bits = 64MB. If the SPAD array size increases, the time-bin length increases from 1024 bins to more, or the representation range of a single time-bin increases from 8 bits to higher precision, the amount of input data will further increase. As network inference progresses, the amount of intermediate result data will further increase. Existing methods often cannot directly reconstruct a 256×256 depth map, but instead divide it into 64×64 blocks for depth reconstruction, with each block requiring 3-5GB of GPU memory.

[0005] Therefore, existing methods cannot perform real-time data processing on mobile devices with limited computing and storage resources. Furthermore, under low-light conditions, SPAD detectors are susceptible to ambient light, making accurate depth reconstruction difficult. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a lightweight SPAD neural network reconstruction method with joint depth-intensity optimization. The technical problem to be solved by this invention is achieved through the following technical solution: This invention proposes a lightweight SPAD neural network reconstruction method with joint depth-intensity optimization, comprising: S1: Perform multi-level wavelet decomposition on the original histogram data of the SPAD array based on wavelet transform to obtain compressed time-frequency domain data; S2: Non-maximum suppression is performed on the compressed time-frequency domain data using multi-scale superpixels, and a predicted depth map is obtained through a voting mechanism; S3: Extract local histograms from the compressed time-frequency domain data based on the estimated depth map, and sum them along the time dimension to obtain the intensity map; at the same time, expand the estimated depth map to obtain the expanded depth map; S4: Construct a dual-branch complementary deep reconstruction network; the network includes parallel deep branches and edge branches, as well as a perceptual interaction module located between the two branches; wherein, the deep branch takes the augmented depth map as input and obtains the depth map through feature extraction and depth reconstruction; the edge branch takes the local histogram and intensity map as input and obtains the gradient map through feature extraction and depth reconstruction; the perceptual interaction module transmits gradient information between the two branches through a dual-branch interaction mechanism to assist in depth reconstruction; S5: The network is guided by real depth maps and real edge gradient maps, and trained using a joint loss function to achieve lightweight SPAD neural network reconstruction.

[0007] The beneficial effects of this invention are: The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization provided by this invention first processes the original histogram data of the SPAD array to construct the network's input data. This includes performing multi-level wavelet decomposition on the original histogram data of the SPAD array using wavelet transform to obtain compressed time-frequency domain data; performing non-maximum suppression on the compressed time-frequency domain data using multi-scale superpixels and obtaining a predicted depth map through a voting mechanism; extracting local histograms from the compressed time-frequency domain data based on the predicted depth map and summing them along the time dimension to obtain an intensity map; simultaneously, expanding the predicted depth map to obtain an expanded depth map; then constructing a bi-branch complementary depth reconstruction network including parallel depth and edge branches, and a perceptual interaction module located between the two branches; finally, based on the constructed input data, guiding the network using the real depth map and the real edge gradient map, and training it using a joint loss function to achieve lightweight SPAD neural network reconstruction. This method achieves efficient noise reduction and compression by utilizing the time-frequency domain information of SPAD histogram data through wavelet transform and multi-scale nonmaximum suppression voting, significantly reducing the size of SPAD histogram data and the resources required for data transmission, storage, and computation. This is beneficial for real-time data processing on mobile devices with limited computing and storage resources. On the other hand, it uses a lightweight bi-branch interactive network for depth reconstruction and uses a perceptual interactive module to distinguish gradient information from invalid noise. At the same time, it transmits gradient information between the two branches, helping the network to perform high-precision depth reconstruction even at low photon count levels.

[0008] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0009] Figure 1 A schematic flowchart of the lightweight SPAD neural network reconstruction method with joint depth-intensity optimization provided in this embodiment of the invention; Figure 2 This is an overall framework diagram of the dual-branch complementary deep reconstruction network provided in an embodiment of the present invention; Figure 3 This is a network structure diagram of the ResDoubleConv layer, downsampling, and upsampling provided in an embodiment of the present invention; Figure 4 This is a network structure diagram of the perception and interaction module provided in an embodiment of the present invention. Detailed Implementation

[0010] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of the lightweight SPAD neural network reconstruction method based on the depth-intensity joint optimization proposed in this invention is provided in conjunction with the accompanying drawings and specific embodiments.

[0011] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0012] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes that element.

[0013] For details, please see Figure 1 , Figure 1 This is a flowchart illustrating the lightweight SPAD neural network reconstruction method with joint depth-intensity optimization provided in this embodiment of the invention. The method mainly includes the following steps: S1: Perform multi-level wavelet decomposition on the original histogram data of the SPAD array based on wavelet transform to obtain compressed time-frequency domain data.

[0014] Histograms are used to statistically analyze the time delay distribution of laser pulses from LiDAR, from emission to reception, reflecting the depth and intensity information of the target object. The horizontal axis of the histogram represents time bins, each corresponding to a specific time interval; the vertical axis records the photon count. Therefore, the histogram data of a single pixel can be viewed as one-dimensional discrete data, where each data point corresponds to the number of photons in the current time bin, which can be represented as... ,in, Indicates the first z The number of photons in each timebox Z This indicates the total number of timeboxes.

[0015] Optionally, in this embodiment, step S1 includes the following sub-steps S1.1-S1.3.

[0016] S1.1: Select the db6 wavelet basis as the basis function for the wavelet transform.

[0017] Specifically, the db6 wavelet is a type of Daubechies wavelet, belonging to the orthogonal compactly-supported wavelet family. The "6" indicates that it has 6 vanishing moments, meaning it can completely represent polynomials from order 0 to 5. Its corresponding low-pass filter coefficient length is 12.

[0018] S1.2: Based on the selected basis functions, perform a 3-level wavelet decomposition on the original histogram data of the SPAD array in the time dimension.

[0019] In practice, the histogram data typically includes 1024 data elements. In this embodiment, for the sake of simplicity, it is assumed that the histogram data is six bits, namely (11, 12, 13, 14, 15, 16), and that the selected wavelet low-frequency filter coefficients are (2, 3, 4), i.e. , , Construct the first-level coding matrix based on the low-frequency filter coefficients: ; in, Indicates the first level of the coding matrix. k Line number i The column element values; the number of rows in this first-level coding matrix is ​​equal to the number of time bins in the histogram corresponding to the histogram data, which is equal to the number of data points in the histogram data. Therefore, the constructed first-level coding matrix is: ; In this first-level coding matrix, each column's data is the data from the previous column shifted down by two units. Specifically, the first-level coding matrix consists of low-frequency filter coefficients and 0s. The first column consists of the last two digits of the low-frequency filter coefficients and 0s. The second column consists of wavelet low-frequency filter coefficients shifted down by two units from the first column. The third column consists of wavelet low-frequency filter coefficients shifted down by two units from the second column, and so on, until the column preceding the all-zero column. That is, the matrix does not include columns containing all zeros.

[0020] Since the high-frequency components of laser pulse point cloud data typically correspond to noise, only the low-frequency components obtained after wavelet transform are retained. For a given length of... L The low-frequency component of the first-order wavelet decomposition of one-dimensional discrete data is the result of convolving the low-frequency filter coefficients with the one-dimensional discrete data and then downsampling by a factor of two. This result is equivalent to the result obtained by convolution with a stride of two, and can be converted into multiplying the coding matrix with the one-dimensional discrete data.

[0021] Continuing with the assumed data above, the data after first-order wavelet decomposition is as follows: .

[0022] In this embodiment, the histogram data is decomposed into multiple levels. Therefore, after obtaining the discrete data after the first-level wavelet decomposition, the second-level wavelet decomposition is performed. First, a second-level coding matrix for the second-level wavelet decomposition needs to be constructed. The form of the second-level coding matrix is ​​the same as that of the first-level coding matrix, but the number of rows and columns are different. The number of rows in the second-level coding matrix is ​​equal to the number of bits in the discrete data after the first-level wavelet decomposition. Taking the above-mentioned hypothetical data as an example, that is, the discrete data after the first-level wavelet decomposition is (81, 119, 137, 32), and the number of bits is 4, then the number of rows in the second-level coding matrix is ​​4. At the same time, according to formula (1), the second-level coding matrix can be obtained as follows: ; The calculation process of second-order wavelet decomposition can be expressed as follows: .

[0023] Then, a three-level wavelet decomposition is performed. First, a first-level coding matrix, a second-level coding matrix, and a third-level coding matrix are constructed. Then, the first-level coding matrix, the second-level coding matrix, and the third-level coding matrix are multiplied together to obtain a total coding matrix. Multiplying the histogram data with this total coding matrix yields the time-frequency data of the current pixel after multi-level wavelet decomposition.

[0024] Continuing with the assumed data above, the third-level wavelet decomposition can be expressed as: .

[0025] By performing the above steps on the histogram data of each pixel, we can obtain the low-frequency data after multi-level wavelet decomposition of each pixel, which is still a set of one-dimensional discrete data.

[0026] For example, if the original histogram data includes 1024 data elements and the low-frequency filter coefficients include 12 data elements, then the data after three-level wavelet decomposition is discrete data including 137 data elements.

[0027] S1.3: Obtain the low-frequency coefficients after decomposition to obtain compressed time-frequency domain data.

[0028] Specifically, the decomposed low-frequency coefficients and high-frequency coefficients can be used as coded data that retains the effective representation in the time-frequency domain. Since the high-frequency coefficients are mainly noise, only the low-frequency coefficients are retained as the final compressed time-frequency domain data.

[0029] S2: Non-maximum suppression is performed on the compressed time-frequency domain data using multi-scale superpixels, and a predicted depth map is obtained through a voting mechanism.

[0030] Optionally, in this embodiment, step S2 includes the following sub-steps S2.1-S2.4.

[0031] S2.1: For the compressed time-frequency domain data, generate three sets of superpixels with the same size in the time dimension and different sizes in the spatial dimension, centered on each bin of each pixel.

[0032] Optionally, the three sets of superpixels generated in this embodiment can have a size of 5 in the time dimension and a size of 3×3×5, 9×9×5, and 11×11×5 in the spatial dimension.

[0033] For example, on the wavelet decomposition data of shape (256,256,137), a superpixel of shape (3,3,5) is constructed with each time bin of the current pixel as the center. If the superpixel exceeds the range when it is constructed, zero padding is performed to ensure that each bin of each pixel has its corresponding complete superpixel range.

[0034] S2.2: Accumulate the data within the three sets of superpixels respectively, find the extreme point index in the time dimension, and obtain three candidate depth values ​​corresponding to each pixel.

[0035] Specifically, in this embodiment, the data within the superpixel range corresponding to each bin of each pixel in S2.1 is accumulated, that is, each bin of each pixel corresponds to an accumulated sum, and then the accumulated sums are used to form a new signal enhancement histogram. Following the above steps, three groups of superpixels with different spatial sizes are processed to obtain three groups of signal enhancement histograms. Then, for each pixel in the three groups of histograms, the index of the maximum value point is found along the time bin dimension. Each pixel corresponds to an index, and three maximum value index maps are obtained. At this time, each pixel corresponds to three maximum value indices, i.e., candidate depth values.

[0036] S2.3: Set an interval threshold Calculate the difference between each pair of the three candidate depth values.

[0037] Specifically, the embodiment selected Assume the three candidate depth values ​​for the current pixel (corresponding to superpixel space sizes from smallest to largest) are: The differences between them are as follows: .

[0038] S2.4: If the difference between at least two candidate depth values ​​is within the interval threshold If the range is within the specified range, the candidate depth value with the smallest corresponding superpixel spatial size is selected as the estimated depth value; if the difference between any two of the three candidate depth values ​​exceeds the interval threshold... If the range is defined, the candidate depth value with the largest corresponding superpixel spatial size is selected as the estimated depth value, thus obtaining the estimated depth map.

[0039] Specifically, if At least one of them is less than or equal to 8, for example , then because The corresponding superpixel space size is smaller and contains more detail, therefore, [the following is chosen]: As a predicted depth value.

[0040] like If all three candidate depth values ​​are greater than 8, it indicates that all three values ​​may be noise values. A larger superpixel space size corresponds to the lowest probability of noise, therefore, it is chosen. As a predicted depth value.

[0041] The above operation is performed on each pixel to obtain the corresponding estimated depth value, thus forming an estimated depth map.

[0042] S3: Extract local histograms from the compressed time-frequency domain data based on the estimated depth map, and sum them along the time dimension to obtain the intensity map; at the same time, expand the estimated depth map to obtain the expanded depth map.

[0043] Optionally, in this embodiment, step S3 includes the following sub-steps S3.1-S3.3.

[0044] S3.1: Using the estimated depth value of each pixel in the estimated depth map as the center, truncate a fixed width ± [value missing] along the time dimension. N Data from each bin is used to obtain several local histograms, which serve as inputs to preserve waveform information. N It is an integer.

[0045] Specifically, the fixed width selected in this embodiment If the estimated depth value of the current pixel is 28, then the index group is {25,26,27,28,29,30,31}. The data at the corresponding index is then extracted from the histogram after wavelet decomposition using this index group as the local histogram data for the current pixel. The shape of the local histogram after extraction for each pixel is (256,256,7). Furthermore, if the estimated depth value is too large or too small, causing the index values ​​in the index group to be less than 0 or greater than 136, then out-of-range index values ​​are limited to 0 or 136.

[0046] S3.2: The histograms after wavelet decomposition are summed along the time dimension to obtain a two-dimensional intensity map, which is used as input to retain intensity information.

[0047] Specifically, the histogram after wavelet decomposition has a shape of (256, 256, 137). Summing the 137 values ​​of each pixel yields a 2D intensity image with a shape of (256, 256).

[0048] S3.3: Expand the estimated depth map in the time dimension, making its dimension similar to the local histogram. Figure 1 This process yields an expanded depth map, which serves as input to preserve prior depth information.

[0049] Specifically, in order to make the estimated depth value match the local histogram in dimensions, the index group in S3.1 is used to replace the estimated depth value of the current pixel value to obtain an expanded depth map with the same shape (256,256,7) as the local histogram.

[0050] At this point, we have obtained the network's input data, namely the augmented depth map, local histogram, and intensity map.

[0051] In constructing the network input data, this implementation achieves efficient noise reduction and compression by using wavelet transform and multi-scale nonmaximum suppression voting, and utilizes the time-frequency domain information of SPAD histogram data. This significantly reduces the size of SPAD histogram data, reduces the resources required for data transmission, storage and computation, and is beneficial for real-time data processing on mobile devices with limited computing and storage resources.

[0052] S4: Construct a dual-branch complementary deep reconstruction network; the network includes parallel deep branches and edge branches, as well as a perceptual interaction module located between the two branches.

[0053] For details, please see Figure 2 , Figure 2 This is an overall framework diagram of the dual-branch complementary deep reconstruction network provided in an embodiment of the present invention. The deep branch UNet takes the expanded depth map as input and obtains the depth map through feature extraction and depth reconstruction; the edge branch UNet takes the local histogram and intensity map as input and obtains the gradient map through feature extraction and depth reconstruction; the perception interaction module transmits gradient information between the two branches through a dual-branch interaction mechanism to assist in depth reconstruction.

[0054] Optionally, in this embodiment, such as Figure 2 As shown in the right half, the processing flow of the depth branch for the input augmented depth map is as follows: The augmented depth map first extracts shallow features through a ResDoubleConv (residual double convolution) layer, then performs depth reconstruction through three downsampling and three upsampling operations, and finally integrates the channels through a convolutional layer to output the depth map.

[0055] Correspondingly, such as Figure 2As shown in the left half, the processing flow of the edge branch for the input local histogram and intensity map is as follows: The local histogram and intensity image are each processed through a ResDoubleConv layer to extract shallow features. The features are then summed and subjected to three downsampling and three upsampling operations for depth reconstruction. Finally, a convolutional layer is used to integrate the channels and output the gradient map.

[0056] For further details, please see Figure 3 , Figure 3 This diagram illustrates the network structure of the ResDoubleConv layer, downsampling, and upsampling provided in this embodiment of the invention. The ResDoubleConv layer in this embodiment, i.e., the residual double convolutional layer, includes a backbone network. and residual branch The output is the sum of the results from the backbone network and the residual branches; where, For input features, The kernel size is Convolutional layers, The kernel size is Convolutional layers, For batch normalization layer, This is the activation layer.

[0057] Please continue reading Figure 3 The upsampling process involves bilinear interpolation upsampling followed by... Finally, it goes through a ResDoubleConv layer. After downsampling to max pooling, it goes through a ResDoubleConv layer.

[0058] For further details, please continue to see Figure 2 In the network constructed in this embodiment, a perceptual interaction module is also set between the depth branch and the edge branch; the perceptual interaction module includes a gradient perception module (GAB) and a stitching and fusion module (CFB); wherein, the gradient perception module (GAB) is responsible for extracting the edge gradient information of the intermediate features of the depth and passing it to the edge branch; the stitching and fusion module (CFB) is responsible for passing the intermediate features of the edge to the depth branch; wherein, Please see Figure 4 , Figure 4 This is a network structure diagram of the perception interaction module provided in an embodiment of the present invention. In the gradient perception module GAB, the intermediate depth features are calculated using the Sobel operator. x , y The gradient in the direction is then concatenated with the edge middle features, and then after passing through two convolutional layers for channel dimensionality reduction, it is added to the edge middle features to realize the interaction between the depth branch and the edge branch.

[0059] Specifically, the Sobel operator in The kernel parameters in the direction are: ; .

[0060] Assuming the depth intermediate feature is Edge middle features are The process expression for this interaction is: ; in, It is a feature concatenation operation. This indicates that the corresponding kernel parameters are used as convolution kernel weights for the input. Perform a convolution operation with a stride of 1 pixel by pixel.

[0061] In the stitching and fusion module CFB, the edge intermediate features and the depth intermediate features are stitched together, and then after channel dimensionality reduction through two convolutional layers, they are added to the depth intermediate features to realize the interaction from the edge branch to the depth branch.

[0062] Specifically, assuming the depth intermediate feature is The middle features of the edge are The process expression for this interaction is: .

[0063] This invention utilizes a lightweight bi-branch interactive network for depth reconstruction and uses a perceptual interaction module to distinguish gradient information from invalid noise. At the same time, gradient information is transferred between the two branches, which helps the network to perform high-precision depth reconstruction even at low photon count levels.

[0064] S5: The network is guided by real depth maps and real edge gradient maps, and trained using a joint loss function to achieve lightweight SPAD neural network reconstruction.

[0065] Optionally, in this embodiment, step S5 includes the following sub-steps S5.1-S5.2.

[0066] S5.1: Calculate the loss using the real depth map and the depth map output by the depth branch to gradually fit the real depth of the depth branch, thereby guiding the training of the depth branch.

[0067] S5.2: Obtain the true depth map using the Sobel operator. x , y The true edge gradient map in the direction is obtained, and then the loss is calculated by comparing the true edge gradient map with the gradient map output by the edge branch to gradually fit the true edge gradient of the edge branch, so as to guide the training of the edge branch.

[0068] It should be noted that the true edge gradient map includes the Sobel operator in x , y Two images obtained in the direction; correspondingly, the gradient map output by the edge branch also includes... x , y Two images in different directions.

[0069] Specifically, in steps S5.1 and S5.2, the joint loss function used to calculate the loss is a joint loss function composed of the L1 norm and the MS-SSIM loss function, denoted as... Loss The expression is: ; In the formula, It is the L1 norm, and its expression is as follows: ; in, To predict the image, For the target image, The height of the image. Image width, To predict image coordinates Pixel value at that location, The target image coordinates are Pixel value at; In the formula, The MS-SSIM loss function is expressed as follows: ; in, To predict edge gradients, For the target edge gradient, The expression is as follows: ; in, For brightness index, For contrast ratio, For structural indicators, The weight of the highest-scale brightness term. For the first j Contrast weights at each scale For the first j Structural weights at each scale M For the scale number, 0 < j ≤ M。

[0070] The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization provided by this invention first processes the original histogram data of the SPAD array to construct the network's input data. This includes performing multi-level wavelet decomposition on the original histogram data of the SPAD array using wavelet transform to obtain compressed time-frequency domain data; performing non-maximum suppression on the compressed time-frequency domain data using multi-scale superpixels and obtaining a predicted depth map through a voting mechanism; extracting local histograms from the compressed time-frequency domain data based on the predicted depth map and summing them along the time dimension to obtain an intensity map; simultaneously, expanding the predicted depth map to obtain an expanded depth map; then constructing a bi-branch complementary depth reconstruction network including parallel depth and edge branches, and a perceptual interaction module located between the two branches; finally, based on the constructed input data, guiding the network using the real depth map and the real edge gradient map, and training it using a joint loss function to achieve lightweight SPAD neural network reconstruction. This method achieves efficient noise reduction and compression by utilizing the time-frequency domain information of SPAD histogram data through wavelet transform and multi-scale nonmaximum suppression voting, significantly reducing the size of SPAD histogram data and the resources required for data transmission, storage, and computation. This is beneficial for real-time data processing on mobile devices with limited computing and storage resources. On the other hand, it uses a lightweight bi-branch interactive network for depth reconstruction and uses a perceptual interactive module to distinguish gradient information from invalid noise. At the same time, it transmits gradient information between the two branches, helping the network to perform high-precision depth reconstruction even at low photon count levels.

[0071] The Middlebury dataset, a representative dataset in the field, is selected below to verify the effectiveness of the method of the present invention. Specifically, simulated histogram data with a ratio of "signal photon count: background photon count" was generated based on eight scenes from the Middlebury dataset, including [10:2, 5:2, 2:2, 10:10, 5:10, 2:10, 10:50, 5:50, 2:50], to verify the deep reconstruction performance of the method of the present invention. Please refer to Table 1, which shows the performance comparison of the method provided by the present invention with existing neural network methods in deep reconstruction results.

[0072] Table 1. Performance comparison results between the method provided by this invention and existing methods.

[0073] Among them, Lindell, UNet, Nonlocal, SSPINet, and SKFNet are all existing methods, and their data sources are: Cai, R.; Guo, H.; Li, X.; Che, J.; He, J.; Liu, H.; Yang, L.; Guo, Y.; Pu, M.; Luo, X. Sub-diffraction-limited single-photon 3D imaging based on domainfeatures extraction network at kilometer-scale distance. Opt. Laser Technol. 2025, 181 Pt B, 111868. Proposed is the method proposed in this invention.

[0074] The results show that the method of the present invention has lower average depth reconstruction error than other methods while having lower computational complexity, thus achieving better depth reconstruction performance.

[0075] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A lightweight SPAD neural network reconstruction method with joint depth-intensity optimization, characterized in that, include: S1: Perform multi-level wavelet decomposition on the original histogram data of the SPAD array based on wavelet transform to obtain compressed time-frequency domain data; S2: Non-maximum suppression is performed on the compressed time-frequency domain data using multi-scale superpixels, and a predicted depth map is obtained through a voting mechanism; S3: Extract local histograms from the compressed time-frequency domain data based on the estimated depth map, and sum them along the time dimension to obtain an intensity map; simultaneously, expand the estimated depth map to obtain an expanded depth map; S4: Construct a dual-branch complementary deep reconstruction network; the network includes parallel deep branches and edge branches, and a perceptual interaction module located between the two branches; wherein, the deep branch takes the augmented depth map as input and obtains a depth map through feature extraction and depth reconstruction; the edge branch takes the local histogram and the intensity map as input and obtains a gradient map through feature extraction and depth reconstruction; the perceptual interaction module transmits gradient information between the two branches through a dual-branch interaction mechanism to assist in depth reconstruction; S5: The network is guided using the real depth map and the real edge gradient map, and trained using the joint loss function to achieve lightweight SPAD neural network reconstruction.

2. The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization according to claim 1, characterized in that, S1 includes: S1.1: Select the db6 wavelet basis as the basis function for the wavelet transform; S1.2: Based on the basis functions, perform a 3-level wavelet decomposition on the original histogram data of the SPAD array in the time dimension; S1.3: Obtain the low-frequency coefficients after decomposition to obtain compressed time-frequency domain data.

3. The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization according to claim 1, characterized in that, S2 includes: S2.1: For the compressed time-frequency domain data, three sets of superpixels with the same size in the time dimension and different sizes in the spatial dimension are generated, centered on each bin of each pixel; where bin represents time bin; S2.2: Accumulate the data within the three groups of superpixels respectively, find the extreme point index in the time dimension, and obtain three candidate depth values ​​corresponding to each pixel; S2.3: Set an interval threshold and calculate the difference between each pair of the three candidate depth values; S2.4: If the difference between at least two candidate depth values ​​is within the range of the interval threshold, then the candidate depth value with the smallest corresponding superpixel space size is selected as the estimated depth value; if the difference between any two of the three candidate depth values ​​exceeds the range of the interval threshold, then the candidate depth value with the largest corresponding superpixel space size is selected as the estimated depth value, thereby obtaining the estimated depth map.

4. The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization according to claim 3, characterized in that, In S2.1, the three sets of superpixels generated each have a size of 5 in the time dimension and sizes of 3×3×5, 9×9×5 and 11×11×5 in the spatial dimension, respectively.

5. The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization according to claim 3, characterized in that, S3 includes: S3.1: Taking the estimated depth value of each pixel in the estimated depth map as the center, truncate a fixed width ± [value] along the time dimension. N Data from each bin is used to obtain several local histograms, which serve as inputs to preserve waveform information. N It is an integer; S3.2: The local histograms are summed along the time dimension to obtain a two-dimensional intensity map, which is used as the input to retain intensity information; S3.3: Expand the estimated depth map in the time dimension so that its dimension is consistent with the local histogram to obtain an expanded depth map, which serves as the input for preserving the prior depth information.

6. The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization according to claim 1, characterized in that, In the network constructed in S4, the processing flow of the deep branch on the input augmented depth map is as follows: The augmented depth map first extracts shallow features through a ResDoubleConv layer, then performs depth reconstruction through three downsampling and three upsampling operations, and finally integrates the channels through a convolutional layer to output the depth map. The processing flow of the edge branch on the input local histogram and intensity map is as follows: The local histogram and intensity image are each processed through a ResDoubleConv layer to extract shallow features. The features are then added together and subjected to three downsampling and three upsampling operations for depth reconstruction. Finally, the channels are integrated through a convolutional layer to output the gradient map.

7. The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization according to claim 6, characterized in that, The ResDoubleConv layer includes a backbone network. and residual branch The output is the sum of the results of the backbone network and the residual branches; in, For input features, The kernel size is Convolutional layers, The kernel size is Convolutional layers, For batch normalization layer, This is the activation layer.

8. The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization according to claim 1, characterized in that, In the network constructed by S4, the perceptual interaction module includes a gradient perception module and a stitching and fusion module; the gradient perception module is responsible for extracting the edge gradient information of the intermediate depth features and passing it to the edge branch; the stitching and fusion module is responsible for passing the intermediate edge features to the depth branch; wherein, In the gradient sensing module, the intermediate depth features are calculated using the Sobel operator. x , y The gradient in the direction is then concatenated with the edge middle features, and then after passing through two convolutional layers for channel dimensionality reduction, it is added to the edge middle features to realize the interaction from the depth branch to the edge branch. In the stitching and fusion module, the edge intermediate features and the depth intermediate features are stitched together, and then after channel dimensionality reduction through two convolutional layers, they are added to the depth intermediate features to realize the interaction from the edge branch to the depth branch.

9. The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization according to claim 1, characterized in that, S5 includes: S5.1: Calculate the loss using the real depth map and the depth map output by the depth branch to gradually fit the real depth of the depth branch, thereby guiding the training of the depth branch. S5.2: Obtain the true depth map using the Sobel operator. x , y The true edge gradient map in the direction is obtained, and then the loss is calculated by comparing the true edge gradient map with the gradient map output by the edge branch to gradually fit the true edge gradient of the edge branch, so as to guide the training of the edge branch.

10. The lightweight SPAD neural network reconstruction method with joint depth-intensity optimization according to claim 9, characterized in that, In S5.1 and S5.2, the joint loss function used to calculate the loss is a joint loss function composed of the L1 norm and the MS-SSIM loss function, denoted as... Loss The expression is: ； In the formula, It is the L1 norm, and its expression is as follows: ； in, To predict the image, For the target image, The height of the image. Image width, To predict image coordinates Pixel value at that location, The target image coordinates are Pixel value at; In the formula, The MS-SSIM loss function is expressed as follows: ； in, To predict edge gradients, For the target edge gradient, The expression is as follows: ； in, For brightness index, For contrast ratio, For structural indicators, The weight of the highest-scale brightness term. For the first j Contrast weights at each scale For the first j Structural weights at each scale M For the scale number, 0 < j ≤ M。

Images