A single-photon depth reconstruction method and system based on time-domain feature aggregation

Abstract

This application provides a method and system for single-photon depth reconstruction based on temporal feature aggregation, belonging to the field of computer vision and image processing technology. The method includes: preprocessing single-photon 3D data using asymmetric 3D convolution, where the convolution downsamples along the time dimension while maintaining spatial resolution, and the kernel weights follow a pre-defined non-uniform distribution matching the prior temporal distribution of the signal photons; inputting the preprocessed features into a multi-scale encoder-decoder network for feature extraction and fusion; during decoding, employing a global feature aggregation module based on temporal distribution similarity to weighted aggregate features from different spatial locations; and finally generating a depth image based on the network output. This application can significantly suppress noise, enhance the signal, and improve robustness through global feature aggregation, achieving high-precision depth reconstruction under extremely low signal-to-noise ratio conditions.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image processing technology, and in particular to a single-photon depth reconstruction method and system based on temporal feature aggregation. Background Technology

[0002] Three-dimensional imaging systems can provide depth information of a scene and have broad application prospects in fields such as remote sensing, autonomous driving, and machine vision. Among them, single-photon 3D imaging systems improve detection sensitivity to the level of a single photon, enabling ultra-long-distance detection and system miniaturization, laying the foundation for high-precision 3D reconstruction in weak signal environments. A typical single-photon imaging system scans the target scene with laser pulses and accumulates the photons reflected by the target, thereby obtaining a histogram of photon arrival times at each pixel, from which the distance information of the target can be calculated.

[0003] However, reconstructing high-quality depth images from raw single-photon measurement data presents significant challenges. On one hand, in applications with limited light flux and detection integration time, the number of effective signal photons is extremely small and easily contaminated by ambient background noise, resulting in a very low signal-to-noise ratio in the raw data. On the other hand, the photon detection process inherently involves Poisson noise, which is particularly pronounced with low photon counts. Therefore, developing high-performance reconstruction algorithms is crucial for single-photon 3D imaging technology.

[0004] In existing technologies, some schemes have attempted to use deep learning methods for single-photon depth reconstruction. For example, some methods employ a general encoder-decoder network architecture, extracting features through standard convolution and downsampling operations. However, these methods have significant drawbacks: they treat single-photon data (a three-dimensional data cube containing spatial and temporal dimensions) as ordinary three-dimensional data, and their general network structures fail to fully utilize the unique physical prior properties of single-photon data. Specifically, effective signal photons typically exhibit a relatively concentrated distribution (e.g., an approximate Gaussian distribution) in the temporal dimension, while background noise photons are approximately uniformly distributed in the temporal dimension. Existing general convolutional networks cannot effectively distinguish this difference in distribution, resulting in low efficiency in noise reduction and feature extraction. Especially under extreme conditions of extremely low signal-to-noise ratio, the performance of these methods deteriorates sharply, making it difficult to accurately recover weak depth information from strong noise, resulting in insufficient reconstruction accuracy and robustness. Summary of the Invention

[0005] The purpose of this application is to provide a single-photon depth reconstruction method and system based on temporal feature aggregation, which aims to solve the technical problem that the existing technology fails to make full use of the physical prior characteristics of the data in the time dimension when processing sparse, low signal-to-noise ratio single-photon data, resulting in insufficient depth reconstruction accuracy and robustness.

[0006] To achieve the above objectives, this application provides a single-photon depth reconstruction method based on temporal feature aggregation, applicable to processing single-photon 3D data containing spatial and temporal dimensions. The method comprises the following steps: temporal feature preprocessing of the single-photon 3D data using asymmetric 3D convolution, wherein the asymmetric 3D convolution downsamples along the temporal dimension while maintaining the spatial resolution, and the kernel weights of the asymmetric 3D convolution follow a preset non-uniform distribution matching the prior distribution of signal photons in the temporal dimension; inputting the preprocessed features into a multi-scale encoder-decoder network for multi-scale feature extraction and fusion; during the decoding process of the encoder-decoder network, using intermediate features generated by the decoder at a certain scale as features to be aggregated, and employing a global feature aggregation module based on temporal distribution similarity to perform weighted aggregation of features at different spatial locations, wherein the weights of the weighted aggregation depend on the temporal distribution similarity of the features at different spatial locations; and generating a reconstructed depth image based on the features output by the encoder-decoder network.

[0007] Optionally, the preset non-uniform distribution is a Gaussian distribution.

[0008] Optionally, the preset non-uniform distribution is a Laplace distribution.

[0009] Optionally, the global feature aggregation module obtains the temporal distribution similarity by projecting the features to be aggregated into a query tensor Q, a key tensor K, and a value tensor V, and by calculating the dot product of the query tensor Q and the key tensor K.

[0010] Furthermore, after completing the weighted aggregation, the global feature aggregation module adds the aggregated features to the features to be aggregated through residual connection.

[0011] Optionally, the global feature aggregation module determines the temporal distribution similarity by calculating the Pearson correlation coefficient of features at different spatial locations in the time dimension.

[0012] Optionally, the multi-scale encoder-decoder network is a U-Net++ architecture, and feature fusion between the encoder and decoder is achieved through nested cross-layer connections.

[0013] Optionally, the encoding process of the multi-scale encoder-decoder network includes the step of encoding features using sparse channel dense blocks.

[0014] In a preferred embodiment of this application, when the global feature aggregation module adds the aggregated features to the features to be aggregated through residual connection after completing the weighted aggregation, the preset non-uniform distribution is a Gaussian distribution.

[0015] This application also provides a single-photon depth reconstruction system, characterized in that it includes: a data acquisition module; a processor; and a memory storing computer program instructions; wherein the processor is connected to the data acquisition module and the memory, and the processor is configured to: acquire single-photon three-dimensional data including spatial and temporal dimensions from the data acquisition module; and execute the instructions stored in the memory to implement one or more single-photon depth reconstruction methods based on temporal feature aggregation as described in this application.

[0016] Compared with existing technologies, the technical solution provided in this application has the following beneficial effects: 1. Significant noise suppression and signal enhancement capabilities. The asymmetric three-dimensional convolution used in this application utilizes the physical prior of the concentrated distribution of signals in the time dimension to perform targeted smoothing and downsampling of the time dimension. This not only effectively suppresses the approximately uniformly distributed background noise and enhances the feature response of the effective signal, but also preserves the spatial structural details without loss, solving the problem that existing general convolution methods cannot effectively distinguish between signals and noise. 2. Global feature aggregation and robustness improvement. The global feature aggregation mechanism based on temporal distribution similarity designed in this application breaks through the local receptive field limitation of traditional convolutional networks. Based on the physical law that "points with similar depths also have similar photon time response patterns," it accurately aggregates pixel information with similar temporal distribution characteristics in the global space. This enables the network to construct robust feature representations even when signal photons are extremely sparse and local information is unreliable, solving the problem of inaccurate feature extraction under extreme conditions in existing methods. 3. High-precision depth reconstruction. By organically combining the two modules with deep fusion of physical priors into a multi-scale network architecture, this application can still achieve high-quality and robust deep information reconstruction under extreme conditions of extremely low signal-to-noise ratio, and its performance is significantly better than that of existing technologies using general network structures. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the overall network architecture provided in the embodiments of this application;

[0019] Figure 2 A schematic diagram illustrating the principle of asymmetric three-dimensional Gaussian convolution provided in this application embodiment;

[0020] Figure 3 This is a schematic diagram of the temporal similarity aggregation module structure provided in an embodiment of this application;

[0021] Figure 4 A flowchart of the single-photon depth reconstruction method provided in the embodiments of this application.

[0022] Key figure labels: AGC: Asymmetric Gaussian Convolution; SCDB: Sparse Channel Dense Block; TSA: Temporal Similarity Aggregation; Feature Fusion; Down Sampling; Up Sampling; Refine Layer; Soft Argmax; Reconstructed Depth; Signal Photon; Background Noise; Gaussian Convolution Kernel; Post-Convolution Features; Input Features; Q: Query Tensor; K: Key Tensor; V: Value Tensor; Reshape; Matrix Multiplication; Element-wise Addition; Output Features; S401: Step to acquire single-photon 3D data to be processed; S402: Step to perform temporal feature preprocessing using asymmetric 3D convolution; S403: Step to input the preprocessed features into a multi-scale encoder-decoder network for feature extraction and fusion; S404: Step to perform weighted aggregation using a global feature aggregation module based on temporal distribution similarity; S405: Step to generate a reconstructed depth image based on the features output by the network. Detailed Implementation

[0023] To better understand the technical solution of this application, the following will provide a more detailed description of this application in conjunction with the accompanying drawings and specific embodiments. It should be noted that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0024] Example 1

[0025] This embodiment provides a specific implementation of a single-photon depth reconstruction method based on temporal feature aggregation. This method is executed through a deep learning network model and aims to accurately recover scene depth information from single-photon 3D data with extremely low signal-to-noise ratios. (Refer to...) Figure 4 The overall process of this method can be summarized into the following core steps.

[0026] First, the three-dimensional data of the single-photon particles to be processed is acquired, which includes both spatial and temporal dimensions. In single-photon imaging systems, this step is typically accomplished by scanning the scene and recording the arrival times of photons detected at each spatial pixel over a given time period. The acquired data forms a three-dimensional data cube, whose dimensions can be represented as (C, T, H, W), where C is the number of feature channels (typically 1 for the raw input), T is the number of time intervals, and H and W are the height and width of the spatial dimension, respectively. For example, a typical input data cube might have dimensions of 1×1024×64×64, meaning that each of the 64×64 pixels records single-photon counts for 1024 time intervals. It is understood that these counts are a mixture of effective signal photons from the target and background noise photons from the environment.

[0027] Next, asymmetric 3D convolution is used to perform temporal feature preprocessing on the 3D data to enhance signal features and suppress noise. This step is a key front-end processing step in this application, aiming to utilize the difference in the distribution characteristics of signal photons and noise photons in the time dimension for preliminary feature purification.

[0028] Subsequently, the preprocessed features are input into a multi-scale encoder-decoder network for feature extraction and fusion. This network is responsible for learning and extracting abstract features related to depth information from the preprocessed data at different scales.

[0029] In the decoding process of this network, a global feature aggregation module based on temporal distribution similarity is further employed to perform weighted aggregation of features from different spatial locations. This step is another core aspect of the proposed solution, aiming to overcome the limitations of the local receptive field in traditional convolution and enhance the robustness of features in photon-sparse conditions through global correlation.

[0030] Finally, a reconstructed depth image is generated based on the features output by the network. The network ultimately outputs a feature map with the same spatial resolution as the input. Each pixel in this feature map encodes probabilistic depth information in the temporal dimension, which can be converted into the final depth value through specific calculations, thus forming a two-dimensional depth image.

[0031] The following will combine Figure 1 , Figure 2 and Figure 3The specific network architecture and modules for implementing the above process in this embodiment are described in detail.

[0032] Figure 1 A schematic diagram of the overall network architecture used in this embodiment is shown. The network employs a nested multi-scale encoder-decoder architecture, specifically an improved U-Net++ network structure. The entire network consists of encoder paths, decoder paths, and a final output layer.

[0033] The single-photon 3D data cube (e.g., 1×1024×64×64) described above is taken as input and fed into the encoder path of the network for processing.

[0034] The core of the encoder path lies in its front-end asymmetric Gaussian convolution module and the subsequent multi-scale feature encoding part.

[0035] Specifically, the data stream first passes through a preprocessing unit consisting of three cascaded asymmetric Gaussian convolution modules, such as... Figure 1 As shown, this unit is located at the network's entry point. It is understandable that the design of the asymmetric Gaussian convolution module utilizes prior physical knowledge from single-photon imaging, namely that effective signal photons typically exhibit a concentrated, approximately Gaussian distribution over time, while background noise photons are approximately uniformly distributed.

[0036] Figure 2 This is a schematic diagram illustrating the principle of asymmetric 3D Gaussian convolution. As shown, a time series contains a concentrated distribution of signal photons and a uniform distribution of background noise. In one embodiment of this application, the asymmetric 3D convolution uses a K×1×1 convolution kernel, meaning that the convolution operation is performed only in the time dimension, and not in the spatial dimension, thus completely preserving the spatial structural details of the scene. The weights of this convolution kernel are not uniformly or randomly initialized, but follow a preset non-uniform distribution; in this preferred embodiment, this distribution is Gaussian. Figure 2 As shown in the Gaussian convolution kernel, its weight distribution matches the distribution of signal photons. When this Gaussian convolution kernel is applied to the input time series, the response value is significantly enhanced because the region where signal photons are concentrated aligns with the region where the kernel has higher weights. Correspondingly, the influence of uniformly distributed background noise is effectively smoothed and suppressed after Gaussian weighted averaging. The final convolutional feature has a significantly improved signal-to-noise ratio compared to the original data.

[0037] In this embodiment, the kernel size K of the asymmetric Gaussian convolution module can be adaptively set according to the characteristics of the actual laser pulse signal. As a preferred setting, K = 2 × FWHM + 1, where FWHM is the full width at half maximum (FWHM) of the laser pulse signal. For example, when FWHM has 3 time intervals, the kernel size K is 7, i.e., a 7 × 1 × 1 convolution kernel is used. Furthermore, the stride of this convolution operation is set to 2 in the time dimension and 1 in the spatial dimension. This means that each module, while enhancing the signal, halves the resolution in the time dimension, effectively compressing data redundancy and reducing the complexity of subsequent calculations. After processing by three cascaded modules, the time dimension of the input feature map decreases sequentially from 1024 to 512, 256, and finally to 128, achieving an 8-fold temporal downsampling while maintaining the spatial resolution of 64 × 64.

[0038] After module preprocessing, the feature map enters a multi-level encoding path consisting of sparse channel 3D dense blocks (SCDB) and downsampling layers. For example... Figure 1 As shown, the encoding path contains multiple scale levels (e.g., from...). arrive At each scale level, sparse-channel 3D dense blocks (SCDB) are used for feature encoding. Within each SCDB module, dense connections reuse features from different levels to enhance their expressive power. Simultaneously, L1 regularization can be applied to the channels of the SCDB module to encourage the network to learn sparse channel weights, dynamically selecting channels more relevant to the current task, thus enhancing features while mitigating the overfitting problem common in deep networks. After each SCDB module, a downsampling layer (e.g., a 2×2 max-pooling layer) halves the spatial resolution of the feature map (e.g., from 64×64 to 32×32, then to 16×16, etc.), and the number of feature channels is increased accordingly. This process is performed step-by-step, enabling the network to learn multi-scale features from local details to global semantics.

[0039] Next, the data enters the decoder path. The decoder in this embodiment employs the nested and dense cross-layer connection structure unique to the U-Net++ architecture. For example... Figure 1 As shown, the decoder path not only includes bottom-up upsampling paths but also a large number of lateral and diagonal connections. A generic decoder node The output (where i represents the downsampling depth and j represents the horizontal convolutional layer index) is obtained by fusing the output from the previous node at the same level. Nodes from the corresponding level of the encoder and upsampling nodes from deeper levels The features are obtained from the data. This feature fusion operation usually involves first concatenating all input feature maps along the channel dimension, and then processing them through a convolutional block (e.g., containing 3D convolution, batch normalization, and activation functions). This dense fusion method allows the decoder to fully utilize deep semantic information and shallow detail information from different scales and levels of abstraction at every stage of image reconstruction, which helps to generate high-quality reconstruction results.

[0040] A key innovation of this embodiment lies in the introduction of a temporal similarity aggregation module during the feature fusion process of the decoder. For example... Figure 1 As shown, a temporal similarity aggregation module is deployed after each nested feature fusion node. The core idea of ​​this module is that object points with similar depths, even if spatially far apart, should have similar temporal response patterns in their reflected photons. The temporal similarity aggregation module utilizes this physical law to overcome the limitations of the local receptive field in traditional convolution operations, and associates and aggregates features in the global space.

[0041] Figure 3 This is a detailed structural diagram of the temporal similarity aggregation module. The input features of this module are the outputs of a feature fusion node of the decoder, and its dimensions can be represented as (B, C, T', H', W'), where B is the batch size, C is the number of channels, T' is the temporal dimension at the current scale, and H' and W' are the spatial dimensions at the current scale.

[0042] Within this module, the input features are first passed through three parallel 1×1×1 3D convolutional layers to generate a query tensor Q, a key tensor K, and a value tensor V, respectively. These three convolutional layers are learnable linear projection layers. To reduce computational cost, the number of channels C generated by the convolutional layers for Q and K can be compressed, for example, to C / 4, while the number of channels generated by the convolutional layer for V can remain unchanged or be adjusted as needed.

[0043] Subsequently, the generated Q, K, and V tensors are reshaped, a crucial step for achieving global spatial correlation. Specifically, the Q and K tensors with dimensions (B, C', T', H', W') are reshaped into (B, H'×W', C'×T'), and the V tensor with dimensions (B, C, T', H', W') is reshaped into (B, H'×W', C×T'). In this way, the spatial dimension (H', W') is flattened into one dimension, while the channel dimension and the temporal dimension (C', T') are also flattened into another dimension. The reshaped tensors can be viewed as a sequence of length H'×W' (i.e., the total number of pixels), where each element is a vector of length C'×T' or C×T', which fully represents the temporal distribution characteristics of the corresponding spatial location, incorporating multi-channel information.

[0044] Next, matrix multiplication is performed to compute attention weights and aggregate features. First, the dot product of the query tensor Q and the transpose of the key tensor K is calculated, i.e. Where Q has dimensions (B, H'W', C'T'), and K is the transpose of Q. The dimensions of the attention matrix are (B, C'T', H'W'), therefore the dimensions of the attention matrix are (B, H'W', H'W'). Each element of this attention matrix, Attention[i, j], measures the similarity in temporal features between the pixel at spatial location i and the pixel at spatial location j. The higher the similarity, the larger the value. The Softmax function then normalizes these similarity scores to weights between 0 and 1.

[0045] Then, the obtained attention matrix is ​​multiplied by the value tensor V, i.e. Where V has dimensions (B, H'W', CT'), therefore the aggregated features The dimension is also (B, H'W', CT'). The physical meaning of this operation is that, for each pixel i, its new feature is obtained by applying the features of all pixels j ( The values ​​are obtained by weighted summation, and the weights (Attention[i,j]) are proportional to the temporal similarity between pixel i and pixel j. In this way, even when the signal is sparse and local information is unreliable, the network can find and aggregate pixel features with similar temporal response patterns from a global perspective, thereby greatly enhancing the robustness of the features.

[0046] The aggregated features undergo a reverse reshaping operation to restore them to the (B, C, T', H', W') dimension, and can then be further transformed using a 1×1×1 convolution. Finally, as... Figure 3As shown in the element-wise addition diagram, the module uses residual connections to add the aggregated and transformed features to the module's original input features, obtaining the final output features. Residual connections facilitate stable gradient propagation, allowing the network to be built deeper while ensuring the flow of original information.

[0047] After the complete decoder path, which includes a temporal similarity aggregation module, the network reaches the highest resolution scale (e.g., X). 0 , 4 The final feature map is then output. This feature map is first fine-tuned by a refining layer (e.g., one or more standard 3D convolutional blocks).

[0048] Finally, the refined feature map is fed into a Soft Argmax layer. This layer is responsible for converting the temporal dimension features encoding the depth probability distribution into a deterministic depth value. Specifically, for each spatial location (h, w) in the output feature map, the Soft Argmax layer first applies the Softmax function to its feature vector in the temporal dimension T', transforming it into a probability distribution P(t|h,w). Then, the depth value is obtained by calculating the expectation of this distribution, which can be expressed by the following formula: Where t is the index of the time interval. By performing this operation on all spatial locations, a reconstructed depth image of dimension H×W is ultimately generated.

[0049] In summary, this embodiment organically combines an asymmetric Gaussian convolution module that utilizes physical priors with a temporal similarity aggregation module that overcomes local constraints within the advanced U-Net++ network architecture, thereby enabling high-precision and robust deep reconstruction of sparse, low signal-to-noise ratio single-photon data.

[0050] Example 2

[0051] This embodiment provides a variant implementation of a single-photon depth reconstruction method based on temporal feature aggregation. It aims to illustrate that the core modules of the present application scheme, namely asymmetric Gaussian convolution and temporal similarity aggregation, are not limited to the complex U-Net++ architecture in Embodiment 1. They can also play a key role in more standard and simpler network architectures, thus proving that the present application scheme has good universality.

[0052] In this embodiment, the overall network architecture adopts the standard U-Net structure, rather than U-Net++. The design of the encoder part, output layer part, and asymmetric 3D convolution and temporal similarity aggregation module are basically consistent with those in Embodiment 1. The main difference lies in the structure of the decoder and the feature fusion method.

[0053] Specifically, the process still follows Figure 4 The steps are shown. After acquiring single-photon 3D data in step S401, in step S402, the data is also first processed by a preprocessing unit consisting of three cascaded asymmetric Gaussian convolution modules. The implementation of the asymmetric 3D convolution module is exactly the same as in Example 1, using a K×1×1 Gaussian kernel to smooth and downsample the time dimension, compressing the time dimension from 1024 to 128, in order to suppress noise, enhance the signal, and reduce data redundancy.

[0054] In step S403, the preprocessed features are fed into a standard U-Net encoder. This encoder consists of alternating encoding blocks and downsampling layers. Each encoding block can be composed of two consecutive 3×3×3 three-dimensional convolutional layers, a batch normalization layer, and a ReLU activation function. This differs from the sparse channel dense block (SCDB) used in Example 1, but still serves the same purpose of extracting features at different scales. Each encoding block is followed by a max pooling layer for downsampling, progressively reducing the spatial resolution and increasing the number of channels.

[0055] The key difference lies in the decoder. The decoder path in standard U-Net is symmetrical to the encoder path. At each stage of the decoder, the spatial resolution of the feature map from the next stage is first doubled through an upsampling layer (e.g., a transposed convolution). Then, the upsampled feature map is directly concatenated with the feature map output from the corresponding encoder layer along the channel dimension. This is U-Net's classic cross-layer connection method, aiming to supplement the decoder with high-resolution, detailed shallow features captured by the encoder to aid in accurate localization and reconstruction.

[0056] As an improvement to this embodiment, to incorporate the concept of global temporal aggregation, a temporal similarity aggregation module is inserted after the concatenation operation and before entering the current-level decoding convolutional block (e.g., two consecutive 3×3×3 convolutions). That is, the input to the temporal similarity aggregation module is the fused feature obtained by concatenating the encoder features and upsampled features. The internal structure and working principle of the temporal similarity aggregation module are the same as in Embodiment 1. Figure 3 The result is completely consistent with the above: by learning Q, K, and V projections, a global attention map based on temporal distribution similarity is calculated, and this map is used as a weight to aggregate features at all spatial locations. Finally, the optimized features are output through residual connections.

[0057] The features optimized by the temporal similarity aggregation module are then fed into the standard convolutional block of the decoding layer for further feature extraction and transformation. This process is repeated at each level of the decoder.

[0058] Finally, in step S405, the output of the decoder's highest resolution layer also passes through a refinement layer and a SoftArgmax layer to calculate the expected depth of each pixel in the time dimension, thereby generating the final reconstructed depth image.

[0059] Compared to the original U-Net network that uses general convolutions, this embodiment deeply integrates the physical priors of single-photon data into network learning by introducing an asymmetric 3D convolution module at the encoder front end and a temporal similarity aggregation module after the decoder fusion node. Although its network skeleton (U-Net) may not be as complex as U-Net++ in Embodiment 1, the introduction of these two core modules still significantly improves the depth reconstruction accuracy and robustness when processing low signal-to-noise ratio and low photon count data. Therefore, this embodiment demonstrates the effectiveness of the technical concept proposed in this application and its modular application potential, supporting a broader scope of protection.

[0060] Example 3

[0061] This embodiment provides another variant implementation of the solution in this application, which aims to illustrate that the core idea of ​​the asymmetric three-dimensional convolution module is to utilize a convolution kernel that matches the prior of the signal's temporal distribution, and this prior distribution itself is not limited to a Gaussian distribution.

[0062] The overall network architecture, data processing flow, and settings of most modules in this embodiment are exactly the same as in Embodiment 1. Specifically, it also adopts a nested encoder-decoder architecture based on U-Net++. The encoder contains a sparse channel 3D dense block SCDB, and the decoder deploys a temporal similarity aggregation module based on the QKV attention mechanism after each feature fusion node. The output layer also uses a refinement layer and a Soft Argmax layer.

[0063] The main difference between this embodiment and Embodiment 1 lies in the implementation details of the asymmetric 3D convolution module at the encoder front end. In Embodiment 1, the asymmetric 3D convolution module uses a K×1×1 convolution kernel, and its weights are initialized to follow a Gaussian distribution. In this embodiment, however, the weights of the convolution kernel are initialized to follow a Laplace distribution.

[0064] The probability density function of the Laplace distribution can be expressed as: Where μ is the location parameter and b is the scale parameter. Compared to the Gaussian distribution, the Laplace distribution has a sharper peak at the mean μ, and its tail is "heavier" than that of the Gaussian distribution, meaning it decays more slowly.

[0065] The motivation for using a Laplace distribution to initialize the convolution kernel weights is that different types of single-photon imaging systems may use different laser sources, and the laser pulses produced by these sources have different waveforms over time. While a Gaussian distribution is a common and good approximation, some lasers may produce pulse signals with steeper rise and fall edges, whose shape may more closely resemble the peaked shape of a Laplace distribution. In this case, initializing the convolution kernel with a Laplace distribution that better matches the signal prior allows the network to make more accurate assumptions about the signal pattern in the early stages of training, which may help the network converge faster or achieve higher matching accuracy.

[0066] Therefore, in this embodiment, step S402 is specifically performed as follows: K×1×1 asymmetric three-dimensional convolution is used to preprocess the input data for temporal features, wherein the weights of the convolution kernel follow a Laplace distribution. Subsequent steps S403, S404, and S405 are exactly the same as in Embodiment 1.

[0067] This embodiment aims to illustrate that the core idea of ​​temporal feature preprocessing in this application lies in designing asymmetric convolution kernels by "using a pre-defined non-uniform distribution that matches the prior distribution of signal photons in the time dimension." Gaussian and Laplace distributions are preferred specific examples under this idea. This design shows that the technical solution of this application is not limited to a specific prior distribution, but covers a class of methods for temporal denoising and signal enhancement using non-uniform prior distributions, thus supporting a higher-level generalization of this technical point.

[0068] Example 4

[0069] This embodiment provides another variant implementation of the solution in this application, which aims to illustrate the core functional idea of ​​"global feature aggregation based on temporal distribution similarity". It can have a different specific implementation method than the query-key-value attention mechanism described in Embodiments 1 and 2.

[0070] The overall network architecture, data processing flow, and module settings (except for the global aggregation module) of this embodiment are consistent with those of Embodiment 1. That is, it also adopts a nested encoder-decoder architecture based on U-Net++, with the encoder front end using an asymmetric Gaussian convolution module for preprocessing, followed by multi-scale encoding using sparse channel 3D dense block SCDB, and the output layer also using a refinement layer and a soft Argmax layer.

[0071] The only difference between this embodiment and Embodiment 1 lies in the specific implementation of the module used for global feature aggregation in the decoder. In Embodiment 1, this function is implemented by a temporal similarity aggregation module that learns Q, K, and V projections and calculates attention. In this embodiment, however, this module is replaced by a "temporal correlation aggregation layer".

[0072] The temporal correlation aggregation layer works as follows: First, the input to this layer is obtained, namely the feature map output by a feature fusion node in the decoder, with dimensions (B, C, T', H', W'). Next, for each spatial location i in the input feature map (i ranging from 1 to H'×W'), its complete temporal feature vector Vᵢ is extracted. This vector Vᵢ has dimensions (C, T'), representing the temporal response pattern of the pixel at the current scale, and can be flattened into a one-dimensional vector of length C×T' for easier computation. Then, a classic statistical metric—the Pearson correlation coefficient—is used to directly calculate the similarity between the temporal feature vectors of different spatial locations. For any two spatial locations i and j, the Pearson correlation coefficient ρᵢ,ⱼ between them is calculated as follows: Where cov( , ) is a vector and covariance, and These are their standard deviations. The coefficient ranges from -1 to 1; the closer the value is to 1, the stronger the linear correlation between the two vectors, meaning their temporal distribution patterns are more similar. By calculating the correlation coefficients between all pixel pairs, a correlation matrix M of size (H'W' × H'W') can be constructed. Subsequently, the Softmax function is applied to each row of the correlation matrix M for normalization, resulting in the attention weight matrix A, i.e., A[i, j] = softmax(M[i, :])[j]. Thus, for each pixel i, a set of weights A[i, :] is obtained, representing the weights contributed by all other pixels j to it. Furthermore, the attention weight matrix A is compared with the original feature vectors of all pixels (which can be viewed as a matrix of size (H'W' × CT')). Perform matrix multiplication to obtain the aggregated new features. For each pixel i, its new feature vector... It is the original feature vector of all pixels j. Weighted sum: Finally, similar to the temporal similarity aggregation module in Example 1, this aggregation layer can also use residual connections to add the aggregated new features (after appropriate dimensional reshaping) to the original input features of the layer element by element to ensure smooth information flow and training stability.

[0073] Through the above steps, this temporal correlation aggregation layer also achieves the function of global feature aggregation based on temporal distribution similarity. This implementation method proves that the core of the proposed solution lies in the functional idea of ​​"global feature aggregation based on temporal distribution similarity", rather than being limited to the specific implementation technology of QKV attention, thus effectively supporting higher-level protection for this technical point.

[0074] System Implementation Examples

[0075] This application also provides a single-photon depth reconstruction system, which is a physical carrier for performing the methods described in any of the above embodiments. It can be an integrated single-photon imaging device, or a general-purpose or special-purpose computer, server, or embedded device.

[0076] Specifically, the system includes: a data acquisition module, a processor, and a memory.

[0077] The data acquisition module is responsible for collecting raw single-photon 3D data. In a typical single-photon lidar or microscope system, this module may include a laser emitter, a scanning galvanometer, a single-photon detector array (such as a single-photon avalanche diode array), and a time-correlated single-photon counting unit. The final output of this module is the single-photon 3D data cube containing spatial and temporal dimensions, which was used as network input in the aforementioned embodiment.

[0078] The memory can be any form of computer-readable storage medium, such as random access memory, read-only memory, solid-state drive, or hard disk drive. The memory contains computer program instructions that fully define the structure of the deep learning network model described in Examples 1 to 4 (including but not limited to asymmetric convolutional modules, multi-scale encoder-decoder architectures, global feature aggregation modules, etc.) and its training and inference processes.

[0079] The processor can be a central processing unit, a graphics processing unit, a dedicated neural network processing unit, or a field-programmable gate array, etc. The processor is connected to the data acquisition module and memory via a bus or other means.

[0080] When the system is working, the processor executes computer program instructions stored in the memory to perform the following operations: receiving single-photon 3D data to be processed from the data acquisition module or reading it from the memory; loading and running a deep learning network model and using the 3D data as model input; performing calculations and processing strictly according to the method steps described in any one of Examples 1 to 4 (including temporal feature preprocessing, multi-scale feature extraction and fusion, global feature aggregation based on temporal similarity, etc.); and finally generating a reconstructed depth image based on the features output by the network model, and outputting the depth image to a display device or storing it back in the memory for subsequent use.

[0081] In this way, the system constitutes a complete physical device capable of realizing the high-precision single-photon depth reconstruction method proposed in this application.

[0082] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A single-photon depth reconstruction method based on temporal feature aggregation, applied to processing single-photon three-dimensional data containing spatial and temporal dimensions, characterized in that, Includes the following steps: The single-photon three-dimensional data is preprocessed using asymmetric three-dimensional convolution in the temporal domain. The asymmetric three-dimensional convolution downsamples along the time dimension while maintaining the spatial dimension resolution. The kernel weights of the asymmetric three-dimensional convolution follow a preset non-uniform distribution that matches the prior distribution of signal photons in the time dimension. The preprocessed features are input into a multi-scale encoder-decoder network for multi-scale feature extraction and fusion. During the decoding process of the encoder-decoder network, the intermediate features generated by the decoder at a certain scale are used as features to be aggregated. A global feature aggregation module based on temporal distribution similarity is used to perform weighted aggregation of features at different spatial locations in the features to be aggregated. The weight of the weighted aggregation depends on the distribution similarity of the features at different spatial locations in the temporal dimension. A reconstructed depth image is generated based on the features output by the encoder-decoder network.

2. The method according to claim 1, characterized in that, The preset non-uniform distribution is a Gaussian distribution.

3. The method according to claim 1, characterized in that, The preset non-uniform distribution is a Laplace distribution.

4. The method according to claim 1, characterized in that, The global feature aggregation module projects the features to be aggregated into a query tensor Q, a key tensor K, and a value tensor V, and obtains the temporal distribution similarity by calculating the dot product of the query tensor Q and the key tensor K.

5. The method according to claim 4, characterized in that, After completing the weighted aggregation, the global feature aggregation module adds the aggregated features to the features to be aggregated through residual connection.

6. The method according to claim 1, characterized in that, The global feature aggregation module determines the temporal distribution similarity by calculating the Pearson correlation coefficient of features at different spatial locations in the time dimension.

7. The method according to claim 1, characterized in that, The multi-scale encoder-decoder network is based on the U-Net++ architecture and achieves feature fusion between the encoder and decoder through nested cross-layer connections.

8. The method according to claim 1, characterized in that, The encoding process of the multi-scale encoder-decoder network includes the step of encoding features using sparse channel dense blocks (SCDB).

9. A single-photon depth reconstruction system, characterized in that, include: Data acquisition module; processor; And memory that stores computer program instructions; The processor is connected to the data acquisition module and the memory, and the processor is configured to: The data acquisition module acquires single-photon three-dimensional data containing spatial and temporal dimensions; and Execute the instructions stored in the memory to implement the method as described in claim 1.

Images