A single-photon imaging super-resolution reconstruction method, system, device and medium

Abstract

The application belongs to the field of computational imaging, and discloses a single-photon imaging super-resolution reconstruction method, system, device and medium. Sub-scan data of a target after spatial sub-scanning and time sub-scanning is acquired. A plurality of single-photon range maps are obtained based on the sub-scan data. Shallow space-time features of the plurality of single-photon range maps are extracted. High-level feature extraction is performed on the plurality of single-photon range maps with shallow space-time features. The plurality of single-photon range maps with high-level features are gradually reduced in time dimension, and then up-sampling is performed through sub-pixel convolution to obtain a high-resolution feature map with fused space-time detail information. The plurality of single-photon range maps with shallow space-time features are weightedly fused, and then up-sampling is performed through sub-pixel convolution to obtain a high-resolution feature map based on weighted average of multiple frames. Corresponding elements of the two high-resolution feature maps are added to obtain a reconstructed super-resolution depth map.

Description

Technical Field

[0001] This invention belongs to the field of computational imaging and relates to a method, system, device and medium for super-resolution reconstruction of single-photon imaging. Background Technology

[0002] LiDAR provides crucial 3D scene information for tasks such as autonomous vehicles, robot vision, 3D object modeling, gesture recognition, pose tracking, scene understanding, segmentation, localization and mapping, and remote sensing. With technological advancements, the detection limits of traditional photodetectors have become a bottleneck for the development of traditional LiDAR. In recent years, LiDAR systems composed of single-photon avalanche detectors have been widely studied as a next-generation imaging technology. Single-photon LiDAR offers single-photon sensitivity and picosecond temporal resolution, making it a key supporting technology for future autonomous systems.

[0003] However, current single-photon 3D imaging technology faces two major challenges in terms of imaging resolution. First, in terms of spatial resolution, the limited size of single-photon lidar arrays and the relatively large individual pixel size (typically around 50 μm) result in a spatial sampling rate far lower than that of traditional CMOS (Complementary Metal Oxide Semiconductor) image sensors (pixel size around 7 μm), leading to lower spatial resolution. Second, in terms of temporal resolution, there is an inherent dead time due to the quenching and recovery process of the device itself, as well as systematic delays in pulsed laser time-of-flight measurements, causing a deviation between the photon arrival time recorded by the detector and the actual arrival time, resulting in a decrease in temporal resolution. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a single-photon imaging super-resolution reconstruction method, system, device and medium to achieve the collaborative acquisition of temporal and spatial information, which not only retains the ability of temporal super-resolution to capture depth details, but also enhances the high-resolution restoration effect of spatial super-resolution on the target contour.

[0005] To achieve the above objectives, the present invention employs the following technical solution: A single-photon imaging super-resolution reconstruction method includes the following steps: Acquire subscan data of the target after spatial and temporal subscanning; Multi-frame single-photon range maps were obtained based on subscan data; Extracting shallow spatiotemporal features from multi-frame single-photon distance maps; High-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal characteristics; For multi-frame single-photon distance maps with high-level features, the temporal dimension is gradually reduced, and then upsampling is performed through sub-pixel convolution to obtain a high-resolution feature map that integrates spatiotemporal detail information. We perform weighted fusion on multi-frame single-photon distance maps with shallow spatiotemporal features, and then upsample through sub-pixel convolution to obtain a high-resolution feature map based on multi-frame weighted averaging. The high-resolution feature map that integrates spatiotemporal details and the high-resolution feature map based on multi-frame weighted averaging are added element by element to obtain the reconstructed super-resolution depth map.

[0006] Optionally, acquire subscan data of the target after spatial and temporal subscanning, including: Spatial subscan: Each pixel is divided into K*K sub-regions. At the start of the scan, the scanning system first scans the entire field of view at the initial position to acquire image data of all pixels in the first sub-region. Subsequently, the scanning system moves in the horizontal and vertical directions, with each movement step being 1 / K of the physical pixel size. Every time the scanning system moves to a new position, it scans the entire field of view again, thereby simultaneously acquiring image data of all pixels in the new sub-region position, until the scanning of all sub-region positions is completed, resulting in K*K image data, which are used as spatial image data. Temporal subscan: During spatial subscan, based on the temporal resolution of the scanning system, the time is subdivided by time delay, and the time is recorded based on the preset time for each delay to obtain time record data; By combining spatial image data with time-recorded data, subscan data is obtained.

[0007] Optionally, the subscan data is preprocessed to obtain multi-frame single-photon range maps, including: Convert the photon histogram of each pixel in the subscan data into a probability distribution function; Calculate the expected value of the probability distribution function, where the expected value represents the centroid of the photon arrival time; Convert the mathematical expectation value into a depth value; The depth values ​​calculated from the photon histogram of each pixel are arranged into a two-dimensional matrix according to the pixel coordinates to obtain a multi-frame single-photon distance map.

[0008] Optionally, shallow spatiotemporal features are extracted from multi-frame single-photon range maps, including: Using multi-frame single-photon distance maps as input tensors, shallow spatiotemporal features are extracted from the input tensors using 3D convolutional layers.

[0009] Optionally, high-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal features, including: By using multiple cascaded residual feature attention modules, high-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal features, and high-level feature descriptions are obtained by global average pooling. Each residual feature attention module includes a first three-dimensional convolutional layer, a first linear rectified activation function layer, a second three-dimensional convolutional layer, and a feature attention module connected in sequence; The feature attention module consists of a global pooling layer, a third 3D convolutional layer, a second linear rectified activation function layer, a fourth 3D convolutional layer, and a sigmoid activation function layer connected in sequence.

[0010] Optionally, for multi-frame single-photon distance maps with high-level features, the temporal dimension is progressively reduced, and then upsampling is performed through sub-pixel convolution to obtain a high-resolution feature map that fuses spatiotemporal detail information, including: The temporal dimension of the multi-frame single-photon distance map with high-level features is gradually reduced by repeating the time reduction module multiple times. The time reduction module includes a residual feature attention module, a time-reduced convolutional layer, and a linear rectified activation function layer connected in sequence. The temporal dimension is reduced to one by using time-shortened convolutional layers, while the number of channels is increased at the same time; Then, upsampling is performed through subpixel convolution to generate a high-resolution spatial feature map that integrates spatiotemporal details.

[0011] Optionally, a weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features is performed, followed by upsampling through sub-pixel convolution to obtain a high-resolution feature map based on a multi-frame weighted average, including: A residual temporal attention mechanism combining global residual paths is adopted to achieve weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features; The residual temporal attention mechanism is executed using a residual temporal attention module, which consists of a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, and a temporal attention module connected in sequence. The temporal attention module consists of a global pooling layer, a third two-dimensional convolutional layer, a linear rectified activation function layer, a fourth two-dimensional convolutional layer, and a sigmoid activation function layer connected in sequence. The fourth two-dimensional convolutional layer of the residual temporal attention module expands the number of channels, and then upsamples through sub-pixel convolution to generate a high-resolution spatial feature map based on multi-frame weighted averaging.

[0012] A single-photon imaging super-resolution reconstruction system includes: The subscan data acquisition module is used to acquire subscan data of the target after spatial and temporal subscans. A multi-frame single-photon range map acquisition module is used to obtain multi-frame single-photon range maps based on subscan data; The shallow spatiotemporal feature module is used to extract shallow spatiotemporal features from multi-frame single-photon distance maps; The high-level feature extraction module is used to extract high-level features from multi-frame single-photon distance maps with shallow spatiotemporal features. The module for fusing spatiotemporal detail information is used to gradually reduce the temporal dimension of multi-frame single-photon distance maps with high-level features, and then upsample them through sub-pixel convolution to obtain a high-resolution feature map that fuses spatiotemporal detail information. The multi-frame weighted averaging module is used to perform weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features, and then upsampling is performed through sub-pixel convolution to obtain a high-resolution feature map based on multi-frame weighted averaging. The super-resolution depth map acquisition module is used to add corresponding elements of the high-resolution feature map that integrates spatiotemporal detail information and the high-resolution feature map based on multi-frame weighted averaging to obtain the reconstructed super-resolution depth map.

[0013] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the single-photon imaging super-resolution reconstruction method.

[0014] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the single-photon imaging super-resolution reconstruction method.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention obtains multi-frame single-photon range maps by acquiring sub-scan data of the target after spatial and temporal sub-scanning. After extracting shallow spatiotemporal features from the multi-frame single-photon range maps, they are processed through two independent paths. The first path performs high-level feature extraction on the multi-frame single-photon range maps with shallow spatiotemporal features, gradually reducing the temporal dimension. This allows for in-depth mining of subtle changes between adjacent spatial pixels and adjacent temporal frames without prematurely destroying the temporal dimension, resulting in a high-resolution feature map that fuses spatiotemporal details. The second path performs weighted fusion on the multi-frame single-photon range maps with shallow spatiotemporal features, effectively smoothing random noise and obtaining a high-resolution feature map based on a multi-frame weighted average. This ensures that the overall structure of the high-resolution feature map is not distorted due to over-processing. Finally, the high-resolution feature map that integrates spatiotemporal details and the high-resolution feature map based on multi-frame weighted averaging are added element by element. This combines rich spatiotemporal details with the benchmark of multi-frame weighted averaging, preserving the ability of temporal super-resolution to capture depth details and enhancing the high-resolution restoration effect of spatial super-resolution on the target contour. The result is a super-resolution depth map, which achieves the collaborative acquisition of information in both temporal and spatial dimensions. Attached Figure Description

[0016] Figure 1 This is an overall flowchart of the single-photon imaging super-resolution reconstruction method of Embodiment 1 of the present invention; Figure 2 This is an overall flowchart of the single-photon imaging super-resolution reconstruction method based on spatiotemporal subscanning and residual attention neural network according to Embodiment 2 of the present invention; Figure 3 This is a diagram showing the overall framework of the residual attention neural network in Embodiment 2 of the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] Example 1 This embodiment provides a single-photon imaging super-resolution reconstruction method, such as... Figure 1 As shown, the process includes the following: Acquire subscan data of the target after spatial and temporal subscanning.

[0019] Multi-frame single-photon distance maps were obtained based on subscan data.

[0020] Extract shallow spatiotemporal features from multi-frame single-photon distance maps.

[0021] High-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal characteristics.

[0022] For multi-frame single-photon distance maps with high-level features, the temporal dimension is gradually reduced, and then upsampling is performed through subpixel convolution to obtain a high-resolution feature map that integrates spatiotemporal detail information.

[0023] We perform weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features, and then upsample through sub-pixel convolution to obtain a high-resolution feature map based on multi-frame weighted averaging.

[0024] The high-resolution feature map that integrates spatiotemporal details and the high-resolution feature map based on multi-frame weighted averaging are added element by element to obtain the reconstructed super-resolution depth map.

[0025] Specifically, acquiring the subscan data of the target after spatial and temporal subscanning includes: Spatial subscan: Each pixel is divided into K*K sub-regions. At the start of the scan, the scanning system first scans the entire field of view from the initial position, acquiring image data of all pixels in the first sub-region. Subsequently, the scanning system moves horizontally and vertically, with each movement taking a step of 1 / K of the physical pixel size. Each time the scanning system moves to a new position, it scans the entire field of view again, simultaneously acquiring image data of all pixels in the new sub-region, until all sub-regions are scanned, resulting in K*K image data, which are then used as the spatial image data.

[0026] Temporal subscan: During spatial subscan, based on the temporal resolution of the scanning system, the time is subdivided by time delay. The time is recorded based on the preset time for each delay, and the time recording data is obtained.

[0027] By combining spatial image data with time-recorded data, subscan data is obtained.

[0028] Spatially, each pixel is subdivided into multiple sub-regions. A scanning system performs precise displacement based on the displacement amount and measures again to obtain multiple sets of data. Temporally, based on the hardware time resolution, time is recorded by subdividing through time delay to obtain sub-scan data. This process, based on the limited scale of the single-photon lidar array and the large size of a single pixel, adds additional spatiotemporal measurement information to the hardware time resolution and pixel size by subdividing into sub-regions and time delay.

[0029] Specifically, the subscan data is preprocessed to obtain multi-frame single-photon range maps, including: The photon histogram of each pixel in the subscan data is converted into a probability distribution function; the expected value of the probability distribution function is calculated, which represents the centroid of the photon arrival time; the expected value is converted into a depth value; the depth values ​​calculated from the photon histogram of each pixel are arranged into a two-dimensional matrix according to the pixel coordinates to obtain a multi-frame single-photon distance map.

[0030] The photon histogram of each pixel in the subscan data is converted into a probability distribution function, and the expected value of the probability distribution function is calculated. This expected value represents the centroid of the photon arrival time. The expected value is then converted into a depth value and arranged into a two-dimensional matrix according to the pixel coordinates to obtain a multi-frame single-photon range map. This process uses the probability distribution function and the expected value to extract the centroid and convert the recording time into a depth value, thus obtaining a multi-frame single-photon range map in two-dimensional matrix form.

[0031] Specifically, shallow spatiotemporal features are extracted from multi-frame single-photon distance maps, including: Using multi-frame single-photon distance maps as input tensors, shallow spatiotemporal features are extracted from the input tensors using 3D convolutional layers. Reflection padding is employed to maintain dimensionality, and the data is then processed using a linear rectified activation function.

[0032] Using multi-frame single-photon distance maps as input tensors, shallow spatiotemporal features are extracted from the input tensors using 3D convolutional layers. Reflection padding is employed to maintain dimensionality, followed by processing with a linear rectified activation function. This process utilizes 3D convolutional layers to simultaneously process data in both spatial and temporal dimensions, preserving spatiotemporal integrity while extracting shallow spatiotemporal features.

[0033] Specifically, high-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal characteristics, including: By using multiple cascaded residual feature attention modules, high-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal features, and high-level feature descriptions are obtained using global average pooling. Each residual feature attention module includes a first three-dimensional convolutional layer, a first linear rectified activation function layer, a second three-dimensional convolutional layer, and a feature attention module connected in sequence. The feature attention module includes a global pooling layer, a third three-dimensional convolutional layer, a second linear rectified activation function layer, a fourth three-dimensional convolutional layer, and a sigmoid activation function layer connected in sequence.

[0034] Specifically, for multi-frame single-photon distance maps with high-level features, the temporal dimension is gradually reduced, and then upsampling is performed through sub-pixel convolution to obtain a high-resolution feature map that integrates spatiotemporal detail information, including: The temporal dimension of the multi-frame single-photon distance map with high-level features is gradually reduced by repeating the temporal reduction module multiple times. The temporal reduction module includes a residual feature attention module, a temporal reduction convolutional layer, and a linear rectified activation function layer connected in sequence. The temporal dimension is reduced to one by the temporal reduction convolutional layer, while the number of channels is expanded at the same time. Then, sub-pixel convolution is used for upsampling to generate a high-resolution spatial feature map that integrates spatiotemporal details.

[0035] The temporal dimension of multi-frame single-photon distance maps with high-level features is gradually reduced by a temporal reduction module. The temporal dimension is further reduced by a temporal reduction convolutional layer while expanding the number of channels, and then upsampling is performed through subpixel convolution. This process reduces computation while preserving important spatiotemporal information during the gradual reduction of the temporal dimension. After upsampling through subpixel convolution, a high-resolution feature map incorporating spatiotemporal details is directly generated.

[0036] Specifically, multi-frame single-photon distance maps with shallow spatiotemporal features are weighted and fused, and then upsampled through sub-pixel convolution to obtain a high-resolution feature map based on multi-frame weighted averaging, including: A residual temporal attention mechanism combining global residual paths is adopted to achieve weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features. The residual temporal attention mechanism is executed by a residual temporal attention module, which includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, and a temporal attention module connected in sequence. The temporal attention module includes a global pooling layer, a third two-dimensional convolutional layer, a linear rectified activation function layer, a fourth two-dimensional convolutional layer, and a sigmoid activation function layer connected in sequence. The fourth two-dimensional convolutional layer of the residual temporal attention module expands the number of channels and then upsamples through sub-pixel convolution to generate a high-resolution spatial feature map based on multi-frame weighted averaging.

[0037] A residual temporal attention mechanism combining global residual paths is employed to achieve weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features through a temporal attention module. The residual temporal attention module expands the number of channels and then performs upsampling through sub-pixel convolution. This process utilizes the temporal attention module to calculate the weighted fused data, and after upsampling through sub-pixel convolution, directly generates a high-resolution feature map based on multi-frame weighted averaging.

[0038] Example 2 This embodiment provides a single-photon imaging super-resolution reconstruction method based on spatiotemporal subscanning and residual attention neural networks, such as... Figure 2 As shown, it includes the following steps: Step 1: Train the residual attention neural network.

[0039] Step 2: Data acquisition of the target is performed using a spatiotemporal joint subscan method.

[0040] Step 3: Preprocess the acquired data to obtain multi-frame single-photon distance maps.

[0041] Step 4: Use a 3D convolutional layer to extract shallow spatiotemporal features from multi-frame single-photon distance maps, while maintaining the integrity of the spatiotemporal dimensions.

[0042] Step 5: High-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal features through multiple cascaded residual feature attention modules, and feature descriptions are obtained using global average pooling.

[0043] Step 6: The temporal dimension of the multi-frame single-photon distance map with high-level features is gradually reduced using a time reduction module. Key spatiotemporal information is preserved through an attention mechanism. The map is then upsampled through subpixel convolution to output a high-resolution feature map that fuses spatiotemporal details.

[0044] Step 7: Employ a temporal attention mechanism that combines global residual paths to perform weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features. Then, perform upsampling through sub-pixel convolution to output a high-resolution feature map based on multi-frame weighted averaging.

[0045] Step 8: Add the corresponding elements of the two high-resolution feature maps obtained in Step 6 and Step 7 to obtain the super-resolution depth map.

[0046] In step one, the NYUV2 dataset (New York University Deep Dataset V2) is selected as the training dataset. Residual feature attention modules are used to predict values, multi-scale composite loss is calculated, gradients are computed, and the Adam (Adaptive Moment Estimation) optimizer is used to update network parameters. The learning rate is adjusted according to a cosine annealing strategy. Model parameters are saved, and the validation set loss is monitored. Training is stopped if there is no improvement after 10 consecutive rounds. The specific process is as follows: Given a low-resolution image and corresponding high-resolution real images After processing:

[0047] in, H For height, W For width, For the number of channels, The height of the high-resolution image, The width of the high-resolution image, The mean of the training set, The standard deviation of the training set. The input image is the standardized version.

[0048] Forward propagation process: Calculation of the output of the residual attention neural network:

[0049] in, To predict the output, For deep neural networks that incorporate residual feature attention mechanisms, Network parameters; residual feature attention block calculation (based on the first...) (For example, a layer)

[0050] in, For the first The feature map obtained from the layer output, For the first The feature map obtained from the layer output, For the first Layer attention function, For the first Layer feature transformation function.

[0051] Loss function calculation: Multi-scale composite loss function:

[0052] Content loss includes:

[0053] Perceived loss:

[0054] Edge preservation loss:

[0055] in, For multi-scale composite loss, For content loss, In order to perceive loss, To preserve loss at the edge, For the summation index, represents the first... One sample, For the first One sample, For the pre-trained network Layer feature extractor, For gradient operators, , and These are the loss weighting coefficients (1.0, 0.1, and 0.05).

[0056] Optimizer and Parameter Updates: Adam Optimizer Update Rules:

[0057] in,

[0058]

[0059]

[0060]

[0061] in, This represents the current number of training steps. For the first Parameters (such as weights and biases) before training. For the first The parameters after training are For the first The first-order moment estimate after bias correction. For the first The second-order moment estimate after bias correction. For the first First-moment estimate without bias correction at step time For the first First-moment estimate without bias correction at step time For the first Second-order moment estimation without bias correction at step time For the first Second-order moment estimation without bias correction at step time The learning rate (initial value is) ), , The momentum parameters are 0.9 and 0.999. Numerical stability constant ( ), This is the current gradient. The training cycle is 500 epochs, the batch size is 16, and the initial learning rate is... Learning rate scheduling: cosine annealing, weight decay: Learning rate scheduling formula:

[0062] in, For the first The learning rate of the step; Minimum learning rate 10 -6 ; Maximum learning rate 10 -3 , This represents the total number of training steps. Finally, when the validation set loss no longer decreases for 10 consecutive training epochs, training is stopped and the model parameters with the best performance on the validation set are selected.

[0063] In this embodiment, the existing nyuv2 dataset is converted into a single-photon time-series detection dataset that conforms to the actual detector acquisition characteristics using a forward model. The single-photon time-series detection dataset is then processed using the centroid method to form a training dataset.

[0064] The forward model can simulate the entire process of photons being emitted from a laser, reflected by the scene, and received by a detector.

[0065] First, for each spatial location in the scene to be transformed Its corresponding state is determined by the depth value. This is determined by both the surface reflectivity and other properties. When the pulsed laser source emits a period of... short light pulses When a photon illuminates a specific spatial location, its round-trip travel time determines the position of the echo signal on the time axis. This is based on the speed of light. The time delay of the echo signal at this location relative to the transmission time is... Defined as: .

[0066] Based on this, a mathematical model is constructed to describe the photon flux at this location within a specific time interval. Let... For the duration of the time interval, at the 1st Within a time interval, the location Theoretical photon flux Given by the following formula:

[0067] in, Indicates the time interval , The photon flux reaching the detector This represents a set of functions that encapsulate distance attenuation, scene reflectivity, and bidirectional reflectance distribution functions. Indicates that it is composed of optical frequencies Photon flux caused by ambient light This represents the integral.

[0068] Subsequently, considering the photoelectric conversion characteristics of the detector itself, the photon flux reaching the detector is further affected by the detector parameters. The detector's quantum efficiency... This describes the probability of an incident photon being successfully converted into an electrical signal, while the detector itself contains an inherent dark counter. Therefore, the detector responds to The number of photons measured during each light pulse illumination cycle can be represented by a time histogram. This measurement is modeled as a Poisson process with a time-varying arrival function, and its probability distribution characteristics are as follows:

[0069] in, Represents the Poisson distribution. Indicates the time interval Inside, location exist The time histogram count values ​​obtained from each cumulative exposure are statistically analyzed. Indicates the spatial location of the single-photon avalanche diode. For the detection wavelength The effective photon detection efficiency can be achieved by using the Poisson sampling process to transform the continuous photon flux model into discrete binary photon counting data containing shot noise characteristics, thereby completing the conversion from the NYUV2 dataset to the single-photon time-series detection dataset.

[0070] The single-photon time-series detection dataset was processed using the centroid method to form a training dataset, which includes low-resolution images. and corresponding high-resolution real images .

[0071] The specific process of step two is as follows: First, a spatial sub-scan is performed: each pixel (the detector's smallest resolving unit) is divided into K×K (4×4 in this embodiment) sub-regions. At the start of the scan, the scanning system is in its initial position, and the entire field of view is scanned once from the initial position to acquire image data of all pixels in the first sub-region. Subsequently, the scanning system moves in the horizontal (X) and vertical (Y) directions, with each movement step being 1 / K of the physical pixel size. Every time the scanning system moves to a new position, the entire field of view is scanned again, thereby simultaneously acquiring image data of all pixels in the new sub-region position, until the scanning of all sub-region positions is completed, resulting in K*K image data, which are used as spatial image data.

[0072] Then perform a time subscan: During the spatial subscan, based on the time resolution of the scanning system, ts (the minimum resolvable time interval of the detector, which is 1 ns), the time is subdivided by time delay. Each time the time is delayed by ts / n (n is the time subscan subdivision multiple, which is 10), the time recording data is obtained.

[0073] By combining spatial image data with time-recorded data, subscan data is obtained.

[0074] The specific process of step three is as follows: In subscan data, the photon histogram of each pixel can be viewed as a sampled estimate of a probability distribution function. Let the true photon arrival time probability density function be given, then the observed histogram is:

[0075] in, For histograms, For the total photon count, This is a discrete time index, identifying which time slot in the histogram, which can be understood as the t-th time slot in the photon histogram. For the first A time capsule For system time resolution, This represents the actual photon arrival time probability density function. To indicate time Accumulate points. It is a continuous time variable. To measure the noise term, the histogram is converted into a probability distribution function:

[0076] in, The count of signal photons recorded in the t-th time chamber. The sum of signal photon counts from time slot 1 to time slot Tb, where Tb is the total number of time slots, or the total number of segments into which the time axis is discretized. t All satisfied , , For photons in the first t The conditional probability of a time capsule arriving, for Its mathematical expectation value for:

[0077] Here, the expected value represents the centroid of the photon's arrival time, which is then converted into a depth value. :

[0078] in, At the speed of light, This refers to the width of the time bay.

[0079] The depth values ​​calculated from the photon histogram of each pixel can be arranged into a two-dimensional matrix according to the pixel coordinates to obtain a multi-frame single-photon distance map.

[0080] The specific process of step four is as follows: Using multi-frame single-photon distance maps as input tensors Using a 3D convolutional layer with a filter size of 3×3×3, from the input tensor Extract F Given a shallow spatiotemporal feature, let the number of output feature channels be... F The process is represented as:

[0081] in, This is a 3D convolution operation. Shape H × W × T × C Output feature tensor Shape H × W × T × F Reflective filling is used to maintain the same dimension. H For height, W For width, T This refers to the number of time frames (number of images). C For the number of channels, F The number of feature channels; It is a shallow spatiotemporal feature. The ReLU activation function is used, and the convolutional kernel weights are initialized using He initialization (He Kaiming).

[0082] in, The weight matrix to be initialized. It is a function of the normal distribution. This represents the number of input channels in the l-th layer.

[0083] The specific process of step five is as follows: High-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal features through 12 cascaded residual feature attention modules. High-level feature descriptions are obtained by global average pooling. Each residual feature attention module includes a first three-dimensional convolutional layer, a ReLU activation function layer, a second three-dimensional convolutional layer, and a feature attention module connected in sequence. The feature attention module includes a global pooling layer, a third three-dimensional convolutional layer, a ReLU activation function layer, a fourth three-dimensional convolutional layer, and a Sigmoid activation function layer connected in sequence.

[0084] The specific operation is as follows: The shallow spatiotemporal features of multi-frame single-photon distance maps are used as input features. The input features are... ,but: Feature extraction path:

[0085] Attention weight calculation:

[0086] Final output:

[0087] in, F＇ For the characteristic tensor, The input features have a dimension of . ; H For height, W For width, T This refers to the number of time frames (number of images).F The number of feature channels; , , and The kernel weight matrix; , , , For bias terms; To modify the activation function of the linear unit; For global average pooling, It is a Sigmoid activation function. For exponential operations, Y This is the final output, i.e., high-level features.

[0088] The specific process of step six is ​​as follows: The time reduction module consists of a residual feature attention module, a time-reduced convolutional layer, and a ReLU activation function layer connected in sequence, repeated... Next, gradually reduce the time dimension, until the last... The Time Reduction-Convolution layer reduces the time dimension to 1 while simultaneously expanding the number of channels. Where s is the upsampling factor, The number of channels is then used for upsampling via subpixel convolution to directly generate a high-resolution spatial feature map that fuses spatiotemporal detail information. This reduces computational load while preserving important spatiotemporal information.

[0089] The specific operation is represented as follows: the input feature is... The number of time reduction modules is :

[0090] The calculation for each time reduction module is as follows:

[0091]

[0092] Calculated by subpixel convolution upsampling:

[0093] in, F＇ For the characteristic tensor, The size of the temporal convolution kernel. The time dimension of the input features, To reduce the number of module repetitions, RFAB For residual feature attention module, To reduce the time complexity of convolutional layers, The weight matrix has dimensions s×s×C× ; The spatial coordinates of the output image; These are the relative coordinates within the weight matrix; For output channel index; This is the upsampling factor; , For adaptive offset; The starting x-coordinate of the current pixel block in the high-resolution image; This represents the starting ordinate of the current pixel block in the high-resolution image; This refers to the number of channels (a high-level feature). Y For upsampling input, X This is a high-resolution output feature map of the fused spatiotemporal details after upsampling.

[0094] The specific process of step seven is as follows: The residual temporal attention module employs a residual temporal attention mechanism that incorporates the global residual path. This module comprises a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, and a temporal attention module, all connected sequentially. The temporal attention sub-module includes a global pooling layer, a third two-dimensional convolutional layer, a ReLU activation function layer, a fourth two-dimensional convolutional layer, and a Sigmoid activation function layer, all connected sequentially. This enables weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features. The fourth two-dimensional convolutional layer of the residual temporal attention module expands the number of channels to... Then, upsampling is performed through subpixel convolution to directly generate a high-resolution spatial feature map based on multi-frame weighted averaging. .

[0095] The specific operation is as follows: The shallow spatiotemporal features of multi-frame single-photon distance maps are used as input features. The input features are... The feature extraction path is as follows:

[0096] Calculation of temporal attention weights:

[0097] The final output is:

[0098] Calculated by subpixel convolution upsampling:

[0099] in, F＇ The input features are a feature tensor with dimensions H×W×. ; H For height,W For width, T This refers to the number of time frames (number of images). F The number of feature channels; , , and It is a 2D convolution kernel weight matrix; , , , For 2D convolution bias terms; This is the temporal attention weight matrix; To modify the activation function of the linear unit; It is the Sigmoid activation function. The weight matrix has dimensions s×s×C× ; The spatial coordinates of the output image; These are the relative coordinates within the weight matrix; For output channel index; This is the upsampling factor; , For adaptive offset; The starting x-coordinate of the current pixel block in the high-resolution image; This represents the starting ordinate of the current pixel block in the high-resolution image; Y is the number of channels (shallow spatiotemporal features), Y is the input for upsampling, and X is the high-resolution output feature map based on multi-frame weighted averaging after upsampling.

[0100] The specific process of step eight is as follows: Add the corresponding elements of the high-resolution feature maps obtained in step six and step seven.

[0101]

[0102] in, The system consists of multi-frame single-photon distance maps with shallow spatiotemporal features extracted, followed by high-resolution feature maps with fused spatiotemporal detail information obtained through steps five and six. Through complex convolution and attention mechanisms, rich spatiotemporal detail information is extracted and fused from multiple images. This provides a benchmark high-resolution estimate based on multi-frame weighted averaging, directly applied to the high-resolution feature map obtained in step seven, which is derived from the multi-frame single-photon distance map after shallow spatiotemporal feature extraction and then processed. For the final output, that is This is the final super-resolution reconstruction result, i.e., the super-resolution depth map.

[0103] like Figure 3As shown, the residual attention neural network described in this method includes a spatiotemporal joint subscan module (which contains a spatial subscan module and a temporal subscan module), a preprocessing module, a three-dimensional convolutional layer, a stack of residual feature attention modules, a temporal reduction module, and a residual temporal attention module.

[0104] The spatial subscan module and the temporal subscan module are combined in parallel to form the spatiotemporal joint subscan module. The output of the spatiotemporal joint subscan module is connected to the preprocessing module to obtain multi-frame single-photon range maps. The output of the preprocessing module is connected to a 3D convolutional layer to extract shallow spatiotemporal features. After shallow spatiotemporal feature extraction, the output is divided into two parallel paths.

[0105] The first path connects the shallow spatiotemporal features to a stack of residual feature attention modules, which consists of N cascaded residual feature attention modules. Each residual feature attention module contains a first 3D convolutional layer, a ReLU (Rectified Linear Unit) activation function layer, a second 3D convolutional layer, and a feature attention module, connected sequentially. The output of the stacked residual feature attention modules is connected to a temporal reduction module. The temporal reduction module is composed of repeated residual feature attention modules and temporally reduced convolutional layers, and finally, the 3D convolutional output is upsampled to a sub-pixel convolution to obtain a high-resolution feature map incorporating spatiotemporal details.

[0106] The second path combines shallow spatiotemporal features with a global residual path connected to a residual temporal attention module. This residual temporal attention module consists of a first two-dimensional convolutional layer, a ReLU activation function layer, a second two-dimensional convolutional layer, and the temporal attention module connected sequentially. The output of the residual temporal attention module is connected to a fourth two-dimensional convolutional layer, and then to a sub-pixel convolutional layer for upsampling, resulting in a high-resolution feature map based on a multi-frame weighted average.

[0107] The high-resolution feature map fused with spatiotemporal details output from the first path and the high-resolution feature map based on multi-frame weighted averaging output from the second path are connected to the addition node, and corresponding elements are added together to finally output a super-resolution depth map.

[0108] Example 3: In this embodiment, a single-photon imaging super-resolution reconstruction system is provided. This single-photon imaging super-resolution reconstruction system can be used to implement the above-mentioned single-photon imaging super-resolution reconstruction method. Specifically, the single-photon imaging super-resolution reconstruction system includes a sub-scan data acquisition module, a multi-frame single-photon distance map acquisition module, a shallow spatiotemporal feature module, a high-level feature extraction module, a fusion spatiotemporal detail information module, a multi-frame weighted averaging module, and a super-resolution depth map acquisition module.

[0109] The subscan data acquisition module is used to acquire subscan data of the target after spatial and temporal subscans.

[0110] The multi-frame single-photon distance map acquisition module is used to obtain multi-frame single-photon distance maps based on subscan data.

[0111] The shallow spatiotemporal feature module is used to extract shallow spatiotemporal features from multi-frame single-photon distance maps.

[0112] The high-level feature extraction module is used to extract high-level features from multi-frame single-photon distance maps with shallow spatiotemporal features.

[0113] The module for fusing spatiotemporal detail information is used to gradually reduce the temporal dimension of multi-frame single-photon distance maps with high-level features, and then upsample them through sub-pixel convolution to obtain a high-resolution feature map that fuses spatiotemporal detail information.

[0114] The multi-frame weighted averaging module is used to perform weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features, and then upsampling is performed through sub-pixel convolution to obtain a high-resolution feature map based on multi-frame weighted averaging.

[0115] The super-resolution depth map acquisition module is used to add corresponding elements of the high-resolution feature map that fuses spatiotemporal detail information and the high-resolution feature map based on multi-frame weighted averaging to obtain the reconstructed super-resolution depth map.

[0116] Example 4: In this embodiment, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this embodiment can be used in the operation of a single-photon imaging super-resolution reconstruction method, including: acquiring sub-scan data of the target after spatial and temporal sub-scans; obtaining multi-frame single-photon range maps based on the sub-scan data; and extracting shallow temporal data from the multi-frame single-photon range maps. The process involves: extracting high-level features from multi-frame single-photon distance maps with shallow spatiotemporal features; progressively reducing the temporal dimension of the multi-frame single-photon distance maps with high-level features and then upsampling them through subpixel convolution to obtain a high-resolution feature map that integrates spatiotemporal details; performing weighted fusion on the multi-frame single-photon distance maps with shallow spatiotemporal features and then upsampling them through subpixel convolution to obtain a high-resolution feature map based on a multi-frame weighted average; and adding corresponding elements of the high-resolution feature map that integrates spatiotemporal details and the high-resolution feature map based on the multi-frame weighted average to obtain the reconstructed super-resolution depth map.

[0117] Example 5: This embodiment provides a computer-readable storage medium (Memory), which is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device.

[0118] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the single-photon imaging super-resolution reconstruction method in the above embodiments. One or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps: acquiring sub-scan data of the target after spatial and temporal sub-scans; obtaining multi-frame single-photon range maps based on the sub-scan data; extracting shallow spatiotemporal features of the multi-frame single-photon range maps; performing high-level feature extraction on the multi-frame single-photon range maps with shallow spatiotemporal features; gradually reducing the temporal dimension of the multi-frame single-photon range maps with high-level features, and then upsampling through subpixel convolution to obtain a high-resolution feature map that fuses spatiotemporal detail information; performing weighted fusion on the multi-frame single-photon range maps with shallow spatiotemporal features, and then upsampling through subpixel convolution to obtain a high-resolution feature map based on multi-frame weighted average; adding corresponding elements of the high-resolution feature map that fuses spatiotemporal detail information and the high-resolution feature map based on multi-frame weighted average to obtain the reconstructed super-resolution depth map.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0120] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0124] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

[0125] It should be understood that the above description is for illustrative purposes and not for limitation. Many embodiments and applications beyond the provided examples will become apparent to those skilled in the art upon reading the above description.

Claims

1. A method for super-resolution reconstruction of single-photon imaging, characterized in that, Includes the following processes: Acquire subscan data of the target after spatial and temporal subscanning; Multi-frame single-photon range maps were obtained based on subscan data, including: Convert the photon histogram of each pixel in the subscan data into a probability distribution function; Calculate the expected value of the probability distribution function, where the expected value represents the centroid of the photon arrival time; Convert the mathematical expectation value into a depth value; The depth values ​​calculated from the photon histogram of each pixel are arranged into a two-dimensional matrix according to the pixel coordinates to obtain a multi-frame single-photon distance map; Extracting shallow spatiotemporal features from multi-frame single-photon distance maps; High-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal characteristics, including: By using multiple cascaded residual feature attention modules, high-level feature extraction is performed on multi-frame single-photon distance maps with shallow spatiotemporal features, and high-level feature descriptions are obtained by global average pooling. Each residual feature attention module includes a first three-dimensional convolutional layer, a first linear rectified activation function layer, a second three-dimensional convolutional layer, and a feature attention module connected in sequence; The feature attention module consists of a global pooling layer, a third 3D convolutional layer, a second linear rectified activation function layer, a fourth 3D convolutional layer, and a sigmoid activation function layer connected in sequence. For multi-frame single-photon distance maps with high-level features, the temporal dimension is progressively reduced, and then upsampling is performed through sub-pixel convolution to obtain a high-resolution feature map that integrates spatiotemporal details, including: The temporal dimension of the multi-frame single-photon distance map with high-level features is gradually reduced by repeating the time reduction module multiple times. The time reduction module includes a residual feature attention module, a time-reduced convolutional layer, and a linear rectified activation function layer connected in sequence. The temporal dimension is reduced to one by using time-shortened convolutional layers, while the number of channels is increased at the same time; Then, upsampling is performed through subpixel convolution to generate a high-resolution spatial feature map that integrates spatiotemporal detail information; A weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features is performed, followed by upsampling via sub-pixel convolution to obtain a high-resolution feature map based on multi-frame weighted averaging, including: A residual temporal attention mechanism combining global residual paths is adopted to achieve weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features; The residual temporal attention mechanism is executed using a residual temporal attention module, which consists of a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, and a temporal attention module connected in sequence. The temporal attention module consists of a global pooling layer, a third two-dimensional convolutional layer, a linear rectified activation function layer, a fourth two-dimensional convolutional layer, and a sigmoid activation function layer connected in sequence. The fourth two-dimensional convolutional layer of the residual temporal attention module expands the number of channels, and then upsamples through sub-pixel convolution to generate a high-resolution spatial feature map based on multi-frame weighted averaging. The high-resolution feature map that integrates spatiotemporal details and the high-resolution feature map based on multi-frame weighted averaging are added element by element to obtain the reconstructed super-resolution depth map.

2. The single-photon imaging super-resolution reconstruction method according to claim 1, characterized in that, Acquire subscan data of the target after spatial and temporal subscanning, including: Spatial subscan: Dividing each pixel into K The system is divided into K sub-regions. At the start of the scan, the scanning system first scans the entire field of view from the initial position, acquiring image data of all pixels in the first sub-region. Then, the scanning system moves horizontally and vertically, with each movement taking a step of 1 / K of the physical pixel size. Each time the system moves to a new position, it scans the entire field of view again, simultaneously acquiring image data of all pixels in the new sub-region, until all sub-regions have been scanned, yielding K... K image data points are used as spatial image data; Temporal subscan: During spatial subscan, based on the temporal resolution of the scanning system, the time is subdivided by time delay, and the time is recorded based on the preset time for each delay to obtain time record data; By combining spatial image data with time-recorded data, subscan data is obtained.

3. The single-photon imaging super-resolution reconstruction method according to claim 1, characterized in that, Extracting shallow spatiotemporal features from multi-frame single-photon range maps, including: Using multi-frame single-photon distance maps as input tensors, shallow spatiotemporal features are extracted from the input tensors using 3D convolutional layers.

4. A single-photon imaging super-resolution reconstruction system based on the method of claim 1, characterized in that, include: The subscan data acquisition module is used to acquire subscan data of the target after spatial and temporal subscans. A multi-frame single-photon range map acquisition module is used to obtain multi-frame single-photon range maps based on subscan data; The shallow spatiotemporal feature module is used to extract shallow spatiotemporal features from multi-frame single-photon distance maps; The high-level feature extraction module is used to extract high-level features from multi-frame single-photon distance maps with shallow spatiotemporal features. The module for fusing spatiotemporal detail information is used to gradually reduce the temporal dimension of multi-frame single-photon distance maps with high-level features, and then upsample them through sub-pixel convolution to obtain a high-resolution feature map that fuses spatiotemporal detail information. The multi-frame weighted averaging module is used to perform weighted fusion of multi-frame single-photon distance maps with shallow spatiotemporal features, and then upsampling is performed through sub-pixel convolution to obtain a high-resolution feature map based on multi-frame weighted averaging. The super-resolution depth map acquisition module is used to add corresponding elements of the high-resolution feature map that integrates spatiotemporal detail information and the high-resolution feature map based on multi-frame weighted averaging to obtain the reconstructed super-resolution depth map.

5. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the single-photon imaging super-resolution reconstruction method as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the single-photon imaging super-resolution reconstruction method as described in any one of claims 1 to 3.

Images