Depth detection method, apparatus, depth detection system, and electronic device
By using a noise reduction model with an encoder-decoder structure to process the photon statistical graph, the ranging failure problem of dToF ranging in special scenarios is solved, and higher-precision depth detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XG TECHNOLOGIES PTE LTD
- Filing Date
- 2026-03-27
- Publication Date
- 2026-07-14
AI Technical Summary
Existing dToF ranging technology has difficulty accurately detecting distances in special scenarios such as corners of space and edges of objects, leading to ranging failures.
A pre-trained denoising model based on an encoder-decoder structure is used to process the photon statistical map through an encoder, a bottleneck layer, and a decoder, extracting multi-scale features and suppressing noise interference to generate a more accurate depth map.
It effectively suppresses noise and multipath interference, improving the accuracy of depth maps, especially the ranging accuracy at spatial corners and object edges.
Smart Images
Figure CN122386327A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to computer vision technology, and in particular to a depth detection method, apparatus, depth detection system and electronic device. Background Technology
[0002] dToF (direct Time-of-Flight) ranging is a distance measurement technology based on the time of flight of photons. Its core principle is to calculate the distance by calculating the time difference between the transmission and reception of light signals.
[0003] The core of dToF ranging is the construction and analysis of histograms. For each pixel in the measured scene, the system performs thousands to millions of "emission-detection" loops. Each time a photon is detected, its flight time t is recorded and accumulated in the corresponding time unit. Ultimately, each pixel obtains a time-photon count histogram. After obtaining the histogram, the dToF ranging process cannot accurately detect distances in some special scenarios, such as corners of space or object edges, resulting in ranging failures. Summary of the Invention
[0004] To address the aforementioned technical problems, this disclosure provides a depth detection method, apparatus, system, and electronic device to improve ranging accuracy and scene adaptability.
[0005] A first aspect of this disclosure provides a depth detection method, comprising: acquiring a denoised photon statistical map generated for a target scene, wherein for any pixel in the denoised photon statistical map, the pixel corresponds to a photon statistical data sequence, the photon statistical data in the photon statistical data sequence being used to characterize the number of reflected photons received from a target location at a corresponding time, the target location being the position in the target scene corresponding to the pixel; encoding the denoised photon statistical map using an encoder included in a pre-trained denoising model to obtain encoded data; calculating the encoded data using a bottleneck layer included in the denoising model to obtain data to be decoded; decoding the data to be decoded using a decoder included in the denoising model to obtain a denoised photon statistical map; and generating a depth map corresponding to the target scene based on the peak data corresponding to each pixel in the denoised photon statistical map.
[0006] A second aspect of this disclosure provides a depth detection device, comprising: an acquisition module, configured to acquire a denoised photon statistical map generated for a target scene, wherein for any pixel in the denoised photon statistical map, the pixel corresponds to a photon statistical data sequence, the photon statistical data in the photon statistical data sequence being used to characterize the number of reflected photons received from a target location at a corresponding time, the target location being the position in the target scene corresponding to the pixel; an encoding module, configured to encode the denoised photon statistical map using an encoder included in a pre-trained denoising model to obtain encoded data; a calculation module, configured to calculate the encoded data using a bottleneck layer included in the denoising model to obtain data to be decoded; a decoding module, configured to decode the data to be decoded using a decoder included in the denoising model to obtain a denoised photon statistical map; and a generation module, configured to generate a depth map corresponding to the target scene based on the peak data corresponding to each pixel in the denoised photon statistical map.
[0007] A third aspect of this disclosure provides a depth detection system, comprising: a photon time-of-flight detection device and a controller, the photon time-of-flight detection device and the controller being communicatively connected; the photon time-of-flight detection device is used to collect the number of reflected photons in a target scene and generate a statistical map of photons to be denoised based on the number of reflected photons; the controller is used to execute the aforementioned depth detection method based on the statistical map of photons to be denoised.
[0008] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed, implements the depth detection method described above.
[0009] A fifth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the depth detection method described above.
[0010] A sixth aspect of this disclosure provides a computer program product including computer program instructions that, when executed by an instruction processor, perform the aforementioned depth detection method.
[0011] Based on the depth detection method, apparatus, system, and electronic device provided in this disclosure, a denoising photon statistical map is generated for the target scene. A pre-trained denoising model, including an encoder, bottleneck layer, and decoder, is used to process the denoising photon statistical map to obtain a denoised photon statistical map. Finally, based on the peak data corresponding to each pixel in the denoised photon statistical map, a depth map corresponding to the target scene is generated. This disclosure's denoising model, based on an encoder-decoder structure, can fully extract multi-scale features of the denoised photon statistical map while retaining detailed information. It effectively suppresses noise and corrects errors caused by multipath interference of photons, thereby eliminating interference signals generated at various locations such as corners and object edges in space, and more accurately denoising the photon statistical map, thus improving the accuracy of generating a depth map corresponding to the target scene. Attached Figure Description
[0012] Figure 1 This is a schematic flowchart of a depth detection method provided in an exemplary embodiment of this disclosure;
[0013] Figure 2 This is a schematic diagram of the structure of a noise reduction model provided in an exemplary embodiment of this disclosure;
[0014] Figure 3 This is a schematic diagram comparing histograms before and after noise reduction provided in an exemplary embodiment of this disclosure;
[0015] Figure 4 This is a flowchart illustrating a depth detection method provided in another exemplary embodiment of this disclosure;
[0016] Figure 5 This is a schematic flowchart of a depth detection method provided in yet another exemplary embodiment of this disclosure;
[0017] Figure 6 This is a schematic diagram of the training process of a noise reduction model provided in an exemplary embodiment of this disclosure;
[0018] Figure 7 This is a schematic diagram of the training process of a noise reduction model provided in another exemplary embodiment of this disclosure;
[0019] Figure 8 This is a structural diagram of a depth detection device provided in an exemplary embodiment of the present disclosure;
[0020] Figure 9 This is a structural diagram of a depth detection device provided in another exemplary embodiment of this disclosure;
[0021] Figure 10 This is a structural diagram of a depth detection system provided in an exemplary embodiment of this disclosure;
[0022] Figure 11This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation
[0023] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.
[0024] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of this disclosure.
[0025] Application Overview
[0026] Related dToF-based ranging technologies may fail to accurately detect distances in certain special scenarios, such as corners of space or edges of objects, due to the high noise in the collected data, making it impossible to extract detailed features of the spatial location.
[0027] To address the aforementioned issues, this embodiment employs a pre-trained denoising model based on a codec structure. This model can fully extract multi-scale features from the photon statistical map to be denoised while preserving detailed information. It effectively suppresses noise and corrects errors caused by multipath interference during photon flight, thereby eliminating interference signals generated at various locations such as corners and object edges in space. This allows for more accurate denoising of the photon statistical map, thereby improving the accuracy of generating the depth map corresponding to the target scene.
[0028] Exemplary methods
[0029] Figure 1 This is a schematic flowchart of a depth detection method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 1 As shown, it includes the following steps:
[0030] Step 101: Obtain the photon statistics map to be denoised for the target scene.
[0031] The photon statistics map to be denoised can be obtained by a dToF lidar system. The aforementioned photon statistics map can reflect the photon flight time corresponding to each pixel in the target scene mapped onto the two-dimensional image. The photon flight time is the time it takes for a photon to be emitted from the laser emitter, pass through an object in the target scene, and be reflected back to the photoelectric sensor.
[0032] The dToF lidar system operates as follows: A laser emitter emits an extremely short light pulse. A single-photon avalanche diode (SPAD) sensor detects photons reflected back from various objects in the scene. For each detected photon, a time-to-digital converter (TDC) precisely records the time between its emission and reception. The scene is mapped onto a two-dimensional image, and the number of received photons is counted for each pixel at each scanned location, thus obtaining a photon statistics map.
[0033] For any pixel in the photon statistics map to be denoised, the pixel corresponds to a photon statistics data sequence. The photon statistics data in the photon statistics data sequence is used to characterize the number of reflected photons received from the target location at the corresponding time. The target location is the position in the target scene corresponding to the pixel.
[0034] Typically, for any pixel in the photon statistics map to be denoised, the corresponding data sequence can form a time-photon count histogram. For example, the histogram corresponding to one pixel has 512 time points, covering a measurement range of 0 to 100 nanoseconds (corresponding to a distance of 0 to 15 meters). After repeatedly counting the detected photons, the data in the resulting histogram is as follows:
[0035] Time points: [1, 2, 3, 4, 5, ..., 510, 511, 512];
[0036] Photon count: [2, 5, 21, 105, 203, 450, 301, 98, ..., 1, 0, 3].
[0037] The peak value of the histogram corresponds to the photon flight time at the location of that pixel.
[0038] Step 102: Encode the photon statistical map to be denoised using the encoder included in the pre-trained denoising model to obtain coded data.
[0039] like Figure 2As shown, the denoising model includes an encoder 201, a bottleneck layer 202, and a decoder 203. The encoder, bottleneck layer, and decoder can constitute a Unet network. The encoder can include at least one coding block, and each coding block can include at least one convolutional layer, batch normalization layer, and activation function layer. Through the encoder, the photon statistical map to be denoised can be downsampled to obtain a low-resolution feature map as encoded data. For example, the input of the encoder is a photon statistical map to be denoised with a size of 240×96×256, and the output is a feature map with a size of 15×6×512 as encoded data.
[0040] Optionally, the encoder, bottleneck layer, and decoder are connected serially. That is, the bottleneck layer only receives the encoded data output from the encoder, and the encoder only receives the data to be decoded output from the bottleneck layer; there is no skip connection between the encoder and decoder. In a typical Unet network, the encoded blocks (or downsampling layers) in the encoder and the decoder blocks (or upsampling layers) in the decoder correspond one-to-one. For a decoder block, it receives not only the feature map output from the upper-layer decoder block or bottleneck layer but also the feature map output from the corresponding encoded block; that is, there is a skip connection between the encoder and decoder, the purpose of which is to preserve the texture details of the original data during decoding. However, in this embodiment, the processing of the photon statistics map to be denoised aims to accurately find the time-of-flight corresponding to the reflection point from the histogram corresponding to each pixel, and it is not necessary to retain too much information from the original data. Therefore, removing the skip connection structure can reduce redundant calculations and improve data processing efficiency.
[0041] Step 103: Calculate the encoded data using the bottleneck layer included in the noise reduction model to obtain the data to be decoded.
[0042] The bottleneck layer is used to increase the number of channels in the feature map, thereby extracting richer semantic information. For example, the input to the bottleneck layer is a feature map of size 15×6×512, and the output is a feature map of size 15×6×1024 as the data to be decoded.
[0043] Step 104: Using the decoder included in the denoising model, decode the data to be decoded to obtain the denoised photon statistics map.
[0044] The decoder may include at least one decoding block, and each decoding block may include at least one transposed convolutional layer, convolutional layer, batch normalization layer, or other units. The decoder can upsample the data to be decoded to obtain a denoised photon statistical map with the same resolution as the original photon statistical map. For example, the decoder may take a feature map of size 15×6×1024 as input and output a denoised photon statistical map of size 240×96×256.
[0045] like Figure 3As shown, the denoising model performs denoising processing on the data sequence corresponding to a certain pixel in the input (as shown in histogram A) to obtain the denoised data sequence (as shown in histogram B).
[0046] Step 105: Based on the peak data corresponding to each pixel in the denoised photon statistical map, generate a depth map corresponding to the target scene.
[0047] Specifically, since the noise interference is removed from the denoised photon statistical map, the time t_peak for photon reflection corresponding to each pixel can be accurately located from the data sequence corresponding to each pixel in the denoised photon statistical map. The distance value corresponding to the pixel can be calculated using the following formula (1):
[0048] d = (c * t_peak) / 2 (1)
[0049] Where d is the distance value and c is the speed of light.
[0050] After performing the above processing on the data sequence corresponding to each pixel in the denoised photon statistical map, a depth map containing the distance value corresponding to each pixel can be obtained.
[0051] The depth detection method provided in this disclosure generates a denoised photon statistical map for a target scene. A pre-trained denoising model, including an encoder, bottleneck layer, and decoder, processes the denoised photon statistical map to obtain a denoised photon statistical map. Finally, based on the peak data corresponding to each pixel in the denoised photon statistical map, a depth map corresponding to the target scene is generated. This disclosure's denoising model, based on an encoder-decoder structure, can fully extract multi-scale features from the denoised photon statistical map while preserving detailed information. It effectively suppresses noise and corrects errors caused by multipath interference in photon flight, thereby eliminating interference signals generated at various locations such as corners and object edges in space, and more accurately denoising the photon statistical map, thus improving the accuracy of generating a depth map corresponding to the target scene.
[0052] In some alternative implementations, such as Figure 4 As shown, step 101 above includes:
[0053] Step 1011: Obtain the photon statistics of the target scene collected by the photon time-of-flight detection device.
[0054] The photon time-of-flight detection device is used to detect and count the number of photons reflected from each location in the target scene. For example, the photon time-of-flight detection device can be a dToF lidar system, and the method for acquiring the photon statistics chart can refer to the description of step 101 above.
[0055] Step 1012: Perform dimensional transformation and normalization on the photon statistical graph to obtain the photon statistical graph to be denoised.
[0056] The acquired photon statistics map consists of multiple pixels, each corresponding to a photon statistics data sequence, which can form a histogram. This photon statistics data sequence is the raw data obtained by statistically analyzing the number of received photons at each time point. To adapt the data to the data format requirements of the denoising model, the photon statistics map needs to be preprocessed, including dimension transformation and normalization.
[0057] Dimension transformation is used to rearrange the input photon statistics graph into a format suitable for convolution processing. For example, the dimensionality transformation process is shown in equation (2) below:
[0058] (2)
[0059] The Permute function is used to adjust the order of dimensions, and the Reshape function is used to add batch dimensions. Input4D represents converting a three-dimensional photon statistical graph into four-dimensional data that includes batch dimensions.
[0060] The normalization process is shown in equation (3) below:
[0061] (3)
[0062] Where i and j represent pixel coordinates, t represents time coordinates, and ε is a constant used to prevent division by zero errors. norm That is, the normalized statistical diagram of photons to be denoised.
[0063] This embodiment performs dimensional transformation and normalization on the collected photon statistical map to obtain a photon statistical map to be denoised that is suitable for the denoising model, which helps the denoising model to perform denoising processing more efficiently and accurately.
[0064] In some alternative implementations, the encoder includes at least one serially connected downsampling layer, and the decoder includes at least one serially connected upsampling layer and output layer. For example... Figure 2 As shown, the encoder includes four downsampling layers, each of which includes multiple convolutional layers, multiple batch normalization layers, and one pooling layer. The decoder includes four upsampling layers, each of which includes a transposed convolutional layer, multiple convolutional layers, multiple batch normalization layers, and multiple ReLU activation function layers.
[0065] Step 102 above includes:
[0066] By using at least one downsampling layer, the photon statistics map to be denoised is downsampled layer by layer to obtain coded data.
[0067] like Figure 2 As shown, the encoder input is a 240×96×256 photon statistical map to be denoised. After passing through four downsampling layers, a feature map of size 15×6×512 is obtained as the encoding data.
[0068] Step 103 above includes:
[0069] By using at least one upsampling layer, the data to be decoded is upsampled layer by layer to obtain the denoised photon statistics map.
[0070] like Figure 2 As shown, the bottleneck layer takes a 15×6×512 feature map as input. After multiple convolutions and batch normalizations, it outputs a 15×6×1024 feature map as the data to be decoded. The data to be decoded is input to the encoder, where it is upsampled layer by layer by four upsampling layers. Finally, after being processed by convolutions and the Softmax activation function in the output layer of the decoder, a denoised photon statistical map of size 240×96×256 is obtained.
[0071] This embodiment uses a serial processing method in the encoder and decoder, where each downsampling layer and upsampling layer only receives data output from the upper layer network for processing, without the need for cross-layer processing via skip connections. This allows the model's data processing to focus on denoising the photon statistical graph without retaining the original information in the photon statistical graph, thus improving the efficiency and accuracy of denoising the photon statistical graph.
[0072] In some alternative implementations, such as Figure 5 As shown, step 105 includes:
[0073] Step 1051: For any pixel in the denoised photon statistics map, if there is peak data in the denoised photon statistics data sequence corresponding to the pixel, determine the target time point where the peak data corresponding to the pixel is located; based on the target time point, the preset light speed value and the time point width corresponding to the photon statistics map to be denoised, calculate the depth value corresponding to the pixel.
[0074] Specifically, since the denoised photon statistics map has undergone denoising processing, the presence of peak data can be directly determined from the denoised photon statistics data sequence corresponding to each pixel in the denoised photon statistics map. For example, if the difference between the maximum and minimum values (or the mean of data other than the peak value) in the histogram corresponding to the denoised photon statistics data sequence exceeds a preset threshold, it is determined that peak data exists, that is, the maximum value is the peak data. The time point corresponding to the peak data is the target time point. Optionally, the target time point can also be calculated by the centroid method. The method for calculating the target time point corresponding to the peak using the centroid method is shown in the following formula (4):
[0075] (4)
[0076] Output(i,j,t) is the value at time t in the photon statistics sequence corresponding to the pixel at coordinate (i,j).
[0077] The time point width mentioned above refers to the actual duration of two adjacent time points (bins) in the histogram. For example, if a pixel's histogram has 512 time bins, covering a measurement range of 0 to 100 nanoseconds, then the time point width is 100 / 511 nanoseconds.
[0078] Multiplying the target time point by the time width gives the time it takes for a photon to travel from emission to return. Multiplying this by the speed of light gives the distance between the target reflection point and the photon emitting device, which can then be used as a depth value.
[0079] Step 1052: Generate a depth map corresponding to the target scene based on the depth values corresponding to each pixel.
[0080] By combining the obtained depth values, a depth map can be obtained.
[0081] This embodiment realizes the accurate photon round-trip time corresponding to the pixel with peak data based on the photon statistical map after noise reduction, and calculates the depth value based on the time, which can improve the accuracy of the generated depth map.
[0082] In some alternative implementations, step 1051 can be performed by calculating the depth value as follows:
[0083] Determine the time difference between the target time point and the preset time offset; calculate the depth value corresponding to the pixel based on the time difference, the speed of light value, and the width of the time point.
[0084] The aforementioned time offset is the inherent time offset of the electronic device executing this method.
[0085] The formula for calculating the depth value based on the time difference, the speed of light, and the width at a given time point is shown in equation (5) below:
[0086] (5)
[0087] Among them, t offset This represents the time offset. Δt represents the width at that time point.
[0088] This embodiment removes the inherent time offset factors of the device by subtracting the time offset from the target time point, making the calculation of photon round-trip time more accurate, and thus improving the accuracy of the calculated depth value.
[0089] In some alternative implementations, such as Figure 6 As shown, the noise reduction model was pre-trained according to the following steps:
[0090] Step 601: Obtain the sample photon statistics map and the corresponding reference photon statistics map.
[0091] The baseline photon statistical map is a photon statistical map generated by annotating the actual positions of objects within the sample scene. That is, the photon statistical data sequence corresponding to each pixel in the baseline photon statistical map is a photon statistical map reflecting the actual position of the object after it has been calibrated using real objects.
[0092] Step 602: Input the sample photon statistical map into the initial denoising model to obtain the sample denoised photon statistical map.
[0093] The structure of the initial noise reduction model and the data processing procedure can be referenced in the above embodiments.
[0094] Step 603: Based on the preset loss function, determine the error between the sample denoised photon statistical map and the reference photon statistical map.
[0095] The type of loss function can be preset. For example, the loss function can be the mean squared error loss function, which is used to represent the error between the sample denoised photon statistical map and the reference photon statistical map.
[0096] Step 604: Adjust the parameters of the initial noise reduction model based on the error.
[0097] Specifically, machine learning methods can be used to adjust the parameters of the initial denoising model through backpropagation and gradient descent, so that the loss value gradually decreases until convergence.
[0098] Step 605: In response to the initial denoising model after parameter adjustment meeting the preset training termination condition, the current initial square graph denoising model is determined as the trained denoising model.
[0099] Optionally, the training termination conditions mentioned above include, but are not limited to, at least one of the following: loss value convergence, training iterations exceeding a preset number of iterations, and training duration exceeding a preset duration. Typically, training a model requires a large amount of sample data, and a denoising model is obtained through multiple training iterations. Therefore, by using multiple sets of training data and repeatedly iterating through the steps of this embodiment, a trained denoising model can be obtained.
[0100] This embodiment trains a denoising model by pre-collecting sample data. It can specifically train a denoising model that meets the denoising requirements for the actual application scenarios of depth maps, thereby improving the accuracy of denoising photon statistical maps.
[0101] In some alternative implementations, such as Figure 7 As shown, step 603 above includes:
[0102] Step 6031: Based on at least one preset loss function, calculate the loss value of the sample denoised photon statistical map and the reference photon statistical map to obtain at least one loss value.
[0103] Among them, at least one loss function includes at least one of the following: main task loss function, distribution similarity loss function, sparsity loss function, and physical prior constraint loss function.
[0104] The main task loss is primarily used to determine the ranging error, and can be selected from functions such as mean squared error loss function and Huber loss function.
[0105] Distribution similarity loss functions can include KL divergence loss function, maximum mean difference loss function, etc., to reduce data distribution errors.
[0106] Sparsity loss functions can include time-dimensional sparse loss functions, adaptive sparse loss functions, etc., to introduce sparsity constraints and prompt the model to automatically select key features or parameters.
[0107] The physical prior constraint loss function is used to incorporate real-world physical laws into the calculation of the loss value, thereby improving the model's generalization ability, physical consistency, and interpretability.
[0108] Step 6032: Perform a weighted summation on at least one loss value to obtain the error between the sample denoised photon statistical map and the reference photon statistical map.
[0109] The weight corresponding to each loss value can be preset according to actual needs.
[0110] This embodiment uses at least one loss function for model training, which references errors from multiple aspects to adjust model parameters, thereby further improving the data processing accuracy of the trained model and making the denoised photon statistical map output by the model more accurate.
[0111] Exemplary device
[0112] Figure 8 This is a schematic diagram of the structure of a depth detection device provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, such as... Figure 8 As shown, the depth detection device includes:
[0113] The acquisition module 801 is used to acquire the photon statistics map to be denoised generated for the target scene. For any pixel in the photon statistics map to be denoised, the pixel corresponds to a photon statistics data sequence. The photon statistics data in the photon statistics data sequence is used to characterize the number of reflected photons received from the target location at the corresponding time. The target location is the position in the target scene corresponding to the pixel.
[0114] The encoding module 802 is used to encode the photon statistical map to be denoised using the encoder included in the pre-trained denoising model to obtain encoded data.
[0115] The calculation module 803 is used to calculate the encoded data using the bottleneck layer included in the noise reduction model to obtain the data to be decoded;
[0116] The decoding module 804 is used to decode the data to be decoded using the decoder included in the denoising model, and obtain the denoised photon statistics map;
[0117] The generation module 805 is used to generate a depth map corresponding to the target scene based on the peak data corresponding to each pixel in the denoised photon statistical map.
[0118] Reference Figure 9 , Figure 9 This is a schematic diagram of the structure of a depth detection device provided in another exemplary embodiment of this disclosure.
[0119] In some optional implementations, the acquisition module 801 includes: an acquisition unit 8011, used to acquire a photon statistical map of the target scene collected by the photon time-of-flight detection device; and a preprocessing unit 8012, used to perform dimensional transformation and normalization processing on the photon statistical map to obtain a photon statistical map to be denoised.
[0120] In some alternative implementations, the encoder, bottleneck layer, and decoder are connected serially.
[0121] In some alternative implementations, the encoder includes at least one serially connected downsampling layer, and the decoder includes at least one serially connected upsampling layer and output layer; the encoding module 802 is further configured to: use at least one downsampling layer to downsample the photon statistical map to be denoised layer by layer to obtain encoded data; the decoding module 804 is further configured to: use at least one upsampling layer to upsample the data to be decoded layer by layer to obtain the denoised photon statistical map.
[0122] In some optional implementations, the generation module 805 includes: a determining unit 8051, used to determine the target time point where the peak data corresponding to any pixel in the denoised photon statistical map is located if there is peak data in the denoised photon statistical data sequence corresponding to the pixel; and to calculate the depth value corresponding to the pixel based on the target time point, a preset light speed value, and the time point width corresponding to the photon statistical map to be denoised; and a generation unit 8052, used to generate a depth map corresponding to the target scene based on the depth values corresponding to each pixel.
[0123] In some optional implementations, the determining unit 8051 includes: a determining subunit 80511, used to determine the time difference between the target time point and a preset time offset; and a calculating subunit 80512, used to calculate the depth value corresponding to the pixel based on the time difference, the speed of light value, and the width of the time point.
[0124] In some optional implementations, the denoising model is pre-trained according to the following steps: acquiring sample photon statistical maps and corresponding reference photon statistical maps, wherein the reference photon statistical map is generated by annotating the actual positions of objects in the sample scene; inputting the sample photon statistical maps into the initial denoising model to obtain sample denoised photon statistical maps; determining the error between the sample denoised photon statistical maps and the reference photon statistical maps based on a preset loss function; adjusting the parameters of the initial denoising model based on the error; and determining the current initial square map denoising model as the trained denoising model in response to the initial denoising model meeting the preset training termination condition after parameter adjustment.
[0125] In some optional implementations, the error between the denoised photon statistical map of the sample and the reference photon statistical map is determined based on a preset loss function. This includes: calculating the loss value of the denoised photon statistical map of the sample and the reference photon statistical map based on at least one preset loss function to obtain at least one loss value. The at least one loss function includes at least one of the following: main task loss function, distribution similarity loss function, sparsity loss function, and physical prior constraint loss function; and performing a weighted summation of the at least one loss value to obtain the error between the denoised photon statistical map of the sample and the reference photon statistical map.
[0126] The exemplary embodiments of this device correspond to the exemplary method section described above in terms of implementation. The corresponding content between the two can be referenced, combined, and cited, and will not be repeated here. The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section described above, and will not be repeated here.
[0127] Exemplary System
[0128] Figure 10 The present disclosure provides a structural diagram of a depth detection system, which includes a photon time-of-flight detection device 1001 and a controller 1002, and the photon time-of-flight detection device and the controller are communicatively connected.
[0129] The photon time-of-flight detection device is used to collect the number of reflected photons in the target scene and generate a statistical map of photons to be denoised based on the number of reflected photons.
[0130] As an example, a photon time-of-flight detection device could be a dToF lidar, which can emit photons toward a target scene, receive photons reflected by objects within the target scene, and count the number of photons over time.
[0131] The controller is used to perform the aforementioned depth detection method based on the statistical map of photons to be denoised.
[0132] The controller can be used in various application scenarios, such as a controller on a vehicle or a controller for a dedicated ranging device.
[0133] The depth detection system provided in this embodiment, by applying the above-mentioned depth detection method, can fully extract the multi-scale features of the photon statistical map to be denoised and retain detailed information, effectively suppress noise and correct errors caused by multipath interference of photon flight, thereby eliminating interference signals generated at various locations such as corners and object edges in space, and performing more accurate denoising of the photon statistical map, thereby improving the accuracy of generating the depth map corresponding to the target scene.
[0134] Exemplary electronic devices
[0135] Figure 11 A structural diagram of an electronic device 1100 provided in an embodiment of this disclosure includes at least one processor 1101 and a memory 1102.
[0136] The processor 1101 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1100 to perform desired functions.
[0137] The memory 1102 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1101 may execute one or more computer program instructions to implement the depth detection methods and / or other desired functions of the various embodiments of this disclosure described above.
[0138] In one example, the electronic device 1100 may also include an input device 1103 and an output device 1104, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).
[0139] The input device 1103 may also include, for example, a keyboard, a mouse, etc.
[0140] The output device 1104 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.
[0141] Of course, for the sake of simplicity, Figure 11 Only some of the components of the electronic device 1100 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1100 may include any other suitable components depending on the specific application.
[0142] Exemplary computer program products and computer-readable storage media
[0143] In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the depth detection methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.
[0144] Computer program products can be written in any combination of one or more programming languages to perform the operations of embodiments of this disclosure. These programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0145] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the depth detection methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.
[0146] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0147] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.
[0148] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.
Claims
1. A depth detection method, comprising: Obtain a photon statistics map to be denoised generated for the target scene. For any pixel in the photon statistics map to be denoised, the pixel corresponds to a photon statistics data sequence. The photon statistics data in the photon statistics data sequence is used to characterize the number of reflected photons received from the target location at the corresponding time. The target location is the position in the target scene corresponding to the pixel. The encoder included in the pre-trained denoising model is used to encode the photon statistical map to be denoised, thereby obtaining coded data; The bottleneck layer included in the noise reduction model is used to calculate the encoded data to obtain the data to be decoded; The data to be decoded is decoded using the decoder included in the denoising model to obtain a denoised photon statistical map; Based on the peak data corresponding to each pixel in the denoised photon statistical map, a depth map corresponding to the target scene is generated.
2. The method according to claim 1, wherein, The step of obtaining the denoising photon statistical map generated for the target scene includes: Obtain a photon statistical graph of the target scene collected by the photon time-of-flight detection device; The photon statistical graph is subjected to dimensionality transformation and normalization to obtain the photon statistical graph to be denoised.
3. The method according to claim 1, wherein, The encoder, the bottleneck layer, and the decoder are connected serially.
4. The method according to claim 1, wherein, The encoder includes at least one downsampling layer connected in a serial manner, and the decoder includes at least one upsampling layer and an output layer connected in a serial manner; The encoder included in the pre-trained denoising model encodes the photon statistical map to be denoised, obtaining encoded data, including: Using the at least one downsampling layer, the photon statistics map to be denoised is downsampled layer by layer to obtain the encoded data; The step of using the decoder included in the denoising model to decode the data to be decoded to obtain a denoised photon statistical map includes: Using the at least one upsampling layer, the data to be decoded is upsampled layer by layer to obtain the denoised photon statistics map.
5. The method according to claim 1, wherein generating a depth map corresponding to the target scene based on the peak data corresponding to each pixel in the denoised photon statistical map comprises: For any pixel in the denoised photon statistics map, if there is peak data in the denoised photon statistics data sequence corresponding to the pixel, determine the target time point where the peak data corresponding to the pixel is located. Based on the target time point, the preset speed of light value, and the time point width corresponding to the photon statistical map to be denoised, the depth value corresponding to the pixel is calculated. Based on the depth values corresponding to each pixel, a depth map corresponding to the target scene is generated.
6. The method according to claim 5, wherein calculating the depth value corresponding to the pixel based on the target time point, the preset speed of light value, and the time point width corresponding to the photon statistical map to be denoised includes: Determine the time difference between the target time point and the preset time offset; Based on the time difference, the speed of light, and the width of the time point, the depth value corresponding to the pixel is calculated.
7. The method according to claim 1, wherein, The noise reduction model was pre-trained according to the following steps: Obtain the sample photon statistical map and the corresponding reference photon statistical map, wherein the reference photon statistical map is a photon statistical map generated by annotating the actual position of the object in the sample scene; The sample photon statistical map is input into the initial denoising model to obtain the sample denoised photon statistical map; Based on a preset loss function, the error between the denoised photon statistical image of the sample and the reference photon statistical image is determined; Based on the error, adjust the parameters of the initial noise reduction model; In response to the initial denoising model after parameter adjustment meeting the preset training termination condition, the current initial graph denoising model is determined as the trained denoising model.
8. The method according to claim 7, wherein, The determination of the error between the denoised photon statistical image of the sample and the reference photon statistical image based on a preset loss function includes: Based on at least one preset loss function, loss values are calculated for the denoised photon statistical map of the sample and the reference photon statistical map to obtain at least one loss value. The at least one loss function includes at least one of the following: main task loss function, distribution similarity loss function, sparsity loss function, and physical prior constraint loss function. The at least one loss value is weighted and summed to obtain the error between the denoised photon statistical map of the sample and the reference photon statistical map.
9. A depth detection device, comprising: The acquisition module is used to acquire a photon statistical map to be denoised generated for the target scene. For any pixel in the photon statistical map to be denoised, the pixel corresponds to a photon statistical data sequence. The photon statistical data in the photon statistical data sequence is used to characterize the number of reflected photons received from the target location at the corresponding time. The target location is the position in the target scene corresponding to the pixel. The encoding module is used to encode the photon statistical map to be denoised using the encoder included in the pre-trained denoising model to obtain encoded data; The calculation module is used to perform calculations on the encoded data using the bottleneck layer included in the noise reduction model to obtain the data to be decoded; The decoding module is used to decode the data to be decoded using the decoder included in the denoising model, and obtain the denoised photon statistics map; The generation module is used to generate a depth map corresponding to the target scene based on the peak data corresponding to each pixel in the denoised photon statistical map.
10. A depth detection system, comprising: A photon time-of-flight detection device and a controller, wherein the photon time-of-flight detection device and the controller are communicatively connected; The photon time-of-flight detection device is used to collect the number of reflected photons in the target scene and generate a statistical chart of photons to be denoised based on the number of reflected photons. The controller is used to perform the depth detection method according to any one of claims 1-8 based on the photon statistical map to be denoised.
11. A computer-readable storage medium storing a computer program, which, when executed, implements the depth detection method according to any one of claims 1-8.
12. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the depth detection method according to any one of claims 1-8.