Method and system for depth completion of sparse depth maps

Abstract

To provide a depth completion method of a sparse depth map, and a system thereof.SOLUTION: A depth completion method of sparse depth map includes: acquiring a grayscale image and a sparse depth map corresponding to the grayscale image; obtaining a nearest neighbor interpolation image and a Euclidean distance transform image based on the sparse depth map; inputting the grayscale image, the nearest neighbor interpolation image, and the Euclidean distance transform image into a neural network model, thereby outputting a predicted residual map; and generating a predicted dense depth map according to the predicted residual map and to the nearest neighbor interpolation image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a depth completion method, and more particularly to a depth completion method and system for a sparse depth map. [Background technology]

[0002] Time-of-flight (ToF) sensors are active depth-sensing devices that have the potential to provide more reliable scene understanding through true three-dimensional (3D) sensing. ToF sensors have recently been integrated into mobile consumer devices due to their low power consumption and real-time frame rate accuracy. However, ToF sensors rely on active illumination, which accounts for a significant portion of their power consumption. And while ToF sensors can provide accurate 3D information, their resolution can be much lower than color images when using projectors with a low number of dots to ensure eye safety and low power consumption. To use the limited power budget of mobile devices more efficiently, a dot-patterned light source can be used to illuminate the scene, so that its radiant intensity is concentrated in a few areas (points). Low-power ToF sensors used for indoor 3D sensing typically capture 500-1500 dots per frame. Due to this sparsity, sensor fusion techniques are required to obtain a dense depth map. Summary of the Invention

[0003] The objective of the present invention is to provide a depth completion method for a sparse depth map, including the steps of obtaining a grayscale image and a sparse depth map corresponding to the grayscale image; obtaining a nearest neighbor interpolation (NNI) image and a Euclidean distance transform (EDT) image based on the sparse depth map; inputting the grayscale image, the nearest neighbor interpolation image and the Euclidean distance transform image into a neural network model to output a prediction residual map; and generating a prediction dense depth map based on the prediction residual map and the nearest neighbor interpolation image.

[0004] In some embodiments, a pixel-level addition method is used to generate a predicted dense depth map based on the prediction residual map and a nearest neighbor interpolated image, where the prediction residual map includes residual information of the nearest neighbor interpolated image.

[0005] In some embodiments, a time-of-flight ranging sensor is used to acquire grayscale images and sparse depth maps.

[0006] In some embodiments, the method for depth completion of the sparse depth map further includes performing a downsampling operation on the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image before inputting the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image into a neural network model, and performing an upsampling operation on the predicted dense depth map, where the downsampling operation and the upsampling operation are performed by bilinear interpolation with anti-aliasing function.

[0007] In some embodiments, the neural network model employs an encoder-decoder scheme based on the UNet network architecture to capture multiple features of the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image.

[0008] In some embodiments, the method for depth completion of a sparse depth map further comprises performing a model pruning operation on the neural network model to compress the neural network model.

[0009] In some embodiments, the model pruning operation is performed only on a number of target layers of the neural network model, where each weight of the number of target layers is greater than a threshold.

[0010] In some embodiments, the method for depth completion of a sparse depth map further includes, after performing the model pruning operation, performing a model clustering operation on the neural network model to further compress the neural network model.

[0011] In some embodiments, the model clustering operation is performed only on a plurality of target layers of the neural network model, each of the weights of the plurality of target layers being greater than a threshold value.

[0012] In some embodiments, the method for sparse depth map depth completion further comprises quantizing the neural network model from a floating point model to an integer model.

[0013] Another object of the present invention is to provide a depth completion system for a sparse depth map, comprising: a time-of-flight ranging sensor for acquiring a grayscale image and a sparse depth map corresponding to the grayscale image; and a processor for receiving the grayscale image and the sparse depth map from the time-of-flight ranging sensor; acquiring a nearest neighbor interpolated image and a Euclidean distance transformed image based on the sparse depth map; inputting the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image into a neural network model to output a prediction residual map; and generating a predicted dense depth map based on the prediction residual map and the nearest neighbor interpolated image.

[0014] In some embodiments, the processor generates the predicted dense depth map based on the prediction residual map and the nearest neighbor interpolated image using a pixel-level addition method, where the prediction residual map includes residual information of the nearest neighbor interpolated image.

[0015] In some embodiments, the processor is further used for performing a downsampling operation on the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image, and performing an upsampling operation on the predicted dense depth map, before inputting the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image into the neural network model, where the downsampling operation and the upsampling operation are performed by bilinear interpolation with anti-aliasing function.

[0016] In some embodiments, the neural network model employs an encoder-decoder scheme based on the UNet network architecture to capture multiple features of the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image.

[0017] In some embodiments, the processor is further adapted to perform a model pruning operation on the neural network model to compact the neural network model.

[0018] In some embodiments, the model pruning operation is performed only on a number of target layers of the neural network model, where each weight of the number of target layers is greater than a threshold.

[0019] In some embodiments, the processor is further adapted to perform a model clustering operation on the neural network model after performing the model pruning operation to further compress the neural network model.

[0020] In some embodiments, the model clustering operation is performed only on a plurality of target layers of the neural network model, each of the weights of the plurality of target layers being greater than a threshold value.

[0021] In some embodiments, the processor is further adapted to quantize the neural network model from a floating point model to an integer model.

[0022] In order to make the above features and advantages of the present invention more comprehensible, the present invention will now be described with reference to the accompanying drawings, in which: FIG. [Brief description of the drawings]

[0023] Aspects of the present invention can be better understood in the following detailed description taken in conjunction with the accompanying drawings, in which: Note that, in accordance with standard industry practice, features have not been drawn to scale, and in fact may be arbitrarily increased or decreased in size to enhance clarity of discussion. [Figure 1] 1 is a flowchart of a method for depth completion of a sparse depth map according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram of a system for depth completion of a sparse depth map in accordance with an embodiment of the present invention. [Diagram 3] 2 is a process flow corresponding to the sparse depth map depth completion method of FIG. 1 according to an embodiment of the present invention. [Figure 4] FIG. 2 is a schematic diagram of an exemplary architecture of a neural network model according to an embodiment of the present invention. [Diagram 5] 1 is a flowchart of a method for depth completion of a sparse depth map according to an embodiment of the present invention. [Figure 6] 1 is an exemplary flow chart of optimizing a neural network model according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In the following, the embodiments of the present invention will be described in detail. However, it will be understood that the embodiments provide many applicable concepts, which can be implemented in a variety of specific contexts. Discussion: The disclosed embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0025] FIG 1 is a flow chart of a method for completing a sparse depth map according to an embodiment of the present invention. FIG 2 is a block diagram of a system for completing a sparse depth map according to an embodiment of the present invention. FIG 3 is a process flow corresponding to the method for completing a sparse depth map of FIG 1 according to an embodiment of the present invention.

[0026] As shown in FIG. 2, the sparse depth map depth completion system of the present invention includes a time-of-flight ranging sensor 120 and a processor 140 communicatively coupled to the time-of-flight ranging sensor 120 .

[0027] As shown in FIG. 1-FIG. 3, the depth completion method for a sparse depth map of the present invention includes steps S1-S4. In step S1, a time-of-flight ranging sensor 120 is used to acquire a grayscale image and a sparse depth map corresponding to the grayscale image. In some embodiments of the present invention, the sparse depth map contains 0.1%-10% (preferably about 1%) of valid pixels with useful information (i.e., sparsity at the resolution of the target depth map). Specifically, the present invention uses a single grayscale image frame and a sparse depth map.

[0028] Specifically, the present invention uses a single capture device (i.e., the time-of-flight ranging sensor 120) to capture grayscale images and sparse depth maps, thereby simplifying the depth completion system for sparse depth maps. Specifically, the present invention feeds grayscale images (not RGB / color images) to the downstream neural network model, which can be implemented on low-power devices with limited resources.

[0029] As shown in FIG. 1 to FIG. 3, in step S2, the processor 140 receives a grayscale image and a sparse depth map from the time-of-flight ranging sensor 120, and the processor 140 pre-processes the sparse depth map to obtain a nearest neighbor interpolation (NNI) image and a Euclidean distance transform (EDT) image. Specifically, the NNI image is obtained by performing a nearest neighbor interpolation (NNI) process, which is to map each coordinate to the nearest sparse depth pixel in Euclidean sense. Specifically, the EDT image is obtained by estimating the Euclidean distance transform (EDT), thereby providing a two-dimensional (2D) distance map to the nearest valid pixel.

[0030] As shown in FIG. 1-FIG. 3, in step S3, the processor 140 inputs the grayscale image, the NNI image, and the EDT image into a neural network model, and outputs a prediction residual map. The neural network model is constructed by an algorithm based on deep learning. The neural network model may be designed as an encoder-decoder convolutional neural network (CNN). Specifically, the neural network model adopts an encoder-decoder fashion based on the UNet network architecture to capture multiple features of the grayscale image, the NNI image, and the EDT image.

[0031] FIG. 4 is a schematic diagram of an exemplary architecture of a neural network model according to an embodiment of the present invention. As shown in FIG. 4, the designed encoder-decoder convolutional neural network is a simple "UNet like" template. For example, as shown in FIG. 4, the neural network model of the present invention adopts an encoder-decoder scheme consisting of a lite version of EfficientNetV2 (i.e., EfficientNetV2-Lite model) as a feature extractor, and adopts regular Conv2D layers with nearest neighbor up-sampling as a decoder. Specifically, the encoder part of the neural network model captures multiple features of grayscale images, NNI images, and EDT images. Specifically, the decoder part of the neural network model fuses features from the encoder part with an addition operation. All skip connections are made by concatenation.

[0032] Regarding the implementation of the encoder, as shown in FIG. 4, the encoder architecture discards the last few layers (of the EfficientNetV2-Lite model) with small feature map resolution (X / 32), so that the EfficientNetV2-Lite model of the present invention has fewer blocks to achieve fewer parameters and faster inference speed. Specifically, the EfficientNetV2-Lite model of the present invention only includes Add, Pad, Cov2D, DepthwiseConv2D, and ReLU6 operations, which achieve better quantization. The EfficientNetV2-Lite model is mainly constructed with Conv2D and DepthwiseConv2D, and uses ReLU6 with low quantization loss. ReLU6 limits the range of the activation value to [0, 6]. Due to poor quantization performance, the attention mechanism (i.e., Squeeze-and-Excitation (SE) block) is not used.

[0033] In other words, for the decoder implementation, as shown in Figure 4, all Conv2D are kernel_size=3, followed by BatchNormalization and ReLU. One Conv2D block contains two Conv2Ds followed by Upsample2D with Nearest Neighbor Interpolation (NNI). There are four blocks that can scale the resolution from X / 16 to X. The final output block has three Conv2Ds without Upsample2D, and the last Conv2D is kernel_size=1 and has a bias parameter. The decoder is implemented with only Conv2D and ReLU, which makes it easier to quantize. Note that the neural network model makes the decoder predict the residual map, not the entire dense depth map. The residual information has a small value range, so it can be better quantized.

[0034] As shown in FIG. 1 to FIG. 3, in step S4, the processor 140 generates a prediction dense depth map based on the prediction residual map and the NNI image. Specifically, the processor 140 generates a prediction dense depth map based on the prediction residual map and the NNI image using a pixel-level addition method. The prediction residual map includes residual information of the NNI image.

[0035] 5 is a flow chart of a method for completing a sparse depth map according to an embodiment of the present invention. The method for completing a sparse depth map shown in FIG 5 is similar to the method for completing a sparse depth map shown in FIG 1, and the difference between the two is that the method for completing a sparse depth map shown in FIG 5 further includes steps P1 and P2.

[0036] As shown in Fig. 5, in step P1, before inputting the grayscale image, the NNI image and the EDT image into the neural network model, the processor 140 further performs a down-sampling process on the grayscale image, the NNI image and the EDT image. In step P2, the processor 140 further performs an up-sampling process on the predicted dense depth map. In some embodiments of the present invention, the down-sampling process and the up-sampling process are performed by bilinear interpolation with anti-aliasing function. For example, the resolution of the down-sampling process can be reduced from 224x304 to 128x160 to reduce the usage of static random access memory (SRAM). For example, adjust the resolution of the sparse depth map from 480x640 to 240x320, center crop to 224x304, adjust the resolution of the grayscale image, the NNI image and the EDT image to 128x160, and input the sized grayscale image, the NNI image and the EDT image into the neural network model. Note that the above downsampling process is optional for the present invention. Corresponding to the downsampling process, the resolution of the predicted dense depth map is also adjusted to 224x304.

[0037] FIG. 6 is an exemplary flowchart of optimizing a neural network model according to an embodiment of the present invention. Specifically, the present invention optimizes a neural network model using the official TensorFlow Model Optimization toolkit (TFMOT). As shown in FIG. 6, in step Q1, the processor 140 performs a model pruning operation on the neural network model to compress the neural network model. Specifically, in step Q1, the neural network model is pruned to examine the trade-off between the compressed size and the model performance. It should be noted that the model pruning operation is optional for the present invention.

[0038] Regarding the model pruning operation, the model pruning operation is performed only on multiple target layers of the neural network model, which is also called selective pruning. Before pruning, the weight (n) of each layer of the neural network model is calculated. Then, a threshold is set to find multiple target layers with large weights and potential redundancies. In other words, the weight of each target layer is greater than the threshold. For example, if the threshold is 10000, a specific layer with a weight (n) greater than 10000 is defined as the target layer, and then the target layer is pruned with a sparsity of 50% or 75%, and other layers that are not the target layer are not changed. For example, a target layer pruned with a sparsity of 50% ensures that 50% of the weight of the layer is zero. For example, if the threshold is 5000, a specific layer with a weight (n) greater than 5000 is defined as the target layer, and then the target layer is pruned with a sparsity of 90%, and other layers that are not the target layer are not changed. Specifically, selective pruning performs a model pruning operation on the target layer, leaving other layers unchanged.

[0039] As shown in Fig. 6, in step Q2, after performing the model pruning operation, the processor 140 performs a model clustering operation on the neural network model to further compress the neural network model. Specifically, in step Q2, the neural network model is further compressed using sparsity-preserved clustering. It should be noted that the model clustering operation is optional for the present invention.

[0040] For the model clustering operation, the model clustering operation is performed only on multiple target layers of the neural network model, which is also called selective clustering. Before clustering, the weight (n) of each layer of the neural network model is calculated. Then, a threshold is set to find multiple target layers with large weights and potential redundancy. In other words, the weight of each target layer is greater than the threshold. For example, if the threshold is 10000, a specific layer with a weight (n) greater than 10000 is defined as the target layer, and then the target layer is clustered with a sparsity of 50% or 75%, and other layers that are not the target layer are not changed. For example, if the threshold is 5000, a specific layer with a weight (n) greater than 5000 is defined as the target layer, and then the target layer is clustered with a sparsity of 90%, and other layers that are not the target layer are not changed. Specifically, selective clustering performs the model clustering operation on the target layer, and other layers are not changed.

[0041] As shown in Fig. 6, in step Q3, the processor 140 performs post-training quantization to compress the neural network model by four times by quantizing the neural network model from a floating-point model (i.e., the data format is FLOAT32) to an integer model (i.e., the data format is fully integer (INT8)), so that the compressed neural network model can be realized in a device with limited resources and low power consumption. It should be noted that the post-training quantization is optional for the present invention.

[0042] Specifically, the floating-point model is further optimized by selective pruning (step Q1), selective clustering (step Q2), and quantization to an INT8 model (Q3) after training, as shown in Fig. 6. The optimized INT8 model deployment is for achieving extremely high energy efficiency, so that the optimized INT8 model can be implemented on low-power devices with limited resources.

[0043] In conclusion, the present invention uses a multi-optimized, efficient and compact neural network to predict a dense depth map from a sparse depth map and a grayscale image. The efficient and compact neural network with low model size and memory usage can be implemented in low-power devices with limited resources. A high-precision and high-resolution depth image can be generated by fusing the sparse depth image and the corresponding grayscale image. The fusion result can improve the performance of subsequent tasks (e.g., 3D object detection, semantic segmentation, etc.).

[0044] Having outlined the features of several embodiments above, those skilled in the art can better understand the aspects of the present invention. It should be understood that those skilled in the art can easily design or modify other manufacturing processes and structures based on the present invention, thereby achieving the same objectives and / or advantages as those embodiments introduced herein. Those skilled in the art will also understand that these equivalent constructions do not depart from the spirit and scope of the present invention, and that various changes, substitutions and alterations can be made without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0045] 120 Time-of-Flight Ranging Sensor 140 processors P1, P2, Q1, Q2, Q3, S1, S2, S3, S4 steps

Claims

1. obtaining a grayscale image and a sparse depth map corresponding to the grayscale image; obtaining a nearest neighbor interpolation (NNI) image and a Euclidean distance transform (EDT) image based on the sparse depth map; inputting the grayscale image, the nearest neighbor interpolated image, and the Euclidean distance transformed image into a neural network model and outputting a prediction residual map; generating a predicted dense depth map based on the prediction residual map and the nearest neighbor interpolated image; A depth completion method for sparse depth maps, including

2. The method for depth interpolation of sparse depth maps of claim 1 , further comprising: generating the predicted dense depth map based on the prediction residual map and the nearest neighbor interpolated image using a pixel-level addition method, wherein the prediction residual map includes residual information of the nearest neighbor interpolated image.

3. The method of claim 1 , wherein the grayscale image and the sparse depth map are acquired using a time-of-flight ranging sensor.

4. performing a downsampling process on the grayscale image, the nearest neighbor interpolated image, and the Euclidean distance transformed image before inputting the grayscale image, the nearest neighbor interpolated image, and the Euclidean distance transformed image into the neural network model; performing an upsampling operation on the predicted dense depth map; Further comprising: The method of claim 1 , wherein the downsampling and upsampling are performed by bilinear interpolation with anti-aliasing.

5. The method for depth completion of sparse depth maps as claimed in claim 1 , wherein the neural network model adopts an encoder-decoder scheme based on a UNet network architecture to capture multiple features of the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image.

6. The method of claim 1 , further comprising: performing a model pruning operation on the neural network model to compress the neural network model.

7. 7. The method of claim 6, further comprising: performing the model pruning operation only on a number of target layers of the neural network model, each of the target layers having a weight greater than a threshold, the threshold being set to find a number of target layers with large weights and potential redundancies.

8. The method of claim 6 , further comprising the step of performing a model clustering operation on the neural network model after performing the model pruning operation to further compress the neural network model.

9. 9. The method of claim 8, further comprising: performing the model clustering operation only on multiple target layers of the neural network model, each of the target layers having a weight greater than a threshold, the threshold being set to find multiple target layers with large weights and potential redundancy.

10. The method of claim 1 , further comprising quantizing the neural network model from a floating point model to an integer model.

11. a time-of-flight ranging sensor for acquiring a grayscale image and a sparse depth map corresponding to the grayscale image; a processor for receiving the grayscale image and the sparse depth map from the time-of-flight ranging sensor; obtaining a nearest neighbor interpolated image and a Euclidean distance transformed image based on the sparse depth map; inputting the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image into a neural network model to output a prediction residual map; and generating a predicted dense depth map based on the prediction residual map and the nearest neighbor interpolated image; A depth completion system for sparse depth maps comprising:

12. 12. The system of claim 11, wherein the processor generates the predicted dense depth map based on the prediction residual map and the nearest neighbor interpolated image using a pixel-level addition method, the prediction residual map including residual information of the nearest neighbor interpolated image.

13. The processor, performing a downsampling process on the grayscale image, the nearest neighbor interpolated image, and the Euclidean distance transformed image before inputting the grayscale image, the nearest neighbor interpolated image, and the Euclidean distance transformed image into the neural network model; performing an upsampling operation on the predicted dense depth map; It is further used in The sparse depth map depth completion system of claim 11 , wherein the downsampling and upsampling are performed by bilinear interpolation with anti-aliasing.

14. 12. The system of claim 11, wherein the neural network model employs an encoder-decoder scheme based on a UNet network architecture to capture multiple features of the grayscale image, the nearest neighbor interpolated image and the Euclidean distance transformed image.

15. The processor, The system for depth completion of sparse depth maps of claim 11 , further used for performing a model pruning operation on the neural network model to compress the neural network model.

16. 16. The system of claim 15, further comprising: performing the model pruning operation only on a number of target layers of the neural network model, each of the target layers having a weight greater than a threshold, the threshold being set to find a number of target layers with large weights and potential redundancies.

17. The processor, The system for depth completion of sparse depth maps of claim 15 , further comprising: after performing the model pruning operation, performing a model clustering operation on the neural network model to further compress the neural network model.

18. 20. The system of claim 17, further comprising: performing the model clustering operation only on a number of target layers of the neural network model, each of the target layers having a weight greater than a threshold, the threshold being set to find a number of target layers with large weights and potential redundancy.

19. The processor, The system of claim 11 , further comprising: quantizing the neural network model from a floating point model to an integer model.

Images