A Monocular Depth Estimation Method with Point Cloud Consistency Constraints for 3D Object Detection

Abstract

This invention belongs to the field of computer vision image processing and discloses a monocular depth estimation method with point cloud consistency constraints for 3D object detection. The method includes: preparing a dataset for unsupervised monocular depth estimation; inputting the training set into an unsupervised monocular depth estimation model in batches, with the model outputting a depth map and pose changes corresponding to the input image; calculating the reprojected image and point cloud corresponding to the input image; further calculating the total training loss, including reprojection loss, point cloud consistency loss, and edge smoothing loss; training the model using the total loss constraint to obtain the optimal unsupervised monocular depth estimation model; preparing a dataset for 3D object detection and generating pseudo-point cloud files; training a 3D object detection model to obtain the optimal 3D object detection model. This invention delves into pure image methods and improves the accuracy of depth estimation and 3D object detection in road scenes by introducing 3D information constraints.

Description

Technical Field

[0001] This invention belongs to the field of computer vision image processing, specifically relating to a monocular depth estimation method with point cloud consistency constraints for 3D object detection. Background Technology

[0002] Depth estimation is a classic problem in computer vision, playing a crucial role in 3D reconstruction, occlusion handling, and illumination estimation. The essence of obtaining depth information from an image is constructing a model that associates image information with depth information. Acquiring depth information helps in better understanding 3D scenes and is beneficial for many computer vision tasks, such as indoor localization, height estimation, Simultaneous Localization and Mapping (SLAM), visual odometry, and classification. Compared to obtaining the depth of a certain number of points on an object's surface using various hardware devices such as laser rangefinders, image-based depth estimation methods have a wider range of applications because they do not require expensive equipment or specialized personnel.

[0003] Currently, deep learning has become the mainstream approach for solving depth estimation problems. Specific methods can be categorized by learning type: supervised, semi-supervised, self-supervised, and unsupervised; or broadly by input data type: single image (monocular single frame), multiple images (stereo single frame, monocular multi-frame), and methods combining semantic and other information. From the perspectives of cost reduction and research potential, unsupervised monocular depth estimation methods using only a single shot are currently a hot research topic.

[0004] Specifically, in the field of intelligent driving, specifically in road scenarios, achieving reliable and stable target detection necessitates fast and accurate depth estimation. In road scenarios, the primary targets of human attention are motor vehicles, non-motor vehicles, and pedestrians; background elements such as the sky and distant vegetation are not the main focus. However, depth estimation evaluation often assesses the entire image, and the proportion of actually useful information within the scene is relatively small. Unsupervised monocular depth estimation practices have also revealed unsatisfactory results for estimating specific useful targets, posing challenges to downstream target detection tasks. Furthermore, current technological explorations often focus solely on pure image processing, neglecting to consider the impact of depth estimation accuracy on 3D target detection performance from a 3D target detection perspective. Unsupervised monocular depth estimation schemes are not specifically designed for optimization. Summary of the Invention

[0005] This invention provides a point cloud consistency-consistent monocular depth estimation method for 3D target detection, addressing the problem of insufficient estimation accuracy of unsupervised monocular depth estimation for specific useful targets, thus affecting the accuracy of downstream 3D target detection. It also proposes introducing 3D information constraints on top of pure image methods to improve the accuracy of both unsupervised monocular depth estimation and 3D target detection.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] This invention is a monocular depth estimation method with point cloud consistency constraints for 3D object detection, comprising the following steps:

[0008] Step S1: Prepare a dataset D1 for unsupervised monocular depth estimation. The dataset includes a training set D1-T and a validation set D1-E. The training set includes multiple sets of temporally continuous monocular images taken by a camera, and the validation set includes multiple discontinuous monocular images taken by a camera.

[0009] Step S2: Input the training set D1-T into the unsupervised monocular depth estimation model in batches. Each batch of training data includes multiple sets of continuous monocular images. The model outputs data for calculating point clouds and loss. The output data includes depth maps corresponding to the input images and pose changes of the camera corresponding to the input images.

[0010] Step S3: Based on the depth map and pose change obtained in step S2, calculate the reprojection image and point cloud corresponding to the input image;

[0011] Step S4: Calculate the total training loss based on the depth map obtained in step S2 and the reprojected image and point cloud obtained in step S3, wherein the total training loss includes reprojection loss, point cloud consistency loss, and edge smoothing loss.

[0012] Step S5: Use the total training loss obtained in step S4 to train the unsupervised monocular depth estimation model described in step S2, optimize it to obtain the optimal unsupervised monocular depth estimation model, verify it using D1-E, and calculate the depth estimation index.

[0013] Step S6: Prepare dataset D2 for 3D object detection. The dataset includes training set D2-T and validation set D2-E. Both the training set and the validation set include multiple discontinuous monocular images taken by a camera and the corresponding point cloud files of the images. The point cloud files are obtained by processing the depth map of the corresponding image using a depth map-point cloud conversion algorithm. The depth map is obtained by processing the corresponding image using the optimal unsupervised monocular depth estimation model obtained in step S5.

[0014] Step S7: Train the 3D object detection model using the training set D2-T to obtain the optimal 3D object detection model, validate it using the validation set D2-E, and calculate the 3D object detection index.

[0015] Further, the calculation of the total training loss in step S4 is expressed as:

[0016] L=αL rp +βL pcc +γL es

[0017] Where L is the total training loss, L rp It is the reprojection loss, L pcc It is the point-cloud consistency loss, L es It is the edge-aware smoothness loss, where α, β, and γ are hyperparameters.

[0018] Furthermore, the reprojection loss is expressed as:

[0019]

[0020] SSIM is a structural similarity metric function used to measure the structural similarity between two images. t It is the target frame image selected from the continuous monocular images in step S2, I s→t This is the reprojected image reconstructed from the source frame image to the target frame in step S3. The source frame is any frame in the continuous monocular image other than the target frame. μ is a hyperparameter, and I... s→t Represented as:

[0021] I s→t =warp(I s , p′ s )

[0022] Where warp is a transformation operation that warps the image based on the sampling points, I s It is the source frame image, p′ s These are the sampling points on the source frame, and the transform operation warp(I) s , p′ s ) refers to image I s Based on sampling point p′ s To distort and deform, the sampling point is the coordinate obtained by reprojecting the target frame coordinates onto the source frame, that is, the point on the source frame corresponding to the target frame, p′. s Represented as:

[0023] p′s ~KT t→s D(p t )K -1 p t

[0024] Among them, the camera internal parameters of the K photography device, T t→s This refers to the pose change output by the model in step S2. The pose refers to the pose change from the target frame to the source frame, and D represents the depth calculated by the depth estimation model. D(p) t ) is p t depth, p t These are the coordinates on the target frame, p′ s p is calculated using the above formula. t The coordinates of the corresponding point on the source frame;

[0025] Additionally, SSIM is represented as:

[0026]

[0027] Where I X I Y These are image X and image Y, respectively, μ X and μ Y Let σ represent the mean of X and Y, respectively. X and σ Y Let σ represent the variances of X and Y, respectively. XY Let C1 and C2 represent the covariance of X and Y, respectively, where C1 and C2 are constants.

[0028] Furthermore, the point cloud consistency loss is expressed as:

[0029]

[0030] SSIM pc This is a point cloud structure similarity measurement function, used here to measure the structural similarity between two point clouds, where P represents the point cloud (a set of three-dimensional coordinates), and P′... s The point cloud P is obtained by performing a warp operation as described in the previous reprojection loss on the point cloud of the source frame calculated in step S3. t→s This is the point cloud obtained by performing pose transformation on the point cloud of the target frame calculated in step S3. `median` represents taking the median, `div` represents dividing point by point, `υ` is a hyperparameter, and `P′`... s and P t→s They are represented as follows:

[0031] P′ s =warp(P s , p′ s ) = warp(D(p s )K -1 p s, p′ s )

[0032] P t→s =T t→s D(p t )K -1 p t

[0033] Transformation operation warp(P) s , p′ s ) refers to point cloud P s Based on sampling point p′ s To twist and deform;

[0034] Additionally, SSIM pc Represented as:

[0035]

[0036] Where P X P Y They are point cloud X and point cloud Y, respectively, μ X and μ Y Let σ represent the mean of X and Y, respectively. X and σ Y Let X and Y represent the variances, respectively, and C1 and C2 are constants.

[0037] Furthermore, the edge smoothing loss is expressed as:

[0038]

[0039] in I represents the mean normalization of depth. t Represents the target frame image. and These represent partial derivatives in the horizontal and vertical directions of the image, respectively.

[0040] The beneficial effects of the present invention are as follows: The point cloud consistency-consistent monocular depth estimation method for 3D target detection provided by the present invention is more beneficial to 3D target detection.

[0041] This invention delves into pure image depth estimation methods and, by introducing 3D information constraints, elevates the constraints from planar to three-dimensional space, thereby improving both the depth estimation performance in road scenes and the accuracy of 3D target detection. Attached Figure Description

[0042] Figure 1 This is a flowchart of an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the depth estimation network structure according to an embodiment of the present invention.

[0044] Figure 3 A detailed diagram of the depth estimation network structure in this embodiment of the invention.

[0045] Figure 4 These are depth estimation and 3D object detection results for a certain scene implemented in an embodiment of the present invention.

[0046] Figure 5 This presents the comparative experimental results of the present invention in depth estimation.

[0047] Figure 6 This presents the comparative experimental results of the present invention in 3D target detection. Detailed Implementation

[0048] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0049] like Figure 1 As shown, this invention is a monocular depth estimation method for point cloud consistency constraints for 3D target detection, characterized in that: the point cloud consistency constraint monocular depth estimation method includes the following steps:

[0050] Step 1: Prepare the dataset D1 for unsupervised monocular depth estimation. The dataset includes the training set D1-T and the validation set D1-E.

[0051] Step 2: Input the training set D1-T mentioned in Step 1 into the unsupervised monocular depth estimation model in batches. Each batch of training data includes multiple sets of continuous monocular images. The unsupervised monocular depth estimation model outputs data for calculating point clouds and loss. The output data includes depth maps corresponding to the input images and pose changes of the camera corresponding to the input images.

[0052] Step 3: Based on the depth map and pose changes obtained in Step 2, calculate the reprojected image and point cloud corresponding to the input image;

[0053] Step 4: Based on the depth map obtained in Step 2 and the reprojected image and point cloud obtained in Step 3, calculate the total training loss, which is expressed as:

[0054] L=αLr p +βL pcc +γL es

[0055] Where L is the total training loss, Lr p It is the reprojection loss, Lpcc It is the point-cloud consistency loss, L es It is the edge-aware smoothness loss, where α, β, and γ are hyperparameters.

[0056] The total training loss includes reprojection loss, point cloud consistency loss, and edge smoothing loss.

[0057] The point cloud consistency loss is expressed as:

[0058]

[0059] Among them, SSIM pc This is a point cloud structure similarity measurement function, used to measure the structural similarity between two point clouds. P represents the point cloud, i.e., the set of three-dimensional coordinates, and P′ is the point cloud. s P is the point cloud obtained by performing a warp operation on the point cloud of the source frame calculated in step 3. t→s This is the point cloud obtained by performing pose transformation on the point cloud of the target frame calculated in step 3. `median` represents taking the median, `div` represents dividing point by point, `υ` is a hyperparameter, and `P′`... s and P t→s They are represented as follows:

[0060] P′ s =warp(P s , p′ s ) = warp(D(p s )K -1 p s , p′ s )

[0061] P t→s =T t→s D(p t )K -1 p t

[0062] Where warp is the transformation operation that warps the image based on the sampling points, D represents the depth calculated by the depth estimation model, K is the camera intrinsic parameter of the photography device, and T is the depth. t→s This refers to the pose change output by the model in step 2, the transformation operation warp(P) s , p′ s ) refers to point cloud P s Based on sampling point p′ s To distort and deform, the sampling point p′ s The coordinates are obtained by reprojecting the target frame coordinates onto the source frame, i.e., the point on the source frame corresponding to the target frame, p′. s Represented as:

[0063] p′ s ~KT t→s D(p t )K -1 p t

[0064] The meanings of D, K, and T are as described above.

[0065] Wherein, the SSIM pc Represented as:

[0066]

[0067] Where P X P Y They are point cloud X and point cloud Y, respectively, μ X and μ Y Let σ represent the mean of X and Y, respectively. X and σ Y Let X and Y represent the variances, respectively, and C1 and C2 are constants.

[0068] The reprojection loss is expressed as:

[0069]

[0070] SSIM is a structural similarity metric function used to measure the structural similarity between two images. t It is the target frame image selected from the continuous monocular images in step 2, I s→t This is the reprojected image reconstructed from the source frame image to the target frame in step 3. The source frame is any frame in the continuous monocular image other than the target frame. μ is a hyperparameter, and I... s→t Represented as:

[0071] I s→t =warp(I s , p′ s )

[0072] Where warp is a transformation operation that warps the image based on the sampling points, I s It is the source frame image, p′ s These are the sampling points on the source frame, and the transform operation warp(I) s , p′ s ) refers to image I s Based on sampling point p′ s To distort and deform, the sampling point is the coordinate obtained by reprojecting the target frame coordinates onto the source frame, that is, the point on the source frame corresponding to the target frame, p′. s Represented as:

[0073] p′s ~KT t→s D(p t )K -1 p t

[0074] Among them, the camera internal parameters of the K photography device, T t→s This refers to the pose change output by the model in step 2. The pose refers to the change in pose from the target frame to the source frame, and D represents the depth calculated by the depth estimation model. D(p) t ) is p t depth, p t These are the coordinates on the target frame, p′ s p is calculated using the above formula. t The coordinates of the corresponding point on the source frame;

[0075] SSIM is represented as:

[0076]

[0077] Where I X I Y These are image X and image Y, respectively, μ X and μ Y Let σ represent the mean of X and Y, respectively. X and σ Y Let σ represent the variances of X and Y, respectively. XY Let C1 and C2 represent the covariance of X and Y, respectively, where C1 and C2 are constants.

[0078] The edge smoothing loss is expressed as:

[0079]

[0080] in I represents the mean normalization of depth. t Represents the target frame image. and These represent partial derivatives in the horizontal and vertical directions of the image, respectively.

[0081] Step 5: Use the total training loss obtained in Step 4 to train the unsupervised monocular depth estimation model from Step 2, optimize it to obtain the optimal unsupervised monocular depth estimation model, validate it using D1-E, and calculate the depth estimation index.

[0082] Step 6: Prepare dataset D2 for 3D object detection. The dataset includes training set D2-T and validation set D2-E. Both training set D2-T and validation set D2-E include multiple discontinuous monocular images taken by a camera and the corresponding point cloud files of the images. The point cloud files are obtained by processing the depth map of the corresponding image using a depth map-point cloud conversion algorithm. The depth map is obtained by processing the corresponding image using the optimal unsupervised monocular depth estimation model obtained in step 5.

[0083] Step 7: Train the 3D object detection model using the training set D2-T to obtain the optimal 3D object detection model, validate it using the validation set D2-E, and calculate the 3D object detection index.

[0084] Example

[0085] This invention is a monocular depth estimation method with point cloud consistency constraints for 3D object detection, comprising the following steps:

[0086] (1) Prepare a dataset for unsupervised monocular depth estimation

[0087] Prepare a dataset D1 for unsupervised monocular depth estimation. The dataset includes a training set D1-T and a validation set D1-E. The training set consists of multiple sets of temporally continuous monocular images taken by a camera, and the validation set consists of multiple discontinuous monocular images taken by a camera. D1 uses the original KITTI dataset, containing: ① Unsynchronized and uncorrected raw and processed (synchronized and corrected) grayscale image sequences: acquired at 10Hz, 1242*375, PNG format; ② Raw (unsynchronized and uncorrected) and processed (synchronized and corrected) color image sequences: acquired at 10Hz, 1242*375, PNG format; ③ 3D Velodyne point cloud: 100k points per frame, stored as a binary floating-point matrix, bin format; ④ 3D GPS / IMU data: position, velocity, acceleration, metadata, stored in text; ⑤ Calibration data: camera, camera-to-GPS / IMU, camera-to-Velodyne calibration data, stored in text; ⑥ 3D target trajectory labels: cars, trucks, trams, pedestrians, cyclists, XML format. The dataset is divided into training and validation sets according to Eigen's data partitioning scheme. D1-T contains 39810 monocular images, and D1-E contains 4424 monocular images.

[0088] (2) Training an unsupervised monocular depth estimation model

[0089] The unsupervised monocular depth estimation model consists of a depth prediction network and a pose estimation network. The depth prediction network is an encoder-decoder architecture, including a ResNet18 encoder and a depth decoder. A schematic diagram of the network structure is shown below. Figure 2 As shown, the detailed diagram of the network structure is as follows: Figure 3 As shown. Its input is a monocular image I. t The output is the depth map D of the monocular image. t The pose estimation network also follows an encoder-decoder architecture, consisting of a ResNet18 encoder (shared with the depth prediction network) and a pose decoder. Its input is two frames of monocular images. t I s The output is the pose change T from the target frame to the source frame. t→s .

[0090] The training set D1-T is input into the unsupervised monocular depth estimation model in batches. Each batch of training data includes multiple sets of continuous monocular images. The model outputs data used to calculate point clouds and losses. The output data includes depth maps corresponding to the input images and pose changes of the camera corresponding to the input images.

[0091] Specifically, taking two monocular images I from the training process... t I s Taking an unsupervised monocular depth estimation model as an example, I is obtained through a depth prediction network. t I s Depth map D t D s The pose change T from the target frame to the source frame is obtained through a pose estimation network. s→s .according to

[0092] p s ′ ~KT t→s D(p t )K -1 p t

[0093] and

[0094] I s→t =warp(I s ,p s ′ )

[0095] In this embodiment, the warp is implemented using the grid_sample method provided by the PyTorch deep learning framework.

[0096] Obtain the "pseudo-target frame" image I reconstructed from the source frame to the target frame. s→t This establishes a self-supervised constraint—reprojection loss—between the original target frame image and the "pseudo-target frame" image.

[0097]

[0098] In this example, μ is set to 0.85.

[0099] for

[0100]

[0101] In this embodiment, C1 = 0.0001 and C2 = 0.0009 are set.

[0102] Similarly, according to

[0103] P t→s =T t→s D(p t )K -1 p t

[0104] and

[0105] P′ s =warp(P s , p′ s ) = warp(D(p s )K -1 p s , p′ s )

[0106] In this embodiment, the warp is implemented using the grid_sample method provided by the PyTorch deep learning framework.

[0107] The "pseudo-source frame point cloud" P obtained by transforming the target frame point cloud into a pose cloud t→s And the source frame point cloud after sampling point distortion (the purpose is to correspond the points to the "pseudo-source frame point cloud"). Similarly, a self-supervised constraint is established between the warped source frame point cloud and the "pseudo-source frame point cloud"—point cloud consistency loss:

[0108]

[0109] `div` normalizes the point cloud data at the micro-level of each point, aiming to use relative distances instead of absolute distances. `median` normalizes the point cloud consistency loss at the overall point cloud level, addressing the issue that introducing consistency loss would result in an estimated depth value that is too small, leading to a scale-dependent problem. Both methods aim to solve the limitation of unsupervised monocular depth estimation models that can only output relative distances instead of absolute distances. Not all implementations are fully implemented in other embodiments. Figure 4 Comparative experimental results for different embodiments are presented.

[0110] In this embodiment, υ = 0.85 is set.

[0111] for

[0112]

[0113] In this embodiment, C1 = 0.0001 and C2 = 0.0009 are set.

[0114] Similarly, add edge smoothing loss:

[0115]

[0116] This loss function can effectively handle the edges of objects in an image, making objects clearly distinguishable in terms of depth.

[0117] Combining the above constraints, we obtain the total loss function:

[0118] L=αL rp +βL pcc +γL es

[0119] In this embodiment, α = 1, β = 0.01, and γ = 0.001 are set.

[0120] The unsupervised monocular depth estimation model described in step S2 is trained using the total training loss obtained in step S4, and the optimal unsupervised monocular depth estimation model is obtained through optimization.

[0121] The above process uses an NVIDIA GTX 1080TI to train the model, uses the PyTorch deep learning framework to build an unsupervised monocular depth estimation model, sets the batch size to 8, trains for 20 epochs, iterates a total of 99,525 times, uses the Adam optimizer, sets the initial learning rate to 1e-4, and reduces the learning rate to 0.1 times the original rate at the 12th and 16th epochs respectively.

[0122] (3) Validate the unsupervised monocular depth estimation model

[0123] The optimal unsupervised monocular depth estimation model in (2) is validated using the validation set D1-E. Currently, the most commonly used quantitative indicators for evaluating monocular depth estimation are absolute relative error (Abs Rel), relative error (Sq Rel), root mean square error (RMSE), logarithmic root mean square error (RMSE log), and threshold accuracy. The following are the calculation formulas for each quantitative indicator:

[0124]

[0125]

[0126]

[0127]

[0128]

[0129] Where d i It is the actual depth value. This is the predicted depth value. The threshold thr is commonly set to 1.25, 1.252, or 1.253. Lower values ​​for Absolute Relative Error (Abs Rel), Relative Error (Sq Rel), Root Mean Square Error (RMSE), and Log Root Mean Square Error (RMSElog) indicate better depth estimation performance, and higher threshold accuracy also indicates better depth estimation performance.

[0130] The experimental results of the examples are as follows Figure 5 As shown, compared with the MonoDepth2 algorithm (Cl′ement Godard, Oisin MacAodha, Michael Firman, Gabriel Brostow. Digging Into Self-Supervised MonocularDepth Estimation. In ICCV, 2019), it is demonstrated that the present invention improves the monocular depth estimation performance.

[0131] (4) Prepare a dataset for 3D object detection

[0132] A dataset D2 for 3D object detection is prepared, comprising a training set D2-T and a validation set D2-E. Both the training and validation sets consist of multiple discontinuous monocular images captured by a photographic device, along with their corresponding point cloud files. The point cloud files are obtained by processing the depth maps of the corresponding images using a depth map-to-point cloud conversion algorithm. The depth maps are obtained by processing the corresponding images using an optimal unsupervised monocular depth estimation model. D2 uses the KITTI object detection dataset, which includes ① color images in PNG format; ② Velodyne point clouds in BIN format; and ③ object labels stored as text. D2-T contains 3712 training images and their corresponding point cloud files and object labels, while D2-E contains 3769 training images and their corresponding point cloud files and object labels.

[0133] Using the unsupervised monocular depth estimation model trained in (2), the depth maps of all color images in D2 are output. Then, the depth maps are converted into point clouds using the depth map-point cloud algorithm and saved as a bin file, called Pseudo-LiDAR. The depth map-point cloud algorithm is expressed as:

[0134] (depth)z = D(u,v)

[0135]

[0136]

[0137] Where (x,y,z) are 3D coordinates, (u,v) are the pixel coordinates on the graph, and (c U ,c V () represents the camera center coordinates, f U and f V These represent the horizontal and vertical focal lengths of the camera lens, respectively. Because the depth output by the unsupervised monocular depth estimation model is not an absolute depth, it is necessary to match the numerical scale of the true depth when generating the point cloud.

[0138] Replace the ②Velodyne point cloud in D2 with the generated pseudo radar point cloud.

[0139] (5) Training the 3D object detection model

[0140] The 3D object detection model adopts the AVOD algorithm (Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L. Waslander. Joint 3D Proposal Generation and Object Detection from View Aggregation. In IROS, 2018). The AVOD is trained using the D2-T dataset prepared in (4), and the optimal AVOD 3D object detection model is obtained by optimization.

[0141] The model was trained using an NVIDIA GTX 1080TI, and the AVOD model was built using the TensorFlow deep learning framework. The training iterations were 120,000, and the Adam optimizer was used. The initial learning rate was set to 1e-4, and the learning rate was reduced to 0.8 times the original rate every 30,000 iterations.

[0142] (6) Validate the 3D target detection model

[0143] The optimal AVOD model obtained in (5) is validated using the validation set D2-E, and 3D object detection metrics are calculated. The two 3D object detection metrics are AP 3D and AP BEV. The former is used to evaluate the accuracy of the 3D detection box, and the latter is used to evaluate the accuracy of the detection box under the BEV view.

[0144] The experimental results of the examples are as follows Figure 6As shown, the performance of this invention in AP 3D and AP BEV at IoU = 0.5 demonstrates the potential of unsupervised algorithms compared to Mono3D (supervised algorithm) (X. Chen, K. Kundu, Z. Zhang, H. Ma, S. Fidler, and R. Urtasun. Monocular 3d object detection for autonomous driving. In CVPR, 2016.). Compared to MonoDepth2+Pseudo-LiDAR, this invention improves the accuracy of 3D object detection, i.e., it optimizes the depth estimation and 3D object detection of specific useful targets in road scenes.

[0145] The depth estimation and 3D detection results of the embodiment are shown in the figure below. Figure 4 As shown, the first image from top to bottom is the original image of the scene, the second image is the depth estimation effect of this invention, the brighter the color, the closer the distance, the third and fourth images are the 2D detection box effect and the 3D detection box effect respectively, the red box line represents the true label of the target detection box, and the green box line represents the detection box drawn by the target detection model.

[0146] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A monocular depth estimation method with point cloud consistency constraints for 3D object detection, characterized in that: The point cloud consistency-consistent monocular depth estimation method includes the following steps: Step 1: Prepare a dataset D1 for unsupervised monocular depth estimation. The dataset includes a training set D1-T and a validation set D1-E. The training set D1-T includes multiple sets of temporally continuous monocular images taken by a camera, and the validation set D1-E includes multiple discontinuous monocular images taken by a camera. Step 2: Input the training set D1-T mentioned in Step 1 into the unsupervised monocular depth estimation model in batches. Each batch of training data includes multiple sets of continuous monocular images. The unsupervised monocular depth estimation model outputs data for calculating point clouds and loss. The output data includes the depth map corresponding to the input image and the pose change of the camera device corresponding to the input image. Step 3: Based on the depth map and pose changes obtained in Step 2, calculate the reprojected image and point cloud corresponding to the input image; Step 4: Based on the depth map obtained in Step 2 and the reprojected image and point cloud obtained in Step 3, calculate the total training loss, which includes reprojection loss, point cloud consistency loss, and edge smoothing loss. Step 5: Use the total training loss obtained in Step 4 to train the unsupervised monocular depth estimation model from Step 2, optimize it to obtain the optimal unsupervised monocular depth estimation model, validate it using D1-E, and calculate the depth estimation index. Step 6: Prepare dataset D2 for 3D object detection. The dataset includes training set D2-T and validation set D2-E. Both training set D2-T and validation set D2-E include multiple discontinuous monocular images taken by a camera and the corresponding point cloud files of the images. The point cloud files are obtained by processing the depth map of the corresponding image using a depth map-point cloud conversion algorithm. The depth map is obtained by processing the corresponding image using the optimal unsupervised monocular depth estimation model obtained in step 5. Step 7: Train the 3D object detection model using the training set D2-T to obtain the optimal 3D object detection model, validate it using the validation set D2-E, and calculate the 3D object detection index.

2. The point cloud consistency-consistent monocular depth estimation method for 3D target detection according to claim 1, characterized in that: Step 4 calculates the total training loss, expressed as: in It is the total training loss. It is reprojection loss. It is a point cloud consistency loss. It is edge smoothing loss. , , It's a hyperparameter.

3. The point cloud consistency-consistent monocular depth estimation method for 3D target detection according to claim 2, characterized in that: The point cloud consistency loss is expressed as: in, It is a point cloud structure similarity metric function, used to measure the structural similarity between two point clouds. Point clouds represent a set of three-dimensional coordinates. The point cloud of the source frame calculated in step 3 is then processed. The point cloud obtained by the operation The point cloud is obtained by performing pose transformation on the point cloud of the target frame calculated in step 3. This indicates taking the median. This indicates division by point. It's a hyperparameter. and They are represented as follows: in It is a transformation operation based on the distorted image of the sampling points. This represents the depth calculated by the depth estimation model. Camera internal parameters of photographic equipment This refers to the pose change output by the model in step 2, the transformation operation. Pointer cloud According to sampling points To distort and deform, the sampling points These are the coordinates obtained by reprojecting the target frame coordinates onto the source frame, i.e., the points on the source frame corresponding to the target frame. Represented as: 。 4. The point cloud consistency-consistent monocular depth estimation method for 3D target detection according to claim 3, characterized in that: The Represented as: in , They are point clouds And point cloud , and They represent and The mean, and They represent and variance , It is a constant.

5. The point cloud consistency-consistent monocular depth estimation method for 3D target detection according to claim 1, characterized in that: The reprojection loss is expressed as: in It is a structural similarity metric function used to measure the structural similarity between two images. It is the target frame image selected from the continuous monocular images in step 2. It is a reprojected image reconstructed from a source frame image to a target frame, wherein the source frame is any frame in a continuous monocular image other than the target frame. It's a hyperparameter. Represented as: in It is a transformation operation based on the distorted image of the sampling points. It is the source frame image. These are the sampling points on the source frame, and the transformation operation... Refers to the source frame image According to sampling points To achieve distortion and deformation, the sampling point is the coordinate obtained by reprojecting the target frame coordinates onto the source frame, that is, the point on the source frame corresponding to the target frame. Represented as: in It refers to the camera's internal parameters for photographic equipment. This refers to the pose change output by the model in step 2, where pose refers to the change in pose from the target frame to the source frame. This represents the depth calculated by the depth estimation model. yes depth, These are the coordinates on the target frame. It is calculated using the above formula. The coordinates of the corresponding point on the source frame; in, Represented as: in , These are images and images , and Let X and Y represent the means of X and Y, respectively. and Let X and Y represent the variances, respectively. This represents the covariance of XY. , It is a constant.

6. The point cloud consistency-consistent monocular depth estimation method for 3D target detection according to claim 1, characterized in that: The edge smoothing loss is expressed as: in Mean normalization representing depth, Represents the target frame image. and These represent partial derivatives in the horizontal and vertical directions of the image, respectively.

Images