An unsupervised monocular depth estimation method based on point cloud learning

Abstract

The present invention belongs to the field of computer vision image processing and discloses an unsupervised monocular depth estimation method based on point cloud learning, comprising: preparing a dataset; inputting the training dataset into an unsupervised monocular point cloud estimation model in batches, the model outputting a predicted point cloud and pose change corresponding to the input image; calculating a reprojected image and reconstructed point cloud corresponding to the input image; calculating the total training loss, including reprojection loss, self-supervised constraint loss of the predicted point cloud, inter-frame constraint loss of the predicted point cloud, and edge smoothing loss; training the model using the total loss constraint to obtain an optimal unsupervised monocular point cloud estimation model; preparing a dataset for 3D object detection, replacing the actual point cloud file with the predicted point cloud file; and training the 3D object detection model to obtain the optimal 3D object detection model. The present invention further explores pure image methods, improving the depth estimation and 3D object detection accuracy of road scenes by directly estimating point clouds and introducing point cloud constraints.

Description

Technical Field

[0001] The present invention relates to the field of computer vision image processing, and more particularly to an unsupervised monocular depth estimation method based on point cloud learning. Background Art

[0002] Depth estimation of a scene is a classic problem in computer vision, playing a crucial role in 3D reconstruction, occlusion processing, and illumination estimation. The essence of obtaining depth information from an image is to construct a model that correlates image and depth information. Obtaining depth information facilitates a better understanding of 3D scenes and is also helpful for many computer vision tasks, such as indoor positioning, height estimation, simultaneous localization and mapping (SLAM), visual odometry, and classification. Compared to obtaining the depth of a certain number of points on an object's surface through various hardware devices such as laser rangefinders, image-based depth estimation methods have a wider range of applications because they do not require expensive equipment and specialized personnel.

[0003] Now, using deep learning has become the mainstream approach to research and solve the problem of depth estimation. Faced with the expensive lidar sensors and the huge cost of ultra-large-scale manually labeled data sets, unsupervised monocular depth estimation methods are a hot topic of current research.

[0004] Specifically in the field of intelligent driving, that is, in road scenes, if you want to achieve reliable and stable target detection, then you must require fast and accurate depth estimation. This shows that the accuracy of depth estimation will greatly affect the effect of 3D target detection, and 3D target detection can be regarded as a downstream task of depth estimation. In terms of depth estimation, previous explorations often only thought and practiced in the pure image direction. From the perspective of 3D target detection, focusing on the impact of depth estimation accuracy on the effect of 3D target detection, unsupervised monocular depth estimation solutions have not been optimized. In terms of 3D target detection, most sensors in real intelligent driving systems use lidar data as input, and the accuracy of 3D target detection solutions with pure images as input is not ideal.

[0005] The pseudo-radar method (Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, Kilian Q. Weinberger. Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving. In CVPR, 2019) builds a bridge between the upstream and downstream tasks of depth estimation and lidar-based 3D target detection. By converting the predicted depth map into pseudo-radar (Pseudo-LiDAR), the pseudo-radar can be directly used to train the lidar-based 3D target detection model. Although this means that the lidar-based 3D target detection task no longer relies on real lidar data, it does introduce additional data conversion operations between upstream and downstream tasks. For example, ZL2023100625454 discloses a point cloud consistency constrained monocular depth estimation method for 3D target detection. Under pure image methods, the unsupervised monocular depth estimation task and the 3D target detection task must rely on pseudo-radar. Summary of the Invention

[0006] The present invention proposes an unsupervised monocular depth estimation method based on point cloud learning, which is used to solve the problem that the unsupervised monocular depth estimation task and the 3D target detection task must rely on the pseudo-radar method under pure image methods. The method proposes an unsupervised monocular point cloud estimation model and uses point cloud constraints to train the model, thereby improving the accuracy of unsupervised monocular depth estimation and 3D target detection.

[0007] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0008] The present invention is an unsupervised monocular depth estimation method based on point cloud learning, comprising:

[0009] An unsupervised monocular point cloud estimation model, comprising a codec architecture, an encoder, a ResNet18 model, for extracting features from a scene image; and a decoder, a point cloud decoder, for decoding the features and outputting a predicted point cloud of the scene. The point cloud decoder comprises multiple upsampling layers and output layers, wherein the multiple output layers correspond to three dimensions of the predicted point cloud corresponding to the input image, the three dimensions corresponding to the three dimensions of three-dimensional space, and the third dimension corresponding to depth.

[0010] Total training loss, including reprojection loss, self-supervised constraint loss of predicted point cloud, inter-frame constraint loss of predicted point cloud, and edge smoothing loss;

[0011] A training method comprises preparing a dataset for unsupervised monocular depth estimation, comprising a monocular image of a road scene, and dividing the dataset into a depth estimation training set and a depth estimation verification set; inputting the depth estimation training set into the unsupervised monocular point cloud estimation model in batches, and the model outputs a predicted point cloud and camera pose change corresponding to the input image, and calculates a reprojected image and a reconstructed point cloud corresponding to the input image; calculating a total training loss; using the total training loss to train and optimize the unsupervised monocular point cloud estimation model; using the depth estimation verification set to evaluate the depth estimation performance of the model; preparing a dataset for 3D target detection, comprising a monocular image of a road scene and its corresponding predicted point cloud, wherein the predicted point cloud is estimated by the unsupervised point cloud estimation model, and dividing the 3D target detection dataset into a 3D detection training set and a 3D detection verification set; using the 3D detection training set to train a 3D target detection model to obtain an optimal 3D target detection model, and verifying the model on the 3D detection verification set.

[0012] Furthermore, the training method specifically includes the following steps:

[0013] Step S1: preparing a dataset D1 for unsupervised monocular depth estimation, the dataset including a training set D1-T and a validation set D1-E, the training set including multiple groups of temporally continuous monocular images taken by a photographic device, and the validation set including multiple discontinuous monocular images taken by the photographic device;

[0014] Step S2: Input the training set D1-T into the unsupervised monocular point cloud estimation model in batches, where each batch of training data includes multiple sets of continuous monocular images. The model outputs data for calculating the loss, and the output data includes the predicted point cloud corresponding to the input image and the change in the camera pose corresponding to the input image.

[0015] Step S3: Calculate the reprojected image and reconstructed point cloud corresponding to the input image based on the point cloud and pose changes obtained in step S2;

[0016] Step S4: Calculate the total training loss based on the predicted point cloud obtained in step S2 and the reprojected image and reconstructed point cloud obtained in step S3, wherein the total training loss includes the reprojection loss, the self-supervised constraint loss of the predicted point cloud, the inter-frame constraint loss of the predicted point cloud, and the edge smoothing loss;

[0017] Step S5: Using the total training loss obtained in step S4 to train the unsupervised monocular point cloud estimation model described in step S2, optimizing to obtain the optimal unsupervised monocular point cloud estimation model, verifying it using D1-E, and calculating the depth estimation index;

[0018] Step S6: Prepare a dataset D2 for 3D object detection. The dataset includes a training set D2-T and a validation set D2-E. The training set and validation set both include multiple discontinuous monocular images taken by a camera and point cloud files corresponding to the images. The point cloud files are the predicted point clouds corresponding to the input images obtained in step S2.

[0019] Step S7: Use the training set D2-T to train the 3D object detection model to obtain the optimal 3D object detection model, use the validation set D2-E to verify it, and calculate the 3D object detection index.

[0020] Furthermore, the total training loss is expressed as:

[0021] L=αL rp +β1L s-pcc +β2L f-pcc +γL es

[0022] Where L is the total training loss, L rp is the reprojection loss, L s-pcc is the self-supervised constraint loss for predicting point clouds, L f-pcc is the inter-frame constraint loss function for predicting point clouds, L es is the edge-aware smoothness loss, and α, β1, β2, and γ are hyperparameters.

[0023] Furthermore, the reprojection loss is expressed as:

[0024]

[0025] SSIM is a structural similarity metric function used to measure the structural similarity between two images. t is the target frame image selected from the continuous monocular image in step S2, I s→t is the reprojected image of the source frame image calculated in step S3 to reconstruct the target frame, the source frame is the other frames in the continuous monocular image except the target frame, μ is a hyperparameter, I s→t Expressed as:

[0026] I s→t =warp(I s,p s ′ )

[0027] Warp is a transformation operation that distorts the image based on the sampling points. s is the source frame image, p s ′ is a sampling point on the source frame, the transformation operation warp(I s ,p s ′ ) refers to image I s According to the sampling point p s ′ To distort the deformation, the sampling point is the coordinate obtained by reprojecting the target frame coordinates to the source frame, that is, the point on the source frame corresponding to the target frame, p s ′ Expressed as:

[0028] p s ′ ~KT t→s D(p t )K -1 p t

[0029] Where K is the camera internal parameter of the photographic equipment, T t→s is the pose change output by the model in step 2, which refers to the pose change from the target frame to the source frame. D represents the depth calculated by the unsupervised monocular point cloud estimation model, which is the third dimension of the point cloud output by the point cloud estimation model. t ) is p t The depth of p t is the coordinate on the target frame, p s ′ is the p calculated by the above formula t The coordinates of the corresponding points on the source frame;

[0030] In addition, SSIM is expressed as:

[0031]

[0032] Among them I X , I Y are image X and image Y, μ X and μ Y represent the mean of X and Y respectively, σ X and σ Y Denote the variance of X and Y, σ XY represents the covariance of XY, where C1 and C2 are constants.

[0033] Furthermore, the self-supervised constraint loss of the predicted point cloud is expressed as:

[0034]

[0035] Among them, SSIM pc It is a point cloud structure similarity measurement function, which is used here to measure the structural similarity of two point clouds. P represents the reconstructed point cloud described in step 3, Q represents the predicted point cloud described in step 2, and P t Represents the reconstructed point cloud corresponding to the target frame image, Q t represents the predicted point cloud corresponding to the target frame image, median represents the median, div represents point-by-point division, υ is a hyperparameter, P t Expressed as:

[0036] P t =D(p t )K -1 p t

[0037] In addition, SSIM pc Expressed as:

[0038]

[0039] Among them, P X 、P Y They are point cloud X and point cloud Y, μ X and μ Y represent the mean of X and Y respectively, σ X and σ Y Denote the variance of X and Y respectively, C1 and C2 are constants.

[0040] Furthermore, the inter-frame constraint loss of the predicted point cloud is expressed as:

[0041]

[0042] Where Q represents the predicted point cloud described in step 2, Q s ′ Represents the point cloud obtained by warping the predicted point cloud of the source frame, Q t represents the predicted point cloud of the target frame, Q t→s represents the point cloud obtained by performing pose transformation on the predicted point cloud of the target frame, Q s ′ and Q t→s Respectively expressed as:

[0043] Q s ′ =warp(Q s ,p s ′ )

[0044] Q t→s =T t→s Qt

[0045] Transformation operation warp(Q s ,p s ′ ) refers to the point cloud Q s According to the sampling point p s ′ To distort and deform;

[0046] Furthermore, the edge smoothing loss is expressed as:

[0047]

[0048] in represents the mean normalization of depth, I t represents the target frame image, and Respectively represent the partial derivatives in the horizontal and vertical directions of the image.

[0049] The beneficial effects of the present invention are as follows: the unsupervised monocular depth estimation method based on point cloud learning provided by the present invention directly estimates point clouds without relying on pseudo radar, which is more beneficial to 3D target detection;

[0050] This paper deeply explores the pure image method, directly estimates the point cloud for the downstream 3D target detection algorithm, and introduces point cloud constraints from planar constraints to three-dimensional space constraints, which not only improves the depth estimation effect in road scenes, but also improves the 3D target detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flowchart of an embodiment of the present invention.

[0052] Figure 2 Schematic diagram of the point cloud estimation network structure according to an embodiment of the present invention.

[0053] Figure 3 This is a detailed diagram of the point cloud estimation network structure of an embodiment of the present invention.

[0054] Figure 4 This is a point cloud constraint solution of an embodiment of the present invention.

[0055] Figure 5 This is a depth estimation rendering and 3D target detection rendering of a certain scene implemented in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following diagrams illustrate embodiments of the present invention. For clarity, many practical details are included in the following description. However, it should be understood that these practical details are not intended to limit the present invention. In other words, in some embodiments of the present invention, these practical details are not essential.

[0057] The present invention is an unsupervised monocular depth estimation method based on point cloud learning, comprising the following steps:

[0058] (1) Prepare a dataset for unsupervised monocular depth estimation

[0059] Prepare a dataset D1 for unsupervised monocular depth estimation, the dataset including a training set D1-T and a validation set D1-E. The training set includes multiple groups of temporally continuous monocular images taken by a photographic device, and the validation set includes multiple discontinuous monocular images taken by the photographic device. D1 uses the original KITTI dataset, which includes: ① raw grayscale image sequences (i.e., unsynchronized, uncorrected, and processed (synchronized, corrected)) acquired at 10Hz, 1242*375 pixels, in PNG format; ② raw color image sequences (i.e., unsynchronized, uncorrected, and processed (synchronized, corrected)) acquired at 10Hz, 1242*375 pixels, in PNG format; ③ 3D Velodyne point cloud (i.e., 100k points per frame, stored as a binary floating-point matrix in bin format); ④ 3D GPS / IMU data (i.e., position, velocity, acceleration, and metadata, stored in text format); ⑤ calibration data (i.e., camera, camera-to-GPS / IMU, and camera-to-Velodyne calibration data, stored in text format); and ⑥ 3D object trajectory labels (i.e., car, truck, tram, pedestrian, and cyclist, in XML format). The dataset is divided into training and validation sets using Eigen's data partitioning scheme. D1-T contains 39,810 monocular images, and D1-E contains 4,424 monocular images.

[0060] (2) Training unsupervised monocular point cloud estimation model

[0061] The unsupervised monocular point cloud estimation model includes a point cloud prediction network and a pose estimation network. The point cloud prediction network is an encoder-decoder architecture, including an encoder ResNet18 model and a point cloud decoder. The network structure diagram is as follows Figure 2 The network structure details are shown in Figure 3 As shown. Its input is a monocular image I t , the output is the predicted point cloud Q of the monocular image t The pose estimation network is also an encoder-decoder architecture, including an encoder ResNet18 model and a pose decoder shared with the depth prediction network. Its input is two frames of monocular images I t , I s, the output is the pose change T from the target frame to the source frame t→s .

[0062] The training set D1-T is input into the unsupervised monocular depth estimation model in batches. Each batch of training data includes multiple sets of continuous monocular images. The model outputs data for calculating the loss. The output data includes the predicted point cloud corresponding to the input image and the change in the photographic device pose corresponding to the input image.

[0063] The loss calculation scheme includes calculating the reprojection loss, the self-supervised constraint loss of the predicted point cloud, the inter-frame constraint loss of the predicted point cloud, and the edge smoothing loss. The constraint schemes of the self-supervised constraint loss of the predicted point cloud and the inter-frame constraint loss of the predicted point cloud are as follows: Figure 4 shown.

[0064] Specifically, take two frames of monocular images I in training t , I s Take the unsupervised monocular point cloud estimation model as an example. Through the point cloud prediction network, we can obtain I t , I s The predicted point cloud Q t , Q s The third dimension of the predicted point cloud is regarded as the depth estimated by the point cloud prediction network, that is, I t , I s The depth map D t 、D s , and I t , I s The reconstructed point cloud P t 、P s :

[0065] P t =D(p t )K -1 p t

[0066] P s =D(p s )K -1 p s

[0067] Obtain the pose change T from the target frame to the source frame through the pose estimation network s→s .according to

[0068] p s ′ ~KT t→s D(p t )K -1 p t

[0069] and

[0070] Is→t =warp(I s ,p s ′ )

[0071] In the present invention, warp is implemented using the grid_sample method provided by the PyTorch deep learning framework.

[0072] Obtain the "pseudo target frame" image I reconstructed from the source frame to the target frame s→t , thereby establishing a self-supervisory constraint between the original target frame and the “pseudo target frame” image - the reprojection loss:

[0073]

[0074] In the embodiment, μ is set to 0.85.

[0075] for

[0076]

[0077] In the embodiment, C1=0.0001 and C2=0.0009 are set.

[0078] Due to obtaining I t The predicted point cloud Q t and reconstruct the point cloud P t , a self-supervised point cloud constraint can be established between the predicted point cloud and the reconstructed point cloud - the self-supervised constraint loss of the predicted point cloud:

[0079]

[0080] Div normalizes the point cloud data at the microscopic level of each point, aiming to use relative distances rather than absolute distances. Median normalizes the point cloud constraints at the overall level of the point cloud. This approach aims to prevent the estimated depth from being too small due to the introduction of point cloud constraints, which can lead to a scaling issue. Both methods aim to address the drawback of unsupervised monocular depth estimation models, which only output relative distances rather than absolute distances.

[0081] In the embodiment, υ is set to 0.85.

[0082] for

[0083]

[0084] In the embodiment, C1=0.0001 and C2=0.0009 are set.

[0085] Due to obtaining I t , I s The predicted point cloud Q t , Qs , combined with the pose change T s→s And warp operation, you can calculate the source frame prediction point cloud according to the sampling point distortion transformation point cloud Q s ′ , and the point cloud Q after the target frame predicted point cloud pose changes t→s :

[0086] Q s ′ =warp(Q s ,p s ′ )

[0087] Q t→s =T t→s Q t

[0088] Get Q s ′ and Q t→s Finally, point cloud constraints can be established between the predicted point clouds of different frames - the inter-frame constraint loss of the predicted point clouds:

[0089]

[0090] median means taking the median, div means point-by-point division, υ is a hyperparameter,

[0091] In the embodiment, υ is set to 0.85.

[0092] for

[0093]

[0094] In the embodiment, C1=0.0001 and C2=0.0009 are set.

[0095] Similarly, add edge smoothing loss:

[0096]

[0097] This loss can handle the edges of objects in the image well, so that objects can be clearly distinguished in depth.

[0098] Combining the above constraints, we get the total loss function:

[0099] L=αL rp +β1L s-pcc +β2L f-pcc +γL es

[0100] In the embodiment, α=1, β1=0.002, β2=0.002, and γ=0.001 are set.

[0101] The unsupervised monocular point cloud estimation model is trained using the total training loss, and the optimal unsupervised monocular point cloud estimation model is obtained through optimization.

[0102] The above process uses an NVIDIA GTX 1080TI to train the model. The PyTorch deep learning framework is used to build an unsupervised monocular depth estimation model. The batch size is set to 4, and the training is carried out for 20 epochs, with a total of 99,525 iterations. The Adam optimizer is used, and the initial learning rate is set to 0.8e-4. The learning rate is reduced to 0.1 times the original value at the 12th and 16th epochs, respectively.

[0103] (3) Verify the depth estimation accuracy of the unsupervised monocular point cloud estimation model

[0104] The optimal unsupervised monocular point cloud estimation model in (2) was verified using the validation set D1-E. Currently, the most commonly used quantitative indicators for evaluating monocular depth estimation are absolute relative error (Abs Rel), relative error (Sq Rel), root mean square error (RMSE), logarithmic root mean square error (RMSE log), and threshold accuracy. The following are the calculation formulas for each quantitative indicator:

[0105]

[0106]

[0107]

[0108]

[0109]

[0110] where d i is the actual depth value, is the predicted depth value. The threshold thr is commonly set to 1.25, 1.252, or 1.253. Lower absolute relative error (Abs Rel), relative error (Sq Rel), root mean square error (RMSE), and logarithmic root mean square error (RMSElog) indicate better depth estimation. Higher threshold accuracy indicates better depth estimation.

[0111] Since the unsupervised monocular point cloud estimation model outputs a predicted point cloud, the third dimension of the predicted point cloud is regarded as a depth map and evaluated using the above quantitative indicators.

[0112] The experimental results of the embodiment are shown in Table 1 below, which are compared with the MonoDepth2 algorithm (Cl'ement Godard, OisinMac Aodha, Michael Firman, Gabriel Brostow. Digging Into Self-Supervised Monocular Depth Estimation. In ICCV, 2019), which proves that the present invention improves the monocular depth estimation effect.

[0113] Table 1

[0114] Abs Rel Sq Rel RM SE RM SE bg δ<1.25 <![CDATA[δ<1.25 2 ]]> <![CDATA[δ<1.25 3 ]]> MonoDepth2 0.115 0.903 4.863 0.193 0.877 0.959 0.981 The present invention 0.112 0.823 4.715 0.190 0.879 0.960 0.982

[0115] (4) Prepare a dataset for 3D object detection

[0116] Prepare a dataset D2 for 3D object detection. The dataset includes a training set D2-T and a validation set D2-E. Both the training set and the validation set consist of multiple discrete monocular images captured by a camera and the corresponding real-world LiDAR images. D2 uses the KITTI object detection dataset, which includes ① color images in PNG format; ② Velodyne point clouds (real-world LiDAR images) in bin format; and ③ object labels stored as text. D2-T contains 3712 training images, their corresponding real-world LiDAR images, and object labels, while D2-E contains 3769 training images, their corresponding real-world LiDAR images, and object labels.

[0117] Use the unsupervised monocular point cloud estimation model trained in (2) to output the predicted point clouds of all color images in D2, perform scale registration on the predicted point clouds and the Velodyne point clouds in D2, and then replace the Velodyne point clouds in D2 with the registered predicted point clouds.

[0118] The registration scheme uses the median registration method. The point cloud can be regarded as a set of vectors, so the ratio of the medians of their respective moduli is taken as the size ratio of the point cloud:

[0119]

[0120] Among them, P ground-truth Denotes the Velodyne point cloud, and Q denotes the predicted point cloud. The predicted point cloud output by the unsupervised monocular point cloud estimation model is scaled to the scale of the Velodyne point cloud based on the point cloud size ratio.

[0121] (5) Training 3D object detection model

[0122] The 3D object detection model adopts the AVOD algorithm. The dataset D2-T prepared in (4) is used to train AVOD and the optimal AVOD 3D object detection model is obtained through optimization.

[0123] The model was trained using NVIDIA GTX 1080TI and the TensorFlow deep learning framework was used to build the AVOD model. The training iterations were 120,000 and the Adam optimizer was used. The initial learning rate was set to 1e-4 and the learning rate was reduced to 0.8 times the original value every 30,000 iterations.

[0124] (6) Verify the 3D object detection model

[0125] The optimal AVOD model obtained in (5) is verified using the validation set D2-E, and the 3D object detection metrics are calculated. The two 3D object detection metrics are AP 3D and AP BEV. The former is used to evaluate the accuracy of the 3D detection box, and the latter is used to evaluate the accuracy of the detection box under the BEV view.

[0126] The experimental results of the embodiment are shown in Table 2 below.

[0127]

[0128] The table above shows the performance of AP 3D and AP BEV of the present invention at IoU = 0.5. Compared with Mono3D (a supervised algorithm), this demonstrates the potential of unsupervised algorithms. Compared with MonoDepth2+Pseudo-LiDAR, this demonstrates that the present invention improves the accuracy of 3D object detection, specifically optimizing depth estimation and 3D object detection for specific useful objects in road scenes.

[0129] The depth estimation and 3D detection effect diagram of the embodiment is as follows Figure 5 As shown, the first picture from top to bottom is the original picture of the scene, the second picture is the rendering of the depth estimation of the present invention, the brighter the color, the closer the distance, the third picture and the fourth picture are the 2D detection frame rendering and the 3D detection frame rendering, respectively. The red frame line represents the true label of the target detection frame, and the green frame line represents the detection frame drawn by the target detection model.

[0130] This paper delves into pure image methods, improves the depth estimation and 3D object detection accuracy of road scenes by directly estimating point clouds and introducing point cloud constraints.

[0131] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. It will be apparent to those skilled in the art that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. An unsupervised monocular depth estimation method based on point cloud learning, characterized by: The unsupervised monocular depth estimation method comprises the following steps: Step 1: Prepare a dataset D1 for unsupervised monocular depth estimation. The dataset includes a training set D1-T and a validation set D1-E. The training set includes multiple groups of temporally continuous monocular images taken by a photographic device, and the validation set includes multiple discontinuous monocular images taken by a photographic device. Step 2: Input the training set D1-T into the unsupervised monocular point cloud estimation model in batches, where each batch of training data includes multiple sets of continuous monocular images. The model outputs data for calculating the loss, and the output data includes the predicted point cloud corresponding to the input image and the change in the camera pose corresponding to the input image. The unsupervised monocular point cloud estimation model is an encoder-decoder architecture, where the encoder is a ResNet18 model for extracting features of the scene image, and the decoder is a point cloud decoder for decoding the features of the scene image and outputting the predicted point cloud of the scene; Step 3: Based on the predicted point cloud obtained in step 2 and the change in the camera's pose, calculate the reprojected image and reconstructed point cloud corresponding to the input image; Step 4: Calculate the total training loss based on the predicted point cloud obtained in step 2, the reprojected image obtained in step 3, and the reconstructed point cloud, wherein the total training loss includes the reprojection loss, the self-supervised constraint loss of the predicted point cloud, the inter-frame constraint loss of the predicted point cloud, and the edge smoothing loss. The self-supervised constraint loss of the predicted point cloud is expressed as: Among them, SSIM pc It is a point cloud structure similarity measurement function used to measure the structural similarity of two point clouds. P represents the reconstructed point cloud of step 3, Q represents the predicted point cloud of step 2, and P t Represents the reconstructed point cloud corresponding to the target frame image, Q t represents the predicted point cloud corresponding to the target frame image, median represents the median, div represents point-by-point division, υ is a hyperparameter, P t Expressed as: P t =D(p t )K -1 p t Where K represents the camera intrinsic parameter of the photographic device, D represents the depth calculated by the unsupervised monocular point cloud estimation model, and the depth is the third dimension of the point cloud output by the point cloud estimation model. t ) is p t The depth of p t are the coordinates on the target frame; The inter-frame constraint loss of the predicted point cloud is expressed as: Among them, Q represents the predicted point cloud of step 2, Q′ s Indicates the point cloud obtained by performing a warp operation on the predicted point cloud of the source frame according to the sampling point distortion point cloud, Q t represents the predicted point cloud of the target frame, Q t→s Represents the point cloud obtained by performing pose transformation on the predicted point cloud of the target frame, SSIM pc It is the point cloud structure similarity measurement function, median means taking the median, div means point-by-point division, υ is a hyperparameter, Q′ s and Q t→s Respectively expressed as: Q′ s =warp(Q s ,p′ s ) Q t→s =T t→ sQ t Among them, T t→s is the pose change of the model output, warp is the transformation operation of distorting the point cloud based on the sampling point, Q s is the predicted point cloud of the source frame, p′ s is the sampling point on the source frame, the transformation operation warp(Q′ s ,p′ s ) refers to the source frame predicted point cloud Q s According to the sampling point p′ s To distort the deformation, the sampling point is the coordinate obtained by reprojecting the target frame coordinates to the source frame, that is, the point on the source frame corresponding to the target frame, p′ s Expressed as: p′ s ~KT t→s D(p t )K -1 p t Where K represents the camera intrinsic parameter of the photographic device, D represents the depth calculated by the unsupervised monocular point cloud estimation model, and the depth is the third dimension of the point cloud output by the point cloud estimation model; Step 5: Use the total training loss obtained in step 4 to train the unsupervised monocular point cloud estimation model described in step 2 to obtain the optimal unsupervised monocular point cloud estimation model. Use the training set D1-E to verify the optimal unsupervised monocular point cloud estimation model and calculate the depth estimation index. Step 6: Prepare a dataset D2 for 3D object detection. The dataset includes a training set D2-T and a validation set D2-E. Both the training set and the validation set include multiple discontinuous monocular images taken by a camera and point cloud files corresponding to the images. The point cloud files are the predicted point clouds corresponding to the input images obtained in step 2. Step 7: Use the training set D2-T to train the 3D object detection model to obtain the optimal 3D object detection model. Use the validation set D2-E to verify the optimal 3D object detection model and calculate the 3D object detection indicators.

2. The unsupervised monocular depth estimation method based on point cloud learning according to claim 1, characterized in that: In step 2, the point cloud decoder includes multiple upsampling layers and output layers, and the multiple output layers correspond to three dimensions of the predicted point cloud corresponding to the input image, and the three dimensions correspond to the three dimensions of the three-dimensional space respectively, and the third dimension of the three dimensions corresponds to depth.

3. The unsupervised monocular depth estimation method based on point cloud learning according to claim 1, characterized in that: In step 4, the total training loss is calculated as: L=αL rp +β1L s-pcc +β2L f-pcc +γL es Where L is the total training loss, L rp is the reprojection loss, L s-pcc is the self-supervisory constraint loss for predicting point clouds, L f-pcc is the inter-frame constraint loss of the predicted point cloud, L es is the edge smoothing loss, and α, β1, β2, and γ are hyperparameters.

Images