A self-supervised monocular depth prediction training method based on pixel matching prediction of camera pose

Abstract

The present application belongs to the field of machine learning, 3D computer vision, monocular depth prediction and self-supervised learning, and provides a self-supervised monocular depth prediction training method based on pixel matching for predicting camera pose. The present application converts the abstract camera motion prediction process in self-supervised depth prediction learning into a process based on pixel matching, and solves the camera motion in a geometric manner, enhances its interpretability, and improves its generalizability. The camera motion solved by the traditional geometric method of the present application is more accurate, so that the self-supervised depth prediction learning is more stable in the training process and the effect is more robust in the scene of indoor environment or large change of camera motion pose.

Description

Technical Field

[0001] This invention belongs to the fields of machine learning, 3D computer vision, monocular depth prediction, and self-supervised learning, and relates to the EPnP algorithm, specifically a self-supervised monocular depth prediction training method based on pixel matching to predict camera pose. Background Technology

[0002] In the field of 3D computer vision, monocular depth prediction plays an important role in many practical applications, such as autonomous driving, robot navigation, and virtual reality, with the aim of predicting pixel-level dense depth values ​​from a single image.

[0003] With the development of neural networks, depth prediction algorithms have made great progress. Eigen first used a neural network to predict depth maps from a single image in 2014. Subsequent methods have improved the accuracy of depth prediction by improving network structure, introducing multi-task training, and incorporating auxiliary geometric cues. However, these supervised methods all require a large amount of ground truth depth annotations to support network training, and the process of obtaining ground truth depth data is usually time-consuming and labor-intensive, requiring high-precision sensors and complex post-processing.

[0004] In recent years, many self-supervised learning methods have been proposed to alleviate this problem. Self-supervised learning methods take two consecutive images as input, reconstruct one image from the other using the predicted depth map and camera pose, and supervise network training by minimizing the image reconstruction loss. Since self-supervised methods require predicting additional camera pose, common methods, such as Monodepth2 (Digging Into Self-Supervised MonocularDepth Estimation) proposed by Godard et al., use a multi-layer convolutional network, PoseNet, to directly predict the 6-DOF camera pose from the stitched image pair. However, this method has the following drawbacks:

[0005] 1. Camera motion pose can be solved using traditional geometric methods (such as Structure-from-Motion), but using neural networks for prediction lacks interpretability and has limited generalization, thus affecting the learning of self-supervised deep prediction.

[0006] 2. The method of predicting camera motion through neural networks cannot predict accurate results in low-texture areas or when the camera pose changes greatly. Therefore, it can only achieve good results on outdoor driving datasets with relatively simple camera motion, but cannot achieve ideal results on indoor datasets. Summary of the Invention

[0007] This invention aims to provide a self-supervised monocular depth prediction training method based on pixel matching to predict camera pose, thereby solving the problem that the inaccurate camera motion pose prediction by PoseNet leads to limited depth prediction accuracy and generalization of self-supervised learning.

[0008] The method described in this invention can be trained and tested on indoor and outdoor datasets, and achieves significant performance improvements when there are large variations in camera motion.

[0009] The technical solution of this invention is as follows:

[0010] A self-supervised monocular depth prediction training method based on pixel matching for camera pose prediction is proposed. This method uses geometric 3D-2D feature point pairs to solve for camera pose, aiding in self-supervised learning of monocular depth. The steps are as follows:

[0011] Step 1: Input two consecutive image frames (target frame and source frame), and obtain dense optical flow predictions as matching results through an optical flow prediction network;

[0012] The optical flow prediction network, based on PWC-Net, constructs a CostVolume using features extracted from two frames of images and then uses this CostVolume to find matching information that minimizes the CostVolume. The optical flow network mainly consists of a feature extraction network and a CostVolume matching network. The feature extraction network constructs a feature pyramid using convolutional and residual modules to obtain downsampled features at 2x, 4x, 8x, 16x, and 32x. The CostVolume matching network constructs the CostVolume using image features from two frames at the same resolution by calculating their correlation.

[0013]

[0014] Where i represents the number of feature layers, x1 and x2 represent the image features of two frames respectively, and N represents the number of features. The optical flow prediction of the i-th layer is output from the cost volume through a five-layer convolutional module. The specific steps are as follows:

[0015] (1.1) Construct the cost volume using the features of the i-th layer and output the optical flow prediction of the i-th layer.

[0016] (1.2) The optical flow prediction of the i-th layer is upsampled by 2 times as the optical flow prior of the (i-1)-th layer. The features of the (i-1)-th layer are warped and the cost volume is constructed. Finally, the optical flow of the (i-1)-th layer is predicted.

[0017] (1.3) Repeat this step until the output is an optical flow prediction with the same resolution as the image.

[0018] Step 2: Assign a reliability weight to each pair of matching points using an adaptive weight prediction network. The input to the adaptive weight prediction network is the feature vector of the corresponding position of the matching point. After concatenation, the output is restricted to the range [0, 1] through five layers of 1D convolution and a sigmoid layer, which serves as the predicted weight value.

[0019] Step 3: Input the image of the target frame and obtain the depth map through a monocular depth prediction network.

[0020] Depth prediction networks use an Encoder-Decoder structure, where the encoder extracts features step by step, and the decoder gradually improves the feature resolution and predicts the depth.

[0021] Step 4: Project the matching points on the target frame into 3D space using the predicted depth values ​​and camera intrinsic parameters:

[0022] P t =D t (p t )K -1 p t

[0023] Where D t The value represents the predicted depth, K represents the camera intrinsic parameter, and p t P t These represent the corresponding 2D and 3D coordinate values, respectively.

[0024] Step 5: Using 3D-2D matching points and their corresponding weights, solve for the corresponding camera motion using the EPnP algorithm to find the camera motion that satisfies the minimum energy function.

[0025]

[0026] in These represent the 3D and 2D coordinate values ​​of the target frame and the source frame, respectively. i To correspond to the reliability weights, π represents the 3D-2D projection of the pinhole model:

[0027] Step 6: After obtaining the predicted depth map and camera motion pose, reconstruct the target frame using the source frame with the help of camera intrinsic parameters:

[0028] p src =KTD tgt (p tgt )K -1 p tgt

[0029] Where p src p tgtRepresents the pixel coordinates of the corresponding positions in the source and target frames, where T represents the predicted camera motion, and D... tgt The depth map is represented by K, which represents the camera intrinsics. The network is then trained by constraining the minimization of the photometric loss between the original and reconstructed images.

[0030] The beneficial effects of this invention are:

[0031] (1) The abstract camera motion prediction process in self-supervised deep prediction learning is transformed into a pixel matching-based process, and the camera motion is solved geometrically to enhance its interpretability and improve its generalization.

[0032] (2) The camera motion is solved more accurately by using traditional geometric methods, making the training process of self-supervised deep prediction learning more stable and robust in indoor environments or scenarios with large changes in camera motion posture. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall process structure.

[0034] Figure 2 This is a schematic diagram of a deep prediction network structure.

[0035] Figure 3 This is a schematic diagram of the optical flow prediction network structure.

[0036] Figure 4 This is a schematic diagram of the adaptive weight prediction network structure. Detailed Implementation

[0037] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0038] Figure 1 This is a schematic diagram of the overall process structure. First, optical flow maps are predicted from two frames of images to find 2D-2D matching points, and an adaptive weight prediction network predicts reliability weights for each pair of matching points. The depth prediction network predicts a depth map from a single frame image and projects the matching points on the map into 3D space using camera intrinsic parameters to obtain weighted 3D-2D matching points. The camera motion pose of the 3D-2D matching points is solved using the EPnP algorithm. The target frame image is reconstructed from the source frame image using the predicted depth map and camera motion pose, and then trained using a network that minimizes photometric error constraints.

[0039] Figure 2 This is a schematic diagram of the depth prediction network structure. The network consists of an encoder-decoder structure. The encoder part uses a ResNet structure to extract feature maps with different downsampling factors, and the decoder part gradually improves the resolution of the features and predicts the depth map.

[0040] Figure 3 This diagram illustrates the structure of an optical flow prediction network, which primarily consists of a feature extraction network and a cost volume matching network. The feature extraction network constructs a feature pyramid using convolutional and residual modules to obtain downsampled features at 2x, 4x, 8x, 16x, and 32x. The cost volume matching network uses image features from two frames at the same resolution to calculate the correlation and construct a cost volume, which is then used to predict optical flow. The same steps are performed progressively at different resolutions until the highest resolution optical flow prediction is output.

[0041] Figure 4 The diagram illustrates the adaptive weight prediction network structure. The network consists of five convolutional layers, and a sigmoid layer restricts the reliability weight output to the range [0, 1]. The network input is the feature vectors of the corresponding positions of the matching point pairs, which are concatenated and then fed into the network.

[0042] The network's training set consists of the KITTI training set and the NYUv2 dataset, which are augmented using random color jitter and random horizontal flipping. The Adam optimization method was chosen, with an initial learning rate of 0.0001, and training for 20 epochs, with the learning rate gradually decreasing during training. The input image size for both training and inference was 192×640.

Claims

1. A self-supervised monocular depth prediction training method based on pixel matching prediction of camera pose, which helps self-supervised learning monocular depth by solving camera pose through geometric 3D-2D feature point pairs, characterized in that, The steps are as follows: Step 1: input two consecutive images, target frame and source frame, and obtain dense optical flow prediction as matching results through an optical flow prediction network; The optical flow prediction network is based on PWC-Net, which constructs a cost volume by using features extracted from two consecutive images and finds matching information that minimizes the cost volume; the optical flow prediction network mainly consists of a feature extraction network and a cost volume matching network; the feature extraction network constructs a feature pyramid through convolution modules and residual modules to obtain down-sampled features at 2 times, 4 times, 8 times, 16 times and 32 times; the cost volume matching network uses features of two consecutive images at the same resolution to construct a cost volume by calculating correlation: ； wherein, the number of layers of features, , represent two consecutive image features, respectively, the number of features; The fifth layer of the cost volume is outputted by a five-layer convolution module The light flow prediction of the layer is as follows: (1.1) by the first layer of the feature construction cost volume and output the first layer of the optical flow prediction; (1.2) upsample the optical flow of the 2nd layer by 2 as the optical flow prior of the 1st layer, and align the features of the 1st layer through it to construct a cost volume, and finally predict the optical flow of the 1st layer; (1.3) repeat steps (1.1)-(1.2) until the output is the same as the image resolution of the optical flow prediction; Step 2: Assign a reliability weight to each matched point pair by an adaptive weight prediction network; the input of the adaptive weight prediction network is the concatenated feature vectors of the matched point pair, and the output is limited between 0 and 1 by a five-layer 1D convolution and a Sigmoid layer, as the predicted weight value. between 0 and 1 by a five-layer 1D convolution and a Sigmoid layer, as the predicted weight value. Step 3: input the image of the target frame, and obtain the depth map through the monocular depth prediction network; The monocular depth prediction network uses an encoder-decoder structure, which extracts features step by step through the encoder and improves the feature resolution and predicts the depth through the decoder; Step 4: project the matching points on the target frame to the 3D space through the predicted depth map and the camera intrinsic parameters: ； wherein, represent predicted depth values, represent camera intrinsic parameters, , respectively represent corresponding 2D, 3D coordinate values; Step 5: use the EPnP algorithm to solve the corresponding camera motion through the 3D-2D matching points and their corresponding weights, and find the camera motion that satisfies the minimum energy function: ； wherein , represent 3D, 2D coordinate values of target frame, source frame respectively, is a corresponding reliability weight, represents 3D-2D projection of pinhole model: ; Step 6: after obtaining the predicted depth map and camera motion pose, reconstruct the target frame using the source frame with the help of the camera intrinsic parameters: ； wherein, , represent the pixel coordinate values of the corresponding positions of the source frame and the target frame, T represents the predicted camera motion, represent the predicted depth map, represent the camera intrinsic parameters; then the network training is constrained by minimizing the photometric error between the original image and the reconstructed image.

Images