A monocular camera pose estimation method based on semantic key point detection

Abstract

A monocular camera pose estimation method based on semantic key point detection, which utilizes the mapping ability of deep learning to learn scene prior in a view offline in a known scene, and then carries out online camera pose estimation on a video sequence of the same scene but different views, comprises two subsystems: a semantic key point matching framework and a SLAM framework fusing matched semantic points. On the basis of the ORB-SLAM system, the semantic key point matching framework is used to obtain two-dimensional to three-dimensional semantic matching point pairs, the pose initialization process and the bundle adjustment method process are improved, so that the SLAM framework fusing matched semantic points can utilize scene prior to estimate the pose. The application can utilize scene prior, significantly improve the accuracy of pose estimation, has good robustness, and also avoids the initialization difficulty problem and the scale uncertainty problem of the monocular SLAM system.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology and relates to monocular camera pose estimation technology. Specifically, it is a method for estimating the camera's pose based on the predicted two-dimensional to three-dimensional semantic matching point pairs on an RGB image in a known fixed scene. Background Technology

[0002] This invention addresses the problem of real-time estimation of the camera's pose using a monocular RGB camera in a fixed scene. This problem has many practical applications, such as robotic vacuum cleaners, automatic lawnmowers, and automated parking systems.

[0003] For camera pose estimation, the traditional Simultaneous Localization and Mapping (SLAM) method utilizes two-dimensional matching points on adjacent frames, employs epipolar constraints to obtain the initial camera pose, and then uses triangulation to obtain the initial map points. This approach has the following problems: 1) it cannot utilize prior knowledge of the scene; 2) due to scale uncertainty, monocular SLAM cannot obtain the scale of the real world; 3) initialization is relatively difficult, requiring a certain amount of camera translation.

[0004] Furthermore, existing deep learning methods for solving camera pose estimation fall into two categories: The first is SFM based on 2D descriptors, such as HF-Net. This method uses 3D descriptors derived from the back projection of 2D descriptors, relying solely on semantic features from the image. It lacks targeted training in the metric space to ensure these 3D descriptors approximate matching 2D descriptors while distancing themselves from mismatched ones. The second category regresses pixel-level scene coordinates from RGB images, such as DSAC++ and HSC-Net. The mapping relationship in this method is not very direct and is easily affected by scene changes.

[0005] Introducing deep learning methods to leverage scene priors into traditional SLAM approaches has been explored in several studies. For example, some research utilizes deep learning for 3D object detection and 2D semantic segmentation to add constraints to SLAM systems, such as CubeSLAM and DeepSLAM++. However, these methods rely on a sufficient number of objects in the scene to serve as landmarks, placing high demands on annotation and making them unsuitable for scenes with few objects. Summary of the Invention

[0006] The problem this invention aims to solve is how to leverage scene priors to improve the camera pose estimation accuracy and robustness of monocular SLAM systems. Traditional monocular SLAM systems do not utilize prior knowledge of the known scene, relying solely on hand-written descriptors to detect points with significant image gradient changes as keypoints. Therefore, this invention aims to utilize the mapping capabilities of deep learning to learn scene priors and integrate them into traditional SLAM systems. This improves the overall system's localization accuracy and robustness while avoiding the scale uncertainty and initialization difficulties associated with solving 2D-to-2D matching problems.

[0007] The technical solution of this invention is as follows: a monocular camera pose estimation method based on semantic keypoint detection. In a known scene, the method utilizes the mapping capability of deep learning to learn scene priors from a single viewpoint offline. Then, online camera pose estimation is performed on video sequences of the same scene but with different viewpoints. The method includes two subsystems: a semantic keypoint matching framework and a SLAM framework that fuses matching semantic points. The semantic keypoint matching framework includes a 3D descriptor network and a 2D descriptor network. Based on the known scene, a 3D semantic keypoint neighbor cloud is obtained. A 3D semantic keypoint database is established offline using the 3D descriptor network. During online estimation, 2D semantic keypoints are detected in the input query image using the 2D descriptor network, and a matching search is performed in the 3D semantic keypoint database to obtain 2D-to-3D matching semantic point pairs. The SLAM framework that fuses matching semantic points uses the 2D-to-3D matching semantic point pairs and the query image to achieve camera pose estimation.

[0008] This invention comprises two subsystems: a semantic keypoint matching framework and a SLAM framework that fuses matching semantic points. The semantic keypoint matching framework establishes an offline database containing all semantic 3D keypoints to be matched. Then, it detects 2D semantic keypoints in the query image online and performs matching searches in the offline database to obtain 2D-to-3D matching semantic point pairs. The SLAM framework that fuses matching semantic points improves the original SLAM system using the 2D-to-3D matching semantic point pairs obtained from the semantic keypoint matching framework. This allows it to use not only geometric matching points but also semantic matching point pairs in camera pose optimization, thereby improving the final prediction accuracy and robustness.

[0009] This invention's semantic keypoint matching framework includes the design of a 2D descriptor network, a 3D descriptor network, and a cross-modal tuple loss function proposed to achieve cross-modal attitude learning. The 2D descriptor network employs a shared backbone network to ensure inference speed, while the 3D descriptor network utilizes keypoint sampling to improve description accuracy. The cross-modal tuple loss function fully utilizes training data and ensures training stability. The SLAM framework, which integrates matched semantic points, uses matched semantic point pairs for EPnP to obtain the camera pose during the camera pose initialization phase. Then, it uses 2D-to-2D matching established by hand-crafted descriptors to verify the camera pose, thus avoiding the scale uncertainty and initialization difficulties caused by solving epipolar geometry. Simultaneously, the framework uses matched semantic point pairs in the tracking and mapping phase to participate in graph optimization, acting as a constraint to optimize the camera pose to a more accurate position. To avoid errors in camera pose estimation caused by incorrect matched semantic points, this invention designs a confidence calculation method to change the weight of each matched semantic point pair in the optimization problem, thereby improving the system's robustness.

[0010] The present invention has the following advantages compared with the prior art.

[0011] This invention proposes a monocular camera pose estimation method based on semantic keypoint detection, comprising a semantic keypoint matching framework and a SLAM framework that fuses matched semantic points. The semantic keypoint matching framework leverages the advantages of deep learning to obtain semantic priors: a unified backbone network is used in the design of the 2D descriptor network, and a simple prediction method is adopted to ensure good real-time performance of network inference. Targeted improvements are made in the design of the 3D descriptor network, using keypoint sampling to enhance descriptive performance. Simultaneously, in terms of metric learning, a cross-modal tuple loss function is used to solve the 2D / 3D cross-modal matching problem, ensuring training stability and resulting in a highly discriminative metric.

[0012] The semantic keypoint matching framework utilizes predicted 2D-to-3D semantic matching points to avoid the scale uncertainty and initialization difficulties introduced by solving the epipolar geometry problem of 2D-to-2D matching in camera pose initialization. In the bundle adjustment method for tracking mapping, the predicted 2D-to-3D semantic matching points are also fully utilized as constraints. Furthermore, to mitigate the negative impact of erroneous semantic matching points, this invention designs the confidence level of semantic matching point pairs based on the intermediate results of the semantic keypoint matching framework to adjust the weight of semantic matching points in the graph optimization problem, thereby improving robustness to erroneous predictions.

[0013] This invention achieves excellent results in camera pose estimation, improving its accuracy and precision. Compared to existing methods, this invention achieves state-of-the-art results on both the 7Scenes and Scannet datasets, obtaining relatively accurate camera motion trajectories while maintaining good robustness. Attached Figure Description

[0014] Figure 1 This is an overall framework diagram of the present invention.

[0015] Figure 2 This is an architecture diagram of the two-dimensional descriptor network of the semantic key point matching framework proposed in this invention.

[0016] Figure 3 This is a diagram illustrating the cross-modal tuple loss function of the semantic keypoint matching framework proposed in this invention.

[0017] Figure 4 This is a flowchart of the pose initialization process of the SLAM framework for fusing and matching semantic points proposed in this invention.

[0018] Figure 5 This is a factor graph example of graph optimization for the SLAM framework that integrates and matches semantic points proposed in this invention. Detailed Implementation

[0019] This invention proposes a monocular camera pose estimation method based on semantic keypoint detection. Tests on 7Scenes and Scannet demonstrate high pose estimation accuracy and robustness. This invention is implemented using Python 3.6 and C++11, with deep neural network training and inference using the PyTorch 1.10 framework. The SLAM component is based on a modified ORB-SLAM system.

[0020] Figure 1 This is an overall framework diagram of the present invention, including a semantic keypoint matching framework and a SLAM framework for fusing and matching semantic points. The semantic keypoint matching framework takes an RGB image as input and outputs 2D-to-3D semantic matching point pairs; the SLAM framework for fusing and matching semantic points takes an RGB image and 2D-to-3D semantic matching point pairs as input and outputs the camera pose of that frame. Specifically:

[0021] 1) The semantic keypoint matching framework includes the following components: generation of the keypoint dataset, two-dimensional descriptor network, three-dimensional descriptor network, and cross-modal tuple loss function.

[0022] 1.1) Generation of the matching keypoint dataset for training. This invention trains a semantic keypoint matching framework using training data and further utilizes a depth filter to filter and generate training data, obtaining matching points for training. The catenary data includes the positions of two-dimensional keypoints in the image and the positions of the matched three-dimensional keypoints.

[0023] The KinectFusion 3D reconstruction method was used to reconstruct the RGBD sequence obtained from multiple scans of a known scene, resulting in 3D patches of the scene. The scene images used in the 3D reconstruction were used as the training set, and the i-th 3D point in the 3D patch was denoted as the scene point. Scene points are points within all 3D patches. 3D semantic keypoints (p) are determined using sampling or manual annotation within the 3D patches. i Then, by projecting the 3D semantic keypoints onto the images in the training set, the corresponding 2D semantic keypoints are obtained. 2D semantic keypoints projected onto the image boundaries are not retained. A depth-based filter is used to filter out occluded 3D semantic keypoints and 3D semantic keypoints projected too close to the image. Let p be the 3D semantic keypoint. i The projection onto the image in the Lth frame of the training set is That is, the corresponding p i Two-dimensional key points, scene points The projection onto the image in the Lth frame of the training set is Let ∈ occluded and ∈ crowded For setting hyperparameters, depth filter Set the points to be filtered out to False, specifically as follows:

[0024] 1. According to depth 2D key points in ascending order Sort the data and then iterate through the sorted 2D keypoints. Let the index of the traversal be k, initially k=1, and for any k, All are preset to True.

[0025] 2. For the points visited like If the result is False, proceed to step 5.

[0026] 3. For the points visited If there exists a projection of scene point n satisfy and The distance between two points is less than the hyperparameter and the projection The depth is less than the projection That is, the corresponding 3D key points are occluded by scene points, and settings are adjusted accordingly. Proceed to step 5.

[0027] 4. For the points visited If a key point k exists ′ (k ′ Projection of >k) satisfy and but and Too close and If the depth is greater, select and keep points with smaller depths, discard points with larger depths, and set... Proceed to step 5.

[0028] 5. k = k + 1, repeat the above judgment for the next key point, and update. If the iteration reaches its end, the process will exit.

[0029] After the depth filter removes points with a value of False, the corresponding matching point pairs are obtained from the 2D keypoints with a value of True. These matching point pairs are used as training data to train the semantic keypoint matching framework.

[0030] 1.2) Two-dimensional descriptive subnetwork. For example... Figure 2 As shown, during training, the two-dimensional descriptor network trains the Lth frame training image I. L ∈R H×W×3 To detect two-dimensional descriptors, features are extracted from the input image using a ResNet101 backbone network, followed by a deconvolution layer to obtain a shared deep feature set. Subsequently, corresponding detection heatmaps were obtained through three different head networks. Local offset map and descriptor graph For detecting heatmaps, a sigmoid activation function is needed to ensure the predicted values ​​are within the range [0,1]. For descriptor maps, feature normalization is also required to ensure the descriptor's modulus is 1. A design using a shared encoder for multiple predictions is employed here, which reduces memory consumption and speeds up inference. The actual heatmap Y... L The generation method is as follows: for each two-dimensional key point Calculate the coordinate position after scaling down by 1 / 4 scale Then use Gaussian kernel This generates a heatmap at each location. If the Gaussian kernels corresponding to two keypoints overlap, the larger kernel is used at the overlapping location. Supervision is performed using Focal Loss.

[0031]

[0032] To compensate for the loss of decimal parts in the keypoint coordinates generated by rounding down, the local offset map is monitored by the offset between the center of the Gaussian kernel and the actual keypoint positions. An L1 loss function is used for monitoring.

[0033]

[0034]

[0035] During online estimation, the two-dimensional descriptor network detects two-dimensional descriptors from the input l-th frame query image.

[0036] 1.3) 3D Descriptive Subnetwork. PointNet++ with a keypoint sampling strategy is used to process the Q-nearest point cloud of semantic keypoints. i Encode the corresponding descriptor. For the 3D semantic keypoint p... i Random sampling was performed within a 1m radius, retaining 1024 points, resulting in the point cloud Q. i To describe the 3D semantic keypoints, PointNet++ with keypoint sampling is used as the backbone network to predict the descriptors of the 3D semantic keypoints. The original PointNet++ network uses the farthest point sampling strategy, and the starting point is randomly selected. This invention adopts the key point sampling strategy, that is, the three-dimensional key point to be described is selected as the starting point, which can effectively improve the accuracy of description.

[0037] 1.4) Cross-modal multivariate loss function. A schematic diagram is shown below. Figure 3 The training samples in this invention are images from pairwise training sets and the co-viewed 3D keypoints between these two frames. Since training all co-viewed 3D keypoints simultaneously requires excessive memory, random sampling is used to retain |Ω| = 8 points. These 8 co-viewed 3D keypoints and their corresponding projections on the two frames have different positive and negative sample relationships, which are used as different loss functions in this invention. For ease of explanation, the following notation is provided: Descriptor distance dist(·,·) – This invention uses cosine similarity; Descriptor graph obtained from the 2D descriptor network. In the middle, based on the actual two-dimensional key point positions The descriptor of the two-dimensional keypoints obtained by bilinear interpolation is denoted as 3D descriptor Derived from a 3D descriptor network. The subscript 'i' represents the index of the descriptor; two-dimensional descriptors with the same index... and 3D descriptors This is a correct matching relationship. L and L′ are two frames used in the training samples. θ is the interval used for negative sample pairs in a triplet-like loss function. If the distance between negative samples exceeds this interval, supervision will no longer be performed. In the experiment, θ = 0.1 was used. The following explains these loss functions:

[0038] Two-dimensional to two-dimensional positive sample loss function

[0039]

[0040] Two-dimensional to two-dimensional negative sample loss function on the same frame Negative sample point j is obtained by sampling the most difficult sample, that is, the point with the smallest angle formed by the descriptor of anchor point i in the same frame is taken as the most difficult negative sample point to distinguish.

[0041]

[0042] Two-dimensional to two-dimensional negative sample loss function on different frames The negative sample point j is obtained by sampling the most difficult sample, that is, selecting a point j in the L′ frame such that the angle between the point j and the descriptor of point i in the l frame is minimized.

[0043]

[0044] The positive sample loss function from 2D to 3D. To make full use of the training data, the corresponding 2D keypoints on the selected two frames L and L′ are used from the 3D keypoints.

[0045]

[0046] The negative sample loss function for 2D to 3D conversion. Hardest sample sampling is performed on frames l and l′, where points j and k are descriptors of the corresponding 3D keypoints in the same frame. The point with the smallest included angle.

[0047]

[0048] The 3D to 3D negative sample loss function also uses the most difficult sample sampling method, that is, for a 3D keypoint i, the 3D keypoint j that is the most difficult to distinguish (with the smallest angle between descriptors) is selected as the negative sample point.

[0049]

[0050] Combining all the above loss functions together forms the cross-modal tuple loss function L. cd-ntuple ,Right now

[0051]

[0052] The training loss can be obtained by weighting all the above loss functions:

[0053] L total =w offset L offset +w heatmap L heatmap +w cd-ntuple L cd-ntuple

[0054] Where w offset =2, w heatmap =2, w cd-ntuple =1. During the training phase, the batch size was set to 8, and the Adam method was used for optimization. The initial learning rate was set to 0.001. A total of 700 training epochs were performed, and the learning rate was reduced to 1 / 10 of its original value at the 500th epoch. During the inference phase, all semantic 3D keypoints were first described offline using a 3D descriptor network to build a kd-tree. Then, online inference was performed on the l-th frame of the query image using a 2D descriptor network. After applying Non-Maximum Suppression (NMS) to the detection heatmap, a rough detection location of the 2D keypoints was obtained. The final detection position is obtained by adding a local offset to the detection position: The two-dimensional descriptor corresponding to the detection can be obtained by performing bilinear interpolation on the predicted descriptor map using the detection location. By finding the closest 3D keypoint in the offline kd-tree, we can obtain the 3D keypoint that matches the 2D keypoint.

[0055] 2) The main modifications to the SLAM framework for fusing and matching semantic points are in the camera pose initialization stage and the tracking and mapping stage:

[0056] 2.1) Camera pose initialization phase. The process is as follows: Figure 4 As shown, for the input l-th frame query image I l Using a trained semantic keypoint matching framework, inference is performed to obtain predicted two-dimensional and three-dimensional matching point pairs. Then, using these matching point pairs, solve for the EPnP with RANSAC to obtain the initial camera pose. This step requires additional verification of the camera pose T obtained from the EPnP. l That is, to calculate the number of 3D keypoints that satisfy the following conditions: (The question is incomplete and requires further context to be accurately translated.) lIf the pixel distance (i.e., reprojection error) between the 3D keypoint projected onto the image and the matched 2D keypoint is less than a threshold of 2, and the number of 3D keypoints meeting the condition exceeds half of the total, then the EPnP prediction is correct. If the camera pose prediction in frame l is correct, then manual descriptor ORB detection is performed on frame l. Then, the process moves to the next frame search. When the SLAM system obtains frame l+1, the semantic keypoint matching framework is used to obtain 2D to 3D semantic matching point pairs, just like in frame l. Then, EPnP with RANSAC is used to solve for the camera pose, with the same requirements as in frame l. If the camera pose in frame l+1 meets the requirements, then the ORB descriptor is detected and matched with the ORB descriptor in frame l. Then, the matched ORB descriptor is used to perform relative motion T. l+1→l Verification: First, triangulation is performed using the matched ORB descriptor and relative motion to obtain the corresponding 3D point. The cosine of the disparity angle observed from the 3D point in both frames is checked to see if it is less than 0.99998, the depth of the 3D point in both frames is positive, and the projection error from the 3D point to both frames is less than 4. If the triangulated 3D point passes these checks, it is considered a correct triangulated point. Finally, it is determined whether the correct triangulated point exceeds 90% of the correct matching points and whether the maximum value of the disparity angle of the triangulated point exceeds a threshold. If all the above checks pass, the camera pose initialization is successful.

[0057] 2.2) Tracking and Mapping Stage. The main idea is to apply the 2D to 3D semantic matching points predicted by the semantic keypoint matching framework to bundle adjustment. Figure 5 A simple example of a factor graph is given. Let the camera pose be K, and ρ(·) represent the operation of transforming homogeneous coordinates into non-homogeneous coordinates. Then, the observation error introduced by the semantic 3D point observed from the query image in the l-th frame is... for:

[0058]

[0059] Among them, T l Let l be the camera pose of the l-th frame. and Two-dimensional and three-dimensional matching point pairs predicted by the semantic keypoint matching framework;

[0060] The observations generated by the ORB descriptor in the existing ORB-SLAM system are referred to here as geometric matching points, and the observation errors generated by the geometric matching points... It can be represented as:

[0061]

[0062] in, and In the ORB-SLAM system, a pair of two-dimensional and three-dimensional matching points is determined by geometric matching points.

[0063] In graph optimization, a cost function of the following form is used to uniformly optimize all observations formed by matching points, including geometric matching points and semantic matching points:

[0064]

[0065] Ω il Ω jl This is called the information matrix, and it's used to express the confidence level for this observation. Ω jl This represents the confidence score of the geometric matching points, calculated using the original ORB-SLAM system method, which will not be elaborated upon here. The confidence score Ω for two-dimensional and three-dimensional semantic matching points is... il In this invention, some intermediate results from a semantic keypoint matching framework are used in the design: Ω il =λ k λ rel λ il E. Where E is a 2×2 identity matrix, λ k λ is a constant, and in the experiment... k =10. λ rel λ is a balancing factor used to balance the observation errors of geometric keypoints and semantic keypoints. It is calculated by dividing the number of observations for each geometric keypoint by the number of observations for each semantic keypoint. This balancing factor ensures that the optimization objective corresponding to a semantic keypoint still occupies a sufficient proportion of the optimization problem, even when the number of semantic keypoints is small. il ∈[0,1] is a confidence coefficient that is different for each observation, and its definition is as follows:

[0066] λ il =c det c desc c nn

[0067] c det ∈[0,1] represents the confidence level of the detection, derived from the predicted heatmap. Let the pixel position retained after non-maximum suppression (NMS) be . Then c det The calculation method is as follows:

[0068]

[0069] c desc ∈[0,1] represents the confidence level of the description, which comes from the matching stage. Let... This represents the descriptor of the i-th two-dimensional key point. In metric space and The closest 3D keypoint descriptor, c desc It's the cosine distance between the two, the same distance function used in the training phase. Here, it's used as the confidence score, consistent with the training objective.

[0070]

[0071] c nn ∈[0,1] represents the distinguishability of the descriptor. Let be... In metric space and The descriptor of the second closest 3D keypoint, θ nn This describes the percentage increase in distance between the closest descriptor and the second closest descriptor; a higher percentage indicates stronger distinguishability. nn The calculation method is as follows, where α is the intensity coefficient, β is the highest proportion, and θ nn After exceeding this ratio, c nn That is, the highest value is 1. In the experiment, α = 100 and β = 1.

[0072]

[0073]

[0074] The method of this invention was tested on the 7Scenes dataset, with a median spatial position error of 3.86 cm and a median rotation angle error of 0.79° for camera pose estimation. Tested on the Scannet dataset, the median spatial position error for camera pose estimation was 5.24 cm and the median rotation angle error was 1.05°. The localization results are significantly more accurate than existing SLAM systems. Furthermore, this invention exhibits excellent robustness; errors in semantic matching points have a relatively small impact on the proposed system. Testing with a GTX1080Ti and an i7-6850K CPU showed that the semantic keypoint matching framework had an average inference time of approximately 0.2 ms, while the SLAM framework fusing matched semantic points had an average tracking time of approximately 24 ms.

Claims

1. A monocular camera pose estimation method based on semantic keypoint detection, characterized in that The known scene utilizes the mapping ability of deep learning to learn the scene prior in one view offline, and then performs online camera pose estimation on video sequences of the same scene but different views, including two subsystems: a semantic key point matching framework and a SLAM framework fusing matching semantic points, the semantic key point matching framework including a three-dimensional descriptor network and a two-dimensional descriptor network, obtaining a three-dimensional semantic key point neighborhood point cloud based on a known scene, establishing a three-dimensional semantic key point database offline through the three-dimensional descriptor network, detecting two-dimensional semantic key points in an input query image through the two-dimensional descriptor network during online estimation, and performing matching search in the three-dimensional semantic key point database to obtain two-dimensional-to-three-dimensional matching semantic point pairs; the SLAM framework fusing matching semantic points realizes camera pose estimation using the two-dimensional-to-three-dimensional matching semantic point pairs and the query image, and the SLAM framework fusing matching semantic points is realized based on an ORB-SLAM system, and the camera pose estimation is as follows: 1) In the pose initialization stage, the two-dimensional-to-three-dimensional semantic matching point pairs predicted by the semantic key point matching framework are used for EPnP calculation of the camera pose, the camera pose calculated is verified by using a hand-crafted descriptor ORB, and the initialization of the camera pose is completed; 2) In the tracking mapping stage, the two-dimensional-to-three-dimensional semantic matching point pairs predicted by the semantic key point matching framework are added to the bundle adjustment method, and the two-dimensional-to-three-dimensional geometric matching points obtained by the hand-crafted descriptor are used for graph optimization together, to obtain the optimized camera pose.

2. The monocular camera pose estimation method based on semantic keypoint detection according to claim 1, characterized in that The implementation of the semantic key point matching framework is as follows: firstly, all three-dimensional semantic key point neighborhood point clouds are described using a three-dimensional descriptor network offline, a k-d tree is established, and a two-dimensional descriptor network is used for online reasoning on a query image in the test stage: firstly, the first frame query image obtains a rough detection position of the second two-dimensional key point after non-maximum suppression NMS on its detection heat map, denoted as , plus a local offset to obtain the final detection position: , the detection position is used to obtain the two-dimensional descriptor corresponding to the detection on the descriptor map predicted by the two-dimensional descriptor network through bilinear interpolation , and the closest three-dimensional key point is found in the offline established k-d tree to obtain the matched three-dimensional key point of the two-dimensional key point.

3. The monocular camera pose estimation method based on semantic keypoint detection according to claim 1 or 2, characterized in that The semantic key point matching framework is trained by training data, and the training data is generated by a depth filter. Specifically, a three-dimensional reconstruction method KinectFusion is used to reconstruct an RGBD sequence obtained by multiple scans of a known scene, to obtain three-dimensional patches of the scene, and the scene images used in the three-dimensional reconstruction are taken as a training set, and a first three-dimensional point in the three-dimensional patches is recorded as a scene point A three-dimensional semantic key point is determined in the three-dimensional patches by using sampling or manual labeling Then, the three-dimensional semantic key point is projected onto the image of the training set to obtain a corresponding two-dimensional semantic key point; wherein the two-dimensional semantic key point projected out of the image boundary is not reserved, and a depth-based filter is used to filter out the three-dimensional semantic key point that is occluded and the three-dimensional semantic key point that is too close in projection distance, and the three-dimensional semantic key point The projection of the scene point on the image of the training set at the first frame is , , and the two-dimensional key point corresponding to the scene point is The projection of the scene point on the image of the training set at the first frame is , and and are set hyperparameters, and the depth filter sets the points to be filtered as False, and the specific implementation is as follows:​ 1) according to depth In order from small to large Sort the two-dimensional key points, traverse the sorted two-dimensional key points, and record the index of the traversal as , initially , and for any , are all preset to True; 2) for the point being traversed , if is False, go to step 5); 3) for the point being traversed , if there exists a projection of a scene point , satisfying and , then the three-dimensional key point is occluded by the scene point, set , and go to step 5); 4) for the point being traversed , if there exists a projection of the key point , , satisfying and , then and are too close and the depth of is greater, the point with smaller depth is selected to be retained and the point with greater depth is discarded, set , and go to step 5); 5) , the value of the next key point is determined , if the traversal is ended, exit; After filtering by the deep filter, points with value False are filtered out, and two-dimensional key points with value True are obtained, and the corresponding matching point pairs are recorded as These matching point pairs are used as training data to train the semantic key point matching framework.

4. The monocular camera pose estimation method based on semantic keypoint detection according to claim 3, characterized in that In the semantic key point matching framework, a cross-modal multi-group loss function is used for cross-modal metric learning of two-dimensional and three-dimensional descriptors, and training data used includes images of two training sets and three-dimensional key points co-observed by the two frames, and the cross-modal multi-group loss function includes the following loss functions: a two-dimensional-to-two-dimensional positive sample loss function, a two-dimensional-to-two-dimensional negative sample loss function on the same frame, a two-dimensional-to-two-dimensional negative sample loss function on different frames, a two-dimensional-to-three-dimensional positive sample loss function, a two-dimensional-to-three-dimensional negative sample loss function, and a three-dimensional-to-three-dimensional negative sample loss function.

5. The monocular camera pose estimation method based on semantic keypoint detection according to claim 1, characterized in that In the 3D descriptive subnetwork, 3D semantic keypoints Random sampling was performed within a 1m radius, retaining 1024 points to obtain the point cloud. To describe the 3D semantic keypoints, PointNet++ with keypoint sampling is used as the backbone network to predict the descriptors of the 3D semantic keypoints. .

6. The monocular camera pose estimation method based on semantic keypoint detection according to claim 1, characterized by two dimensions. The descriptor subnetwork is applied to the first frame query image The two-dimensional descriptor is detected, and the two-dimensional descriptor network uses a backbone network to extract image features, and different small networks are used thereon to obtain a detection heat map , a local offset map , and a descriptor map The two-dimensional descriptor is expressed in the form of a detection heat map, and the detection position offset caused by prediction on a 1 / 4 image scale is adjusted in the form of a local offset map.

7. The monocular camera pose estimation method based on semantic keypoint detection according to claim 1, characterized in that The semantic matching points are added to the graph optimization, and camera intrinsic parameters are denoted as , The operation represented by the equation is to convert the homogeneous coordinates into non-homogeneous coordinates, and the first The error corresponding to the observation introduced by the three-dimensional semantic key points observed by the frame query image matching is : wherein, is the camera pose for the 1st frame, is the two-dimensional three-dimensional matched point pairs predicted by the semantic keypoint matching framework; Let the observations generated by the ORB descriptors in the ORB-SLAM system be geometrically matched points, and the observation error generated by the geometrically matched points be is represented as: wherein and is a two-dimensional three-dimensional matching point pair determined by the geometric matching points under the ORB-SLAM system; In the graph optimization, all matching points formed by geometric matching points and semantic matching points are uniformly optimized by using a cost function in the following form: , is the information matrix, which is used to express the confidence for this observation, is the confidence of the geometric matching points, which is calculated by the ORB-SLAM system, and the confidence of the two-dimensional and three-dimensional semantic matching points , wherein, is the identity matrix of , is a constant, is a balance coefficient, which is used to balance the observation error of the geometric key points and the observation error of the semantic key points, and is calculated by the number of observations corresponding to the geometric key points divided by the number of observations of the semantic key points, is a confidence coefficient different for each observation, which is defined as follows: represents the confidence of detection, from the predicted heat map , let the pixel position reserved after non-maximum suppression NMS be , then The calculation method is: represents the confidence of the description, from the matching stage, note represents the description of the second two-dimensional keypoint, is the description of the closest three-dimensional keypoint in the metric space, is the cosine distance between the two, which is used as the confidence: represents the distinguishability of the descriptor, note is in the metric space and the descriptor of the second closest three-dimensional key point, represents how much the closest descriptor distance is improved compared to the second closest descriptor distance, the higher the ratio, the stronger the distinguishability, The calculation method is as follows, is the intensity coefficient, is the highest ratio, After exceeding this ratio, is the highest value 1: 。

Images