Camera repositioning method based on key point guidance

Abstract

The invention belongs to the field of camera relocation in the field of computer three-dimensional vision, and discloses a camera relocation method based on key point guidance. A key point guide scene coordinate generation method based on a matching rate is provided, a key point selection network and a training framework thereof are integrated, and a camera relocation process is optimized. 3D point quality in the point cloud is obtained by evaluating motion structure reconstruction, low-quality key points are filtered and re-projected to a 2D space, and a supervision mask optimization key point selection network is generated. And finally, evaluating the reliability and the accuracy of attitude estimation in combination with the interior point ratio. And when the confidence coefficient of the camera pose estimation is not greater than the threshold value, estimating the six-degree-of-freedom pose of the camera by using the scene coordinate data regressed by the scene coordinate regression network as the final camera pose estimation, thereby improving the precision and generalization ability of the pose estimation. Experimental results verify the effectiveness of the method, and the method is suitable for indoor and outdoor scenes.

Description

Technical Field

[0001] The present invention belongs to the field of camera relocalization in the field of computer three-dimensional vision, and in particular relates to a camera relocalization method based on key point guidance. Background Art

[0002] The camera positioning task is mainly to help the device understand its position information in the real scene, and determine the six-degree-of-freedom pose information of the shooting device in the scene based on the visual image information captured by the device, so as to subsequently perform positioning navigation or interaction with the environment. The development of artificial intelligence technology has enabled 3D vision-related tasks, such as scene modeling, depth estimation, object pose estimation, camera repositioning, etc., to achieve significant performance improvements and performance results.

[0003] The evolution of camera relocalization began with image retrieval, and its original form relied on robust image retrieval for localization. The introduction of random regression forests for regressing scene coordinates significantly improved the accuracy of camera relocalization. Subsequently, deep learning frameworks used deep networks to achieve direct regression from image to pose. Subsequently, new mainstream camera relocalization methods based on scene coordinate regression were developed, which designed a network specifically for associating input images with scene coordinates. By combining RANSAC and PnP, the camera pose can be estimated indirectly from selected 2D-3D keypoint correspondences. Although the above methods are able to estimate camera pose in some cases, they are still far from meeting the high accuracy requirements of real-world applications.

[0004] However, in practice, the complete scene coordinates are not required, and selecting partial scene coordinates may further improve the 2D-3D keypoint correspondences for pose estimation using RANSAC and PnP. The present invention proposes to generate keypoint-guided scene coordinates, which requires integrating the learning-based keypoint selection process into the overall camera relocalization process. This approach raises several questions, namely: What are the criteria for keypoint selection? Which keypoints should be selected? And how should the network be trained to recognize these keypoints? The above are the problems that need to be solved in the current field.

[0005] Based on the matching rate, this paper designs a new camera pose estimation framework to select points that are truly suitable for camera relocalization, and introduces a key point selection network and its corresponding training framework. This paper selects key points for camera relocalization and develops methods to evaluate the possibility of obtaining accurate camera poses, thereby providing reliable support for practical applications. This method can not only evaluate the reliability of pose estimation, but also improve accuracy. Summary of the invention

[0006] Based on the above technical problems, the present invention designs a new camera pose estimation framework based on matching rate to select points that are truly suitable for camera relocalization, and introduces a key point selection network and its corresponding training framework. The present invention selects key points for camera relocalization and develops a method to evaluate the possibility of obtaining accurate camera poses, thereby providing reliable support for practical applications. This method can not only evaluate the reliability of pose estimation, but also improve accuracy.

[0007] The technical solution of the present invention is as follows: A camera repositioning method based on key point guidance comprises the following steps:

[0008] Step 1. Select the camera relocalization dataset;

[0009] Step 2: Sparse reconstruction of the scene based on the sampled image;

[0010] Step 3: Select scene key points based on scene sparse reconstruction;

[0011] Step 4: Project the selected scene key points to obtain a scene key point mask of a single image;

[0012] Using the camera extrinsic and intrinsic parameters corresponding to the image, the scene key points are projected to obtain the scene key point mask corresponding to a single image;

[0013] Step 5: Train the scene coordinate regression network guided by non-keypoints;

[0014] Step 6: Convert the 3D scene coordinate reprojection error obtained by the non-keypoint guided scene coordinate regression network into a probabilistic keypoint mask;

[0015] Step 7: Obtain the final key point mask guided by the scene key point mask and the probability key point mask;

[0016] The final key point mask is obtained by taking the intersection of the scene key point mask obtained based on the scene point projection and the probability key point mask obtained based on the reprojection error;

[0017] Step 8: Use the final key point mask as supervision information to train the key point selection network;

[0018] Step 9: Using the final key point mask as a screening condition, train a key point guided scene coordinate regression network;

[0019] Step 10: Based on the mask estimated by the key point selection network, camera pose estimation and confidence assessment are performed;

[0020] Step 11. Select the final camera pose.

[0021] The step 1 specifically comprises: selecting a camera relocalization dataset, wherein the camera relocalization dataset comprises a sequence of scene images captured and corresponding six-degree-of-freedom postures of the camera obtained by a motion-to-structure algorithm; and dividing the images of the camera relocalization dataset into a training set and a test set.

[0022] The step 2 specifically comprises: sampling the image sequence of the camera relocalization data set to obtain sampled image data that is less than or equal to the number of the image sequence; performing feature extraction, feature matching and scene sparse reconstruction based on the sampled image data using a motion-to-structure algorithm; and aligning the reconstructed scene with the six-degree-of-freedom posture data given in the camera relocalization data set.

[0023] The step 3 is specifically as follows: extracting all the scene 3D points obtained after the motion-to-structure algorithm reconstruction step in step 2, as well as all the image key points and their corresponding feature descriptions associated with these scene 3D points during the calculation process, and calculating the matching effect using the following formula:

[0024]

[0025] Where φ(p) is the average value of the feature descriptions of all image key points associated with the scene 3D point p, desc(p i ) is the image key point p i Characteristic description of , σ is a hyperparameter used to adjust the degree of smoothness;

[0026] The overall matching rate of each key point is calculated using the following formula:

[0027]

[0028] Among them, w i is the weight of the i-th feature, and n is the number of features;

[0029] The following formula is used to select the scene key points:

[0030] K={p∈P|M(p)≥τ}

[0031] Among them, τ is the threshold, which is used to decide whether the point is selected as a key point; all points greater than the threshold will be selected as scene key points.

[0032] The step 5 is specifically as follows: inputting a single image data in the scene sequence image into the feature extraction network to obtain the feature description of each pixel of the image; randomly extracting the feature description combination and inputting it into the scene coordinate regression network for scene coordinate regression training, and using the following formula for training supervision:

[0033]

[0034] Among them, xi is the real two-dimensional projection coordinate, y i are the estimated three-dimensional coordinates, are assumed three-dimensional coordinates, is the actual camera pose, e π (·) is the reprojection error equation, It is a judgment condition used to determine whether the reprojection error is within the set range.

[0035] The step 6 is specifically as follows: inputting a single image data in the scene sequence image into the feature extraction network, obtaining the features of the pixel points, using the scene coordinate regression network to calculate the 3D scene coordinates corresponding to each feature, using the scene key point mask to obtain the scene coordinates guided by the key point, and then using the following formula to convert the reprojection error into the key point probability:

[0036]

[0037] Among them, ∈ i is the reprojection error of the point, and λ is used to control the error degree of the reprojection error. The threshold is set to filter the key points and obtain a probabilistic key point mask.

[0038] The step 8 is specifically as follows: inputting a single image data in the scene sequence image into the feature extraction network, obtaining the pixel feature and inputting it into the key point selection network, and using the final key point mask obtained in step 7 as supervision information to train the key point selection network; the training is supervised by using the following cross entropy loss:

[0039]

[0040] Where k represents the set of probabilities that the key point selection network predicts whether each pixel is a key point. represents the set of actual probabilities of each pixel, k i represents the probability of the prediction of the i-th pixel, represents the actual probability of the i-th pixel, and N is the total number of image pixels involved in the calculation that need to be used to distinguish whether they are key points.

[0041] The step 9 is specifically as follows: using the final key point mask obtained in step 7 as a screening condition, optimizing and training the scene coordinate regression network obtained in step 5, still using a single image data in the scene sequence image as input, but the supervision part only involves the part covered by the final key point mask; the supervision function used in the training is the same as that in step 5.

[0042] The step 10 is specifically as follows: during the test process, the mask estimated by the key point selection network in step 8 is used as a basis, and the scene coordinate information corresponding to a single image data in the scene sequence image is obtained by using the scene coordinate regression network guided by the key points after optimization training in step 9, and the scene coordinate information covered by the mask is selected to estimate the camera's six-degree-of-freedom pose using the PnP and RANSAC algorithms. In the calculation process, the key points used to calculate the camera's six-degree-of-freedom pose are obtained, and these key points are taken as internal points. The confidence of the camera pose estimation is calculated using the following formula:

[0043]

[0044] Among them, N i is the number of internal points, N a is the number of pixels of the final keypoint mask estimated by the network in step 8, r i is the confidence of the pose estimate.

[0045] The step 11 is specifically as follows: if the confidence of the camera pose estimation obtained in step 10 is greater than a threshold value of 90, the camera pose estimated in step 10 is output as the final camera pose estimation; otherwise, the scene coordinate data regressed by the scene coordinate regression network in step 5 is used to estimate the camera's six-degree-of-freedom pose using the PnP and RANSAC algorithms, and this is used as the final camera pose estimation.

[0046] Beneficial effects of the present invention:

[0047] (1) Selection of effective scene points and key points for camera pose estimation

[0048] In the present invention, the designed algorithm realizes the selection of key points that are more effective for the camera relocalization task. The present invention starts from two directions. One direction explores the selection of key points that are effective for camera relocalization in scene points, and realizes the selection by selecting points with higher matching rates. The present invention verifies the effectiveness both theoretically and experimentally. The other direction explores the selection of key points that are effective for camera relocalization in the scene coordinates estimated after network training, and the reprojection error is obtained by projecting the estimated scene coordinate points with the real camera pose, and then the error is mapped to the probability interval using the probability conversion formula to generate the mask. The final key point mask is obtained by taking the intersection of the obtained scene key point mask and the probability key point mask.

[0049] (2) More accurate camera pose estimation

[0050] The present invention extracts key points that are beneficial to camera pose estimation for camera pose estimation, which makes the network pay more attention to those areas that should be focused on, making the scene coordinate estimation more accurate, thereby improving the accuracy of camera pose estimation under sufficient key point observation angles. The present invention first selects key points that are beneficial to camera relocalization, and designs a key point selection scheme based on matching rate. Then, a key point evaluation method based on reprojection error is designed. Finally, a gated camera pose estimation strategy based on confidence threshold is proposed, combining the key point guided network with a more general network to further improve the accuracy of camera relocalization. Compared with the most advanced methods, the present invention does not significantly increase the training time or training amount, and achieves higher accuracy. Through extensive experimental comparisons, the effectiveness of the proposed method is proved, surpassing the most advanced results.

[0051] (3) Greater practical application significance

[0052] The design of the present invention evaluates the confidence of the camera pose estimation. However, most of the existing methods are currently unable to effectively evaluate the confidence of the camera pose. However, in real applications, estimating the confidence of the camera pose estimation has great practical significance for stable scene rendering, reliability assessment of positioning navigation, robot safety, and autonomous driving navigation safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is the overall process structure of the present invention.

[0054] Figure 2 The visualization results of trajectory differences of comparative experiments of embodiments of the present invention; (a) is the pumpkin sequence trajectory of the comparative method; (b) is the staircase sequence trajectory of the comparative method; (c) is the pumpkin sequence trajectory of the method of the present invention; (d) is the staircase sequence trajectory of the method of the present invention. DETAILED DESCRIPTION

[0055] Specifically, the present invention explores the matching rate to evaluate each scene point in the point cloud obtained by motion structure reconstruction, and preliminarily filters out low-quality scene key points. Through the selected scene key points, the present invention can reproject them into the image space as a supervision mask, and optimize the image key point selection network guided by the real camera pose. Such an image supervision mask is further refined by the reprojection error through the learned scene coordinate network. After predicting the image key point mask, the present invention inputs it into the scene coordinate regression network together with the features extracted from the feature backbone network to generate partial scene coordinates, which can predict the camera pose through RANSAC and PnP. Based on the predicted camera pose, the present invention introduces the inlier ratio to evaluate the confidence in the estimated pose.

[0056] The proposed pose estimation framework is dual-branched and has a well-designed gating mechanism based on the inlier ratio. In particular, when the inlier ratio confidence of the estimated pose from the keypoint-guided branch is below a preset threshold, the present invention switches to the general full scene coordinate branch to avoid the final camera pose being heavily biased towards low-quality keypoints. Experiments conducted on indoor and outdoor scenes clearly verify the accuracy and generalization ability of the dual-branch framework proposed in the present invention.

[0057] A camera relocalization method based on key point guidance comprises the following steps:

[0058] Step 1. Select the camera relocalization dataset

[0059] Select a Camera Re-localization Datasets (CRD), which includes the captured scene sequence images and the corresponding six-degree-of-freedom (6DOF) postures of the camera obtained by the algorithm; and the dataset covers most of the viewpoints in the scene, and the observations between the viewpoints have a lot of overlap; divide the images in the obtained CRD dataset into a training set and a test set;

[0060] Step 2: Sparse reconstruction of the scene based on the sampled image

[0061] The image sequence of the camera relocalization dataset is sampled to obtain sampled image data that is less than or equal to the number of image sequences; based on the sampled image data, feature extraction, feature matching and scene sparse reconstruction are performed using open source tools based on the Structure from Motion (SfM) algorithm; the reconstructed scene is aligned with the pose data given in the dataset.

[0062] Step 3: Select scene key points based on scene sparse reconstruction

[0063] All reconstructed key points, as well as all image key points associated with these key points and their corresponding features are extracted. The present invention uses the following formula to calculate the matching effect:

[0064]

[0065] Among them, φ(p) is the average value of all associated key point features, desc(p i ) is p i The feature description of the point, σ is a hyperparameter used to adjust the degree of smoothness. The present invention uses the following formula to calculate the overall matching rate of each point:

[0066]

[0067] Among them, w i is the weight of the ith feature, and the default value is 1. The present invention adopts the following formula to select scene points:

[0068] K={p∈P|M(p)≥τ},

[0069] Among them, τ is the threshold, which is used to decide whether the point is selected as a key point. All points greater than the threshold will be selected as scene key points.

[0070] Step 4: Project the selected scene points to obtain the scene key point mask of a single image

[0071] After obtaining the scene key points selected in the scene in step 3, the scene key points are projected using the camera extrinsic parameters and intrinsic parameters corresponding to the image to obtain the scene key point mask corresponding to the single image.

[0072] Step 5: Train the non-keypoint guided scene coordinate regression network

[0073] Input the image data into the feature extraction network to obtain the feature description of each pixel in the image. Use the open source ACE feature extraction network to extract pixel-by-pixel features. Store the feature description of each pixel, randomly extract the feature description combination and input it into the scene coordinate regression network for scene coordinate regression training. Use the following formula for training supervision:

[0074]

[0075] Among them, x i is the real two-dimensional projection coordinate, y i are the estimated three-dimensional coordinates, are assumed three-dimensional coordinates, is the actual camera pose, e π (·) is the reprojection error equation, This step is independent of the previous step and can be adjusted to any of the previous steps.

[0076] Step 6: Convert reprojection error into probabilistic keypoint mask

[0077] The image data is input into the feature extraction network. After obtaining the features of the pixel points, the scene coordinate regression network is used to calculate the 3D scene coordinates corresponding to each feature, and the reprojection error is converted into the key point probability using the following formula:

[0078]

[0079] Among them, ∈ i is the reprojection error of the point, and λ is used to control the error degree of the reprojection error. This method sets a threshold to filter key points. So far, this method obtains a probabilistic key point mask.

[0080] Step 7: Obtain the final key point mask guided by scene key points and key point probabilities

[0081] At present, this method has obtained a scene key point mask based on scene point projection and a probabilistic key point mask based on reprojection error. The intersection of these two masks is used to obtain the final key point mask.

[0082] Step 8: Train the key point selection network

[0083] The image is input into the feature extraction network and further into the keypoint selection network. The keypoint selection network is trained using the final keypoint mask obtained in step 7 as supervision information. The training is supervised using the following cross entropy loss:

[0084]

[0085] in They represent the probability predicted by the network and the actual probability respectively.

[0086] Step 9: Train the key point guided scene coordinate regression network

[0087] Using the final keypoint mask obtained in step 7 as the screening condition, the scene coordinate regression network obtained in step 5 is optimized and trained. The image is still used as input, but the supervision part only involves the part covered by the final keypoint mask. The supervision function used in training is the same as that in step 5.

[0088] Step 10: Camera pose estimation and confidence assessment

[0089] During the test, the mask estimated by the network in step 8 is used as the basis, and the scene coordinate information covered by the mask is selected to estimate the camera's six-degree-of-freedom pose using the PnP and RANSAC algorithms. During the calculation process, key points for calculating the camera's six-degree-of-freedom pose are obtained. These key points are taken as internal points, and the confidence of the camera pose estimate is calculated using the following formula:

[0090]

[0091] Among them, N i is the number of internal points, N a is the number of pixels of the mask estimated by the network in step 8, r i is the confidence of the pose estimate.

[0092] Step 11. Select the final camera pose

[0093] Determine the confidence of the camera pose estimation obtained in step 10. If it is greater than a threshold of 90, output the camera pose estimated in step 10 as the final camera pose estimation. Otherwise, use the scene coordinate data regressed by the scene coordinate regression network in step 5, and use the PnP and RANSAC algorithms to estimate the camera's six-degree-of-freedom pose as the final camera pose estimation.

[0094] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0095] This embodiment uses the public 7Scenes dataset, which contains images taken in 7 indoor scenes, depth maps, scene models obtained by Kinect Fusion technology, and corresponding camera poses. This embodiment uses improved camera pose estimation, uses COLMAP technology to estimate the camera pose, and aligns it with the camera pose obtained by Kinect Fusion. Each scene contains multiple shooting sequences, and the sequences used for training and testing have been specified.

[0096] The present invention is implemented on PyTorch, based on the public code of ACE. To train the keypoint selection network, the present invention uses a simple convolutional neural network (CNN). The initial learning rate is set to 0.0001, and the AdamW optimizer is always used in all network configurations. When initializing a typical scene coordinate regression network, a training buffer containing 8.8 million samples is set, and all features are randomly sampled from random images and enhancements. For the training of the keypoint selection network, a training buffer containing 8 million samples is established, and features are extracted from randomly selected images. It should be noted that when training the scene coordinate regression network for image supervision mask, all outputs of the initial keypoint mask are obtained by projecting the matching rate from the selected 3D keypoints to the real camera pose. When training the keypoint-guided scene coordinate regression network, the initialized network head is fine-tuned to retain the network's ability to perform scene coordinate regression on non-keypoint features. The present invention follows the implementation of ACE on the pose estimator, but some modifications are made to it to adapt to the method in the present invention. Using the keypoint selection network as a guide, we select those points with an estimated probability value greater than 0.8, corresponding to a reprojection error of approximately 5.0.

[0097] Figure 1is the overall flow structure of the present invention. The pose estimation framework proposed in the present invention consists of two different branches and adopts a carefully designed gating mechanism to select the final pose. The upper general branch is responsible for estimating the camera pose of images with higher generality, but its accuracy cannot be guaranteed. The lower keypoint guided branch is equipped with image confidence estimation, aiming to more accurately estimate the camera pose of images with higher confidence levels. The present invention explores the matching rate to select high-quality 3D keypoints and uses the real camera pose as supervision to reproject it into 2D space to optimize the keypoint selection network. All query images are feature extracted through the feature backbone network, and the extracted features are initially input to the keypoint selection network and the keypoint branch of the upper path. In the keypoint branch, only keypoints are used for pose estimation to generate a confidence indicator, specifically the inlier ratio. Finally, in the pose selection stage, the gating mechanism based on the confidence value decides whether to (1) execute the general branch and adopt its output pose, or (2) directly use the pose obtained from the keypoint guided branch as the final pose output.

[0098] Figure 2 The display result of the comparative test shows the difference between the pose trajectory estimated by the method proposed by the present invention and ACE and the real pose trajectory. One line is the real translation trajectory, and the other is the estimated translation trajectory. The line between the two trajectories represents the difference between the corresponding translations. The larger the line distance, the greater the difference. It can be seen from the figure that the method of the present invention has a smaller line distance and better positioning accuracy.

Claims

1. A camera relocalization method based on key point guidance, characterized in that: The following steps are involved: Step 1. Select the camera relocalization dataset; Step 2: Sparse reconstruction of the scene based on the sampled image; Step 3: Select scene key points based on scene sparse reconstruction; Step 4: Project the selected scene key points to obtain a scene key point mask of a single image; Using the camera extrinsic and intrinsic parameters corresponding to the image, the scene key points are projected to obtain the scene key point mask corresponding to a single image; Step 5: Train the scene coordinate regression network guided by non-keypoints; Step 6: Convert the 3D scene coordinate reprojection error obtained by the non-keypoint guided scene coordinate regression network into a probabilistic keypoint mask; Step 7: Obtain the final key point mask guided by the scene key point mask and the probability key point mask; The final key point mask is obtained by taking the intersection of the scene key point mask obtained based on the scene point projection and the probability key point mask obtained based on the reprojection error; Step 8: Use the final key point mask as supervision information to train the key point selection network; Step 9: Using the final key point mask as a screening condition, train a key point guided scene coordinate regression network; Step 10: Based on the mask estimated by the key point selection network, camera pose estimation and confidence assessment are performed; Step 11. Select the final camera pose.

2. The camera relocalization method based on key point guidance according to claim 1, characterized in that: The step 1 specifically comprises: selecting a camera relocalization dataset, wherein the camera relocalization dataset comprises a sequence of scene images captured and corresponding six-degree-of-freedom postures of the camera obtained by a motion-to-structure algorithm; and dividing the images of the camera relocalization dataset into a training set and a test set.

3. The camera relocalization method based on key point guidance according to claim 2, characterized in that: The step 2 specifically comprises: sampling the image sequence of the camera relocalization data set to obtain sampled image data that is less than or equal to the number of the image sequence; performing feature extraction, feature matching and scene sparse reconstruction based on the sampled image data using a motion-to-structure algorithm; and aligning the reconstructed scene with the six-degree-of-freedom posture data given in the camera relocalization data set.

4. The camera relocalization method based on key point guidance according to claim 3, characterized in that: The step 3 is specifically as follows: extracting all the scene 3D points obtained after the motion-to-structure algorithm reconstruction step in step 2, as well as all the image key points and their corresponding feature descriptions associated with these scene 3D points during the calculation process, and calculating the matching effect using the following formula: Where φ(p) is the average value of the feature descriptions of all image key points associated with the scene 3D point p, desc(p i ) is the image key point p i Characteristic description of , σ is a hyperparameter used to adjust the degree of smoothness; The overall matching rate of each key point is calculated using the following formula: Among them, w i is the weight of the i-th feature, and n is the number of features; The following formula is used to select the scene key points: K={p∈P|M(p)≥τ} Among them, τ is the threshold, which is used to decide whether the point is selected as a key point; all points greater than the threshold will be selected as scene key points.

5. The camera relocalization method based on key point guidance according to claim 4, characterized in that: The step 5 is specifically as follows: inputting a single image data in the scene sequence image into the feature extraction network to obtain the feature description of each pixel of the image; randomly extracting the feature description combination and inputting it into the scene coordinate regression network for scene coordinate regression training, and using the following formula for training supervision: Among them, x i is the real two-dimensional projection coordinate, y i are the estimated three-dimensional coordinates, are assumed three-dimensional coordinates, is the actual camera pose, e π (·) is the reprojection error equation, It is a judgment condition used to determine whether the reprojection error is within the set range.

6. The camera relocalization method based on key point guidance according to claim 5, characterized in that: The step 6 is specifically as follows: inputting a single image data in the scene sequence image into the feature extraction network, obtaining the features of the pixel points, using the scene coordinate regression network to calculate the 3D scene coordinates corresponding to each feature, using the scene key point mask to obtain the scene coordinates guided by the key point, and then using the following formula to convert the reprojection error into the key point probability: Among them, ∈ i is the reprojection error of the point, and λ is used to control the error degree of the reprojection error. The threshold is set to filter the key points and obtain a probabilistic key point mask.

7. The camera relocalization method based on key point guidance according to claim 6, characterized in that: The step 8 is specifically as follows: inputting a single image data in the scene sequence image into the feature extraction network, obtaining the pixel feature and inputting it into the key point selection network, and using the final key point mask obtained in step 7 as supervision information to train the key point selection network; the training is supervised by using the following cross entropy loss: Where k represents the set of probabilities that the key point selection network predicts each pixel is a key point. represents the set of actual probabilities of each pixel, k i represents the probability of the prediction of the i-th pixel, represents the actual probability of the i-th pixel, and N is the total number of image pixels involved in the calculation that need to be used to distinguish whether they are key points.

8. The camera relocalization method based on key point guidance according to claim 7, characterized in that: The step 9 specifically includes: using the final key point mask obtained in step 7 as a screening condition, optimizing and training the scene coordinate regression network obtained in step 5, still using a single image data in the scene sequence image as input, but the supervision part only involves the part covered by the final key point mask; The supervision function used in training is the same as that in step 5.

9. The camera relocalization method based on key point guidance according to claim 8, characterized in that: The step 10 is specifically as follows: during the test process, the mask estimated by the key point selection network in step 8 is used as a basis, and the scene coordinate information corresponding to a single image data in the scene sequence image is obtained by using the scene coordinate regression network guided by the key points after optimization training in step 9, and the scene coordinate information covered by the mask is selected to estimate the camera's six-degree-of-freedom pose using the PnP and RANSAC algorithms. In the calculation process, the key points used to calculate the camera's six-degree-of-freedom pose are obtained, and these key points are taken as internal points. The confidence of the camera pose estimation is calculated using the following formula: Among them, N i is the number of internal points, N a is the number of pixels of the final keypoint mask estimated by the network in step 8, r i is the confidence of the pose estimate.

10. The camera relocalization method based on key point guidance according to claim 9, characterized in that: The step 11 is specifically as follows: if the confidence of the camera pose estimation obtained in step 10 is greater than a threshold value of 90, the camera pose estimated in step 10 is output as the final camera pose estimation; otherwise, the scene coordinate data regressed by the scene coordinate regression network in step 5 is used to estimate the camera's six-degree-of-freedom pose using the PnP and RANSAC algorithms, and this is used as the final camera pose estimation.

Images