Positioning method and system of panoramic camera in point cloud map

Abstract

The invention discloses a method and system for positioning a panoramic camera in a point cloud map, and the method comprises the steps: carrying out the preprocessing of a panoramic image and three-dimensional point cloud map data, and obtaining two-dimensional line segment features and three-dimensional line segment features; based on the two-dimensional and three-dimensional line segment features, calculating two-dimensional and three-dimensional line segment main directions and comparing to obtain candidate rotation estimation; designing a two-dimensional distance function and a three-dimensional distance function based on candidate rotation estimation to evaluate the rough pose, designing a two-dimensional / three-dimensional line segment loss function to score the candidate pose, and screening out # imgabs0 candidate poses meeting requirements according to scores; and carrying out two-dimensional and three-dimensional key point extraction on the # imgabs1 # candidate pose, matching the two-dimensional key points with the three-dimensional key points, and carrying out further pose estimation on the candidate pose with the maximum matching quantity by adopting a PnP-RANSAC algorithm, so as to obtain final fine pose estimation. The method overcomes the defects that in the prior art, point feature detection is unstable and depends too much on an ideal static environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a system for positioning a panoramic camera in a point cloud map. Background Art

[0002] The process of localizing a camera within a map consists of two main steps: first, establishing a correspondence between the current observation and known 3D structures in the map through feature matching; second, estimating the camera's position and pose in the world coordinate system based on this matching relationship using perspective geometry constraints (such as the PnP algorithm) and optimization methods (such as nonlinear least squares). Camera localization within a map is a crucial technology in computer vision. Scenarios such as virtual reality, indoor navigation, mobile robotics, and autonomous driving all require localization technology to obtain accurate position and pose information. The accuracy and robustness of localization directly impact the performance of autonomous systems in these tasks. However, current traditional point-based localization methods still rely heavily on highly textured environmental features or static scene assumptions, and are particularly sensitive to factors such as weak texture, illumination variations, and scene changes. In scenarios where it is difficult to detect and match a sufficient number of stable keypoints, such as indoor areas, the effectiveness of point-based visual localization methods is limited.

[0003] The positioning environment in actual application scenarios is very complex, and there are no stable point features in public training data sets, which hinders the implementation of positioning methods based on visual point features. Compared with the use of point features for positioning, line structure features are robust to lighting changes and motion blur, and can represent the relationship between scene structured elements and spatial layout. Indoors or in artificial scenes such as cities, there are also a large number of line segment features. The rich line segments in these scenes can improve the stability and accuracy of positioning technology. On the other hand, panoramic cameras have the advantage of a wide field of view compared to traditional perspective cameras. They can provide an overall view of the surrounding environment and are not easily affected by slight changes in the scene and local blur. However, line segment features will encounter the problem of line fragmentation caused by partial occlusion, and have the disadvantage of visual blur compared to point features, making them more difficult to be successfully positioned. The large field of view of panoramic cameras also brings the disadvantages of severe image distortion and uneven resolution, further exacerbating the visual blur of line segments. Summary of the Invention

[0004] The purpose of the present invention is to address the shortcomings of the existing technology and provide a method for positioning a panoramic camera in a point cloud map. This method can obtain more accurate pose estimation with the help of a small number of local features under the conditions of wide viewing angle and high distortion of the panoramic camera, and improve the robustness of pose calculation in real scenes, thereby solving the defects of the existing technology in which point feature detection is unstable and overly dependent on an ideal static environment.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions: A method for positioning a panoramic camera in a point cloud map comprises the following steps: Step 1: Preprocess the panoramic image and 3D point cloud map data to obtain 2D line segment features in the panoramic image and 3D line segment features in the point cloud map; Step 2: Based on the 2D and 3D line segment features obtained in step 1, the main directions of the 2D and 3D line segments are calculated and compared to obtain candidate rotation estimates. Step 3: Design 2D and 3D line segment distance functions based on the candidate rotation estimates obtained in step 2, and evaluate the rough poses according to the similarity of the 2D / 3D line segment distance functions to select 𝑘 candidate poses that meet the requirements; Step 4: For the The two-dimensional and three-dimensional key points are extracted from each candidate pose respectively, and the two-dimensional key points are matched with the three-dimensional key points. The PnP-RANSAC algorithm is used to screen out the optimal pose for the candidate pose with the largest number of matches, thereby obtaining the final fine pose estimation.

[0006] Furthermore, the method for preprocessing the panoramic image in step 1 includes: using the ULSD deep learning algorithm to detect line segment features in the panoramic image, predicting the line segment bisection points through an end-to-end model to establish a Bezier curve model, and finally directly outputting vectorized two-dimensional line segments.

[0007] Furthermore, the method for preprocessing the three-dimensional point cloud map data in step 1 includes: Use the 3DLineDetection algorithm to detect 3D line segments from the 3D point cloud map data and filter out the noise segments in the initial 3D line segment map. 3D point cloud map, filter out points with length less than 3D line segments, is a constant, and the filtering rate of the three-dimensional line segment is At the same time, filter the two-dimensional line segments extracted from the panoramic image and adjust the threshold to maintain the filtering rate of the two-dimensional and three-dimensional line segments. consistency.

[0008] Furthermore, the specific implementation of step 2 includes: Define two-dimensional and three-dimensional line segments, and project two-dimensional line segments onto the unit sphere; The 2D principal direction is calculated by multiplying the 2D line segment unit normal vector coordinate vector with the vanishing point unit normal vector coordinate vector, and the 3D principal direction is calculated by multiplying the 2D line segment unit normal vector coordinate vector with the vanishing point unit normal vector coordinate vector; Then, the Kabsch algorithm is used to find the optimal rotation matrix for aligning the 2D and 3D main directions for all possible combinations of the 2D and 3D main directions, and infeasible rotations whose mean square error exceeds the threshold are eliminated to obtain Nr feasible rotations.

[0009] Furthermore, in step 3, the designed two-dimensional line segment distance function is used to detect the line segment structure in the spherical projection of the two-dimensional panoramic query image; for a point on the unit sphere , and its two-dimensional line segment distance function is:

[0010] in, D(x, l) It is from the point The spherical distance to the line segment l = (s, e); D(x, l) is defined as:

[0011] in is a spherical quadrilateral; when the point In a spherical quadrilateral, the line segment from the point to the sphere distance For Start perpendicular to line segment; when point Outside the spherical quadrilateral, the line segment from the point to the sphere distance for To the starting point The line segment and To the end The minimum length of a line segment.

[0012] Furthermore, the 3D line segment distance function designed in step 3 is used to detect the line segment structure in the 3D arbitrary posture spherical projection. To represent the projection of a line segment in three-dimensional space onto the unit sphere, the three-dimensional line segment distance function is defined as: ; in, D Indicates from arrive spherical distance.

[0013] Furthermore, in step 3, Methods for finding candidate poses include: Quantify the similarity of 2D / 3D line segment distance functions by counting the number of inliers whose 2D / 3D line segment distance function difference is below a threshold The number of voting points is the number of inliers, and the calculation formula is: ; Divide the line segment set along multiple main directions for rotation value estimation, define a separate 2D / 3D line segment distance function for each group of line segments, and record the decomposed distance function as and , represents the i-th main direction, n represents the total number of main directions, and the decomposed loss function is: ; Calculate the loss function score for all candidate poses and select the one with the highest score. The candidate poses are used as a feasible pose estimation set and enter the next step of pose optimization.

[0014] Furthermore, in step 4, Methods for extracting and matching key points for each candidate pose include: Use the SuperPoint algorithm to extract local features on the panoramic image and obtain a set of key points; To detect local features in a 3D environment, a 3D point cloud map of the environment is used to render a synthetic view of fixed-interval sampling locations. The input point cloud is sampled at fixed intervals to obtain a set of locations. The point cloud is projected onto a virtual panoramic camera at each location to create a synthetic image. The color of the 3D environment point cloud is assigned to the projected location of the corresponding 3D coordinate. The SuperPoint algorithm is used to extract local features in each synthetic panoramic image view. The key points detected in the 2D synthetic planar image are then back-projected to the corresponding locations in 3D space to obtain a 3D environment map with embedded local features. After obtaining the local features of the 2D query image and 3D space, for each candidate pose, we first search for all visible local features in the 3D environment under that pose, record the set of visible 3D key points, and then use the SuperGlue algorithm to perform local feature matching with the 2D key points in the 2D query image.

[0015] Furthermore, the method of using the PnP-RANSAC algorithm to screen out the optimal pose includes: Input all matching pairs, randomly select a minimum sample set from the matching set, and solve a candidate camera pose through the EPnP solver; Use the solved candidate poses to verify all matching pairs in the matching set. For each matching pair, transform the 3D point into the camera coordinate system and calculate the direction vector of the transformed point. The angular error between the above direction vector and the observation direction vector obtained by back-projection of the two-dimensional matching point is calculated. If the angular error is less than a preset threshold, the matching point is considered to be an inlier of the current candidate pose, and the total number of inliers supported by the current candidate pose is recorded. If the total number of inliers of the current candidate pose is greater than the maximum number of inliers found in the previous iteration, the best inlier set is updated. When the number of iterations reaches the initially set number of iterations, the iteration is terminated, and the pose corresponding to the best inlier set is the final refined pose estimate.

[0016] Another object of the present invention is to provide a system for implementing the above-mentioned method for positioning a panoramic camera in a point cloud map, comprising: The line segment detection module is used to pre-process the panoramic image and 3D point cloud map data to obtain the 2D line segment features in the panoramic image and the 3D line segment features in the point cloud map; A candidate rotation estimation module is used to calculate and compare the main directions of the two-dimensional and three-dimensional line segments based on the two-dimensional and three-dimensional line segment features obtained by the line segment detection module to obtain candidate rotation estimates; The candidate pose screening module is used to design two-dimensional and three-dimensional line segment distance functions based on the candidate rotation estimation obtained by the candidate rotation estimation module, and evaluate the rough pose according to the similarity of the two-dimensional / three-dimensional line segment distance function to screen out the ones that meet the requirements. candidate poses; The fine pose estimation module is used to obtain The two-dimensional and three-dimensional key points are extracted from each candidate pose respectively, and the two-dimensional key points are matched with the three-dimensional key points. The PnP-RANSAC algorithm is used to screen out the optimal pose for the candidate pose with the largest number of matches, thereby obtaining the final fine pose estimation.

[0017] Compared with the prior art, the present invention has the following advantages: the present invention obtains an initial rotation estimate by comparing the principal directions of line segment structures in the panoramic image and point cloud map; enhances the method's directional perception capability by designing robust 2D / 3D line segment distance functions and loss functions, and selects an initial rough pose based on the loss function score; optimizes local feature matching using the PnP-RANSAC algorithm to obtain a final precise pose, thereby achieving robust positioning of line segment features in the panoramic camera image in the point cloud map; furthermore, the present invention uses the SuperPoint algorithm to extract local features in the 2D query image and 3D environment to obtain a set of key points; and optimizes local feature matching using the PnP-RANSAC algorithm. Adding the RANSAC algorithm to the pose calculated by the PnP algorithm can filter out mismatched pairs, avoid the influence of outliers on the results, and improve the robustness of pose calculation in real scenes. It can obtain more accurate pose estimates with the help of a small number of local features under the conditions of wide viewing angles and high distortion of panoramic cameras, and improves the robustness of pose calculation in real scenes, thereby addressing the defects of the prior art in terms of instability of point feature detection and over-reliance on an ideal static environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a block diagram of the overall structure of a method for positioning a panoramic camera in a point cloud map according to an embodiment of the present invention; Figure 2 This is a comparison diagram of the reprojection result and the input query image for positioning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0021] The present invention will be further described below with reference to specific examples, but they are not intended to limit the present invention.

[0022] like Figure 1 As shown, an embodiment of the present invention discloses a method for positioning a panoramic camera in a point cloud map, comprising the following steps: Step 1: Preprocess the panoramic image and 3D point cloud map data to obtain 2D line segment features in the panoramic image and 3D line segment features in the point cloud map; In this step, the panoramic image captured by the panoramic camera is preprocessed. The ULSD deep learning algorithm is used to detect line segment features in the image. The end-to-end model predicts the segment equidivision points to establish a Bezier curve model, and directly outputs vectorized line segments. Specifically, the ULSD algorithm is a line segment detection method based on deep learning. It uses a Bezier curve representation based on equidivision points to model the detected line segments. The panoramic image is input into the ULSD algorithm's neural network, and the feature extraction backbone network outputs a shared feature map. The shared feature map is input into the line segment proposal network to predict the image's Bezier curve equidivision points and model the predicted line segments. The Bezier alignment module then determines the validity of the line segments, and refines the Bezier curve control points to improve the accuracy of line segment detection. Finally, the line segments in the panoramic image are extracted.

[0023] For the preprocessing of 3D point cloud maps, the 3DLineDetection algorithm is used to detect 3D line segments from large-scale unorganized point clouds. In order to prevent the positioning process from being affected by noise and enhance the robustness of the algorithm, it is necessary to filter out short noise line segments in the initial 3D line segment map. Given a bounding box range of 3D point cloud map, filter out points with length less than 3D line segments, is a constant, and in this embodiment the value is = 0.1, the filtering rate of the three-dimensional line segment is At the same time, filter the two-dimensional line segments extracted from the panoramic image and adjust the threshold to maintain the filtering rate of the two-dimensional and three-dimensional line segments. consistency.

[0024] Step 2: Based on the 2D and 3D line segment features obtained in step 1, calculate the main directions of the 2D and 3D line segments and compare them to obtain candidate rotation estimates. This step specifically includes: The first step is to define two-dimensional and three-dimensional line segments and project the two-dimensional line segments onto the unit sphere. is a set of two-dimensional line segments, where is an element containing the start and end points of a line segment; define is a set of two-dimensional line segments, where is an element containing the start and end points of a line segment.

[0025] The second step is to calculate the main directions of the two-dimensional and three-dimensional segments. for The most common line directions, which are the directions where the most line segments converge in the image. To calculate the 2D principal direction, we need to find the intersection of the detected line segments and extract the vanishing points. Then, we multiply the coordinates of the 2D line segment unit normal vector by the coordinates of the vanishing point unit normal vector. If the value is less than a given threshold, the line segment normal vector is approximately perpendicular to the vanishing point vector, meaning that the vanishing point is on the extended line of the 2D line segment on the sphere.

[0026] Define the main direction of a 3D line segment The one with the most votes in the three-dimensional direction The direction of the three-dimensional line segment. Multiply the coordinate vector of the unit normal vector of the line segment with the coordinate vector of the unit normal vector of the vanishing point. If the product is greater than the set threshold, the two three-dimensional line segments are approximately parallel, that is, the two three-dimensional line segments are in the same main direction. In the approximate case, all candidate vanishing points form a uniformly sampled spherical grid, and the line segment with the most votes through the above voting is and points, two-dimensional line segments and three-dimensional line segments are obtained through the above voting mechanism respectively 2D vanishing points and The directions represented by these vanishing points are the main directions of the two-dimensional and three-dimensional line segments respectively.

[0027] The third step is to estimate feasible candidate rotation values. Due to the uncertainty caused by the combination ambiguity in matching, it is necessary to consider multiple matching combinations to estimate candidate rotations by matching the 2D and 3D main directions. In order to fully consider the ambiguity of the line segment arrangement, it is necessary to The principal directions of the two-dimensional line segments and Three directions are selected from the main directions of the three-dimensional line segments to form a triplet, and a total of For each matching triple, the Kabsch algorithm is used to find the optimal rotation matrix for aligning the 2D main direction with the 3D main direction, and the infeasible rotations with a mean square error greater than the set threshold are eliminated, and finally the Possible rotations.

[0028] Step 3: Design 2D and 3D line segment distance functions based on the candidate rotation estimates obtained in step 2, and evaluate the rough poses according to the similarity of the 2D / 3D line segment distance functions to select 𝑘 candidate poses that meet the requirements; This step evaluates the rough pose based on the candidate rotation estimate and 2D / 3D line segment distance function obtained in step 2, compares the similarity of the unit sphere projection, and can match the candidate pose without establishing an explicit correspondence. The specific approach is: for the evaluation and selection of candidate poses, it is necessary to design a specific distance function and loss function, make full use of the line segment structure feature information, and screen the most feasible camera pose to provide the initial value for the final pose optimization link. To this end, a 2D / 3D line segment distance function is designed, and a pose loss function is designed based on the line segment distance function. The candidate pose set comes from the combination of the candidate rotation value and the sampled translation value obtained in step 2. Select in the 3D point cloud map translation values, a total of Candidate poses In order to evaluate candidate poses, a 2D distance function is designed to detect line segment structures in the 2D panoramic query image and the 3D arbitrary pose spherical projection. , and its two-dimensional line segment distance function is:

[0029] in, D(x, l) It is from the point The spherical distance to the line segment l = (s, e); D(x, l) is defined as:

[0030] in is a spherical quadrilateral; when the point In a spherical quadrilateral, the line segment from the point to the sphere distance For Start perpendicular to line segment; when point Outside the spherical quadrilateral, the line segment from the point to the sphere distance for To the starting point The line segment and To the end The minimum length of a line segment.

[0031] use To represent the projection of a line segment in three-dimensional space onto the unit sphere. Then the three-dimensional line segment distance function is defined as:

[0032] Where, D Indicates from arrive spherical distance.

[0033] Evaluated based on the similarity of 2D / 3D line segment distance function In this embodiment, the number of inliers is used to quantify the similarity of the 2D / 3D line segment distance function. The 2D line segment distance function is calculated from the query panoramic image information, and the 3D line segment distance function is calculated from the line segment map information. For candidate poses that are more similar to the true pose, their 2D / 3D line segment distance functions are more similar. Therefore, the similarity of the 2D / 3D line segment distance function can be used to evaluate the candidate pose. For each candidate pose, the similarity of the 2D / 3D line segment distance function can be quantified by counting the number of inliers whose 2D / 3D line segment distance function difference is lower than a threshold The number of voting points is the number of inliers, and the calculation formula for the number of inliers is: ; In order to enhance the pose search capability of the 2D / 3D line segment distance function, it is necessary to decompose the 2D / 3D line segment distance function along the three main directions. The line segment set is divided along the three main directions used for rotation value estimation, and a separate 2D / 3D line segment distance function is defined for each group of line segments. The decomposed distance function is recorded as and and , the decomposed loss function is: ; for The loss function score is calculated for each candidate pose, and the one with the highest score is selected. The candidate poses are used as a feasible pose estimation set and enter the next step of camera pose optimization.

[0034] Step 4: For the The two-dimensional and three-dimensional key points are extracted from each candidate pose respectively, and the two-dimensional key points are matched with the three-dimensional key points. The PnP-RANSAC algorithm is used to screen out the optimal pose for the candidate pose with the largest number of matches, thereby obtaining the final fine pose estimation.

[0035] Step 3 obtained The candidate poses are rough poses during global localization. Further pose optimization is required to obtain a refined optimal pose estimate. Using local feature matching combined with the PnP-RANSAC algorithm can achieve more accurate pose estimates and improve the robustness of pose calculation in real-world scenarios. Specifically, the SuperPoint algorithm is used directly on the 2D query image from the panoramic camera to extract local features and generate a set of keypoints. Because the SuperPoint algorithm is robust to viewpoint changes and distortion, the 2D query image from the panoramic camera can be directly input into the SuperPoint algorithm. The input image passes through the encoder network to generate a high-dimensional feature map. This feature map is then fed into the keypoint detection branch, which outputs a low-resolution keypoint probability heatmap proportional to the original image size. The feature map output by the encoder is also fed into the descriptor generation branch, which, through convolution, outputs a dense feature descriptor graph. By filtering the heatmap and descriptor graph to suppress non-maximum points and apply confidence thresholds, the keypoint coordinates are extracted and then upsampled to the original image coordinates, forming a set of keypoints and their descriptors for the 2D image. To detect local features in a 3D environment, a 3D point cloud map of the environment is used to render a synthetic view of locations sampled at fixed intervals. The input point cloud is sampled at fixed intervals to obtain a set of locations. The point cloud is projected onto a virtual panoramic camera at each location to create a synthetic image. The color of the 3D environment point cloud is assigned to the projected location of the corresponding 3D coordinate. The SuperPoint algorithm is used to extract local features from each synthetic panoramic image view. Key points detected in the 2D synthetic planar image are then back-projected to corresponding locations in 3D space, resulting in a 3D environment map embedded with the local features.

[0036] After obtaining the local features of the two-dimensional query image and the three-dimensional space, for each candidate pose, first search for all visible local features in the three-dimensional environment under the pose, record the set of visible three-dimensional key points, and then use the SuperGlue algorithm to match the local features with the two-dimensional key points in the two-dimensional query image. Record the visible three-dimensional point set. For the two sets of visible three-dimensional key points and two-dimensional image key points, use the descriptor and coordinates of each key point to encode them into high-dimensional vectors, and obtain two high-dimensional vector sets to input into the SuperGlue algorithm. SuperGlue's graph neural network regards the elements in the two sets as nodes of two graphs, performs self-attention operations within the sets to understand the contextual relationship, and performs cross-attention operations between sets to evaluate the similarity between the two graphs. The matching layer calculates the soft assignment matrix for the node features output by the graph neural network, which represents the confidence of the match between the key points of the two sets. After the confidence filtering operation, the matching pairs with confidence higher than the threshold are retained to obtain the matching relationship between the two-dimensional and three-dimensional key points. For the obtained in step 3 The key points are extracted and matched for each candidate pose respectively, and the candidate pose with the largest number of matches is selected to proceed to the next step.

[0037] The PnP-RANSAC algorithm is applied to the optimal pose with the most 2D and 3D keypoint matches to obtain the final refined pose estimate. The position with the most 2D and 3D keypoint matches is the optimal pose. The matching relationship between the 2D and 3D keypoints obtained above may contain some incorrect matching pairs. The PnP-RANSAC algorithm can filter out inliers and solve for the optimal pose. The specific process is to input all matching pairs, randomly select a minimum sample set from the matching set (usually 4), and use the EPnP solver to solve a candidate camera pose. The solved candidate pose is then used to verify all matching pairs in the matching set. For each matching pair, the 3D point is transformed into the camera coordinate system, and the direction vector of the transformed point is calculated. The angular error between this direction vector and the observation direction vector obtained by back-projecting the 2D matching point is calculated. If the angular error is less than a preset threshold, the matching point is considered an inlier in the current candidate pose, and the total number of inliers supported by the current candidate pose is recorded. If the total number of inliers in the current candidate pose is greater than the maximum number of inliers found in the previous iteration, the best inlier set is updated. When the number of iterations reaches the initial set number of iterations, the iteration is terminated, and the pose corresponding to the best inlier set is the final refined pose estimate.

[0038] Because local features are less affected by factors such as spatial scale and perspective changes, they can maintain discriminative capabilities under the wide viewing angles and high distortion conditions of panoramic cameras. With the help of a small number of local features, they can provide additional information for accurate pose estimation. This embodiment uses the SuperPoint algorithm to extract local features in a two-dimensional query image and a three-dimensional environment to obtain a set of key points. The PnP-RANSAC algorithm is used to optimize local feature matching. Adding the RANSAC algorithm to the pose calculation based on the PnP algorithm can filter out mismatched pairs, avoid the influence of outliers on the results, and improve the robustness of pose calculation in real scenes.

[0039] The present invention conducts experiments on the Stanford-2D-3D-S dataset and the OmniScenes dataset. The 3D point cloud map is reprojected at the output pose of the method and compared with the input query image. The experimental results are shown in the attached figure. Figure 2As shown in Table 1, the reprojection result is almost perfectly consistent with the input query image, which is sufficient to demonstrate that the method achieves the expected results in terms of positioning accuracy. In terms of quantitative indicators, the experiment measured the median error between the algorithm's estimated translation value and the true translation value, and the median error between the estimated rotation value and the true rotation value to evaluate the stability of the positioning method. At the same time, the proportion of accurate positioning was measured. As shown in Table 1, the translation error was controlled within 0.04 meters on the dataset, the rotation error was controlled within 0.77 degrees, and the accuracy was controlled above 0.84. This well demonstrates that the method of the present invention is extremely effective in positioning in complex indoor environments.

[0040] Table 1 Error evaluation values ​​of the embodiment of the present invention

[0041] An embodiment of the present invention further provides a system for implementing the above-mentioned method for positioning a panoramic camera in a point cloud map, comprising: The line segment detection module is used to pre-process the panoramic image and 3D point cloud map data to obtain the 2D line segment features in the panoramic image and the 3D line segment features in the point cloud map; A candidate rotation estimation module is used to calculate and compare the main directions of the two-dimensional and three-dimensional line segments based on the two-dimensional and three-dimensional line segment features obtained by the line segment detection module to obtain candidate rotation estimates; The candidate pose screening module is used to design two-dimensional and three-dimensional line segment distance functions based on the candidate rotation estimation obtained by the candidate rotation estimation module, and evaluate the rough pose according to the similarity of the two-dimensional / three-dimensional line segment distance function to screen out the ones that meet the requirements. candidate poses; The fine pose estimation module is used to obtain The two-dimensional and three-dimensional key points are extracted from each candidate pose respectively, and the two-dimensional key points are matched with the three-dimensional key points. The PnP-RANSAC algorithm is used to screen out the optimal pose for the candidate pose with the largest number of matches, thereby obtaining the final fine pose estimation.

[0042] The above are only preferred embodiments of the present invention and do not limit the implementation mode and protection scope of the present invention. For those skilled in the art, it should be aware that all solutions obtained by equivalent substitutions and obvious changes made using the contents of the present invention specification should be included in the protection scope of the present invention.

Claims

1. A method for positioning a panoramic camera in a point cloud map, characterized in that: The following steps are involved: Step 1: Preprocess the panoramic image and 3D point cloud map data to obtain 2D line segment features in the panoramic image and 3D line segment features in the point cloud map; Step 2: Based on the 2D and 3D line segment features obtained in step 1, the main directions of the 2D and 3D line segments are calculated and compared to obtain candidate rotation estimates. Step 3: Design 2D and 3D line segment distance functions based on the candidate rotation estimates obtained in step 2, and evaluate the rough poses according to the similarity of the 2D / 3D line segment distance functions to select the ones that meet the requirements. candidate poses; Step 4: For the The two-dimensional and three-dimensional key points are extracted from each candidate pose respectively, and the two-dimensional key points are matched with the three-dimensional key points. The PnP-RANSAC algorithm is used to screen out the optimal pose for the candidate pose with the largest number of matches, thereby obtaining the final fine pose estimation.

2. The method for positioning a panoramic camera in a point cloud map according to claim 1, wherein: The method for preprocessing the panoramic image in step 1 includes: using the ULSD deep learning algorithm to detect line segment features in the panoramic image, predicting the line segment bisection points through an end-to-end model to establish a Bezier curve model, and finally directly outputting vectorized two-dimensional line segments.

3. The method for positioning a panoramic camera in a point cloud map according to claim 1, wherein: The method for preprocessing the 3D point cloud map data in step 1 includes: Use the 3DLineDetection algorithm to detect 3D line segments from the 3D point cloud map data and filter out the noise segments in the initial 3D line segment map. 3D point cloud map, filter out points with length less than 3D line segment, is a constant, and the filtering rate of the three-dimensional line segment is At the same time, filter the two-dimensional line segments extracted from the panoramic image and adjust the threshold to maintain the filtering rate of the two-dimensional and three-dimensional line segments. consistency.

4. The method for positioning a panoramic camera in a point cloud map according to claim 1, wherein: The specific implementation methods in step 2 include: Define two-dimensional and three-dimensional line segments, and project two-dimensional line segments onto the unit sphere; The 2D principal direction is calculated by multiplying the 2D line segment unit normal vector coordinate vector with the vanishing point unit normal vector coordinate vector, and the 3D principal direction is calculated by multiplying the 2D line segment unit normal vector coordinate vector with the vanishing point unit normal vector coordinate vector; Then, the Kabsch algorithm is used to find the optimal rotation matrix for aligning the 2D and 3D main directions for all possible combinations of the 2D and 3D main directions, and infeasible rotations whose mean square error exceeds the threshold are eliminated to obtain Nr feasible rotations.

5. The method for positioning a panoramic camera in a point cloud map according to claim 1, wherein: In step 3, the designed two-dimensional line segment distance function is used to detect the line segment structure in the spherical projection of the two-dimensional panoramic query image; for a point on the unit sphere , and its two-dimensional line segment distance function is: in, D(x, l) It is from the point The spherical distance to the line segment l = (s, e); D(x, l) is defined as: in is a spherical quadrilateral; when the point In a spherical quadrilateral, the line segment from the point to the sphere distance For Start perpendicular to line segment; when point Outside the spherical quadrilateral, the line segment from the point to the sphere distance for To the starting point The line segment and To the end The minimum length of a line segment.

6. The method for positioning a panoramic camera in a point cloud map according to claim 4, wherein: The 3D line segment distance function designed in step 3 is used to detect the line segment structure in the spherical projection of 3D arbitrary posture. To represent the projection of a line segment in three-dimensional space onto the unit sphere, the three-dimensional line segment distance function is defined as: ； in, D Indicates from arrive spherical distance.

7. The method for positioning a panoramic camera in a point cloud map according to claim 1, wherein: Filter in step 3 Methods for finding candidate poses include: Quantify the similarity of 2D / 3D line segment distance functions by counting the number of inliers whose 2D / 3D line segment distance function difference is below a threshold The number of voting points is the number of inliers, and the calculation formula is: ； Divide the line segment set along multiple main directions for rotation value estimation, define a separate 2D / 3D line segment distance function for each group of line segments, and record the decomposed distance function as and , represents the i-th main direction, n represents the total number of main directions, and the decomposed loss function is: ； Calculate the loss function score for all candidate poses and select the one with the highest score. The candidate poses are used as a feasible pose estimation set and enter the next step of pose optimization.

8. The method for positioning a panoramic camera in a point cloud map according to claim 1, wherein: In step 4, Methods for extracting and matching key points for each candidate pose include: Use the SuperPoint algorithm to extract local features on the panoramic image and obtain a set of key points; To detect local features in a 3D environment, a 3D point cloud map of the environment is used to render a synthetic view of fixed-interval sampling locations. The input point cloud is sampled at fixed intervals to obtain a set of locations. The point cloud is projected onto a virtual panoramic camera at each location to create a synthetic image. The color of the 3D environment point cloud is assigned to the projected location of the corresponding 3D coordinate. The SuperPoint algorithm is used to extract local features in each synthetic panoramic image view. The key points detected in the 2D synthetic planar image are then back-projected to the corresponding locations in 3D space to obtain a 3D environment map with embedded local features. After obtaining the local features of the 2D query image and 3D space, for each candidate pose, we first search for all visible local features in the 3D environment under that pose, record the set of visible 3D key points, and then use the SuperGlue algorithm to perform local feature matching with the 2D key points in the 2D query image.

9. The method for positioning a panoramic camera in a point cloud map according to claim 1, wherein: Methods for selecting the optimal pose using the PnP-RANSAC algorithm include: Input all matching pairs, randomly select a minimum sample set from the matching set, and solve a candidate camera pose through the EPnP solver; Use the solved candidate poses to verify all matching pairs in the matching set. For each matching pair, transform the 3D point into the camera coordinate system and calculate the direction vector of the transformed point. The angular error between the above direction vector and the observation direction vector obtained by back-projection of the two-dimensional matching point is calculated. If the angular error is less than a preset threshold, the matching point is considered to be an inlier of the current candidate pose, and the total number of inliers supported by the current candidate pose is recorded. If the total number of inliers of the current candidate pose is greater than the maximum number of inliers found in the previous iteration, the best inlier set is updated. When the number of iterations reaches the initially set number of iterations, the iteration is terminated, and the pose corresponding to the best inlier set is the final refined pose estimate.

10. A system for implementing the method for positioning a panoramic camera in a point cloud map according to any one of claims 1 to 9, characterized in that: include: The line segment detection module is used to pre-process the panoramic image and 3D point cloud map data to obtain the 2D line segment features in the panoramic image and the 3D line segment features in the point cloud map; A candidate rotation estimation module is used to calculate and compare the main directions of the two-dimensional and three-dimensional line segments based on the two-dimensional and three-dimensional line segment features obtained by the line segment detection module to obtain candidate rotation estimates; The candidate pose screening module is used to design two-dimensional and three-dimensional line segment distance functions based on the candidate rotation estimates obtained by the candidate rotation estimation module, and evaluate the rough pose according to the similarity of the two-dimensional / three-dimensional line segment distance functions to screen out 𝑘 candidate poses that meet the requirements; The fine pose estimation module is used to obtain The two-dimensional and three-dimensional key points are extracted from each candidate pose respectively, and the two-dimensional key points are matched with the three-dimensional key points. The PnP-RANSAC algorithm is used to screen out the optimal pose for the candidate pose with the largest number of matches, thereby obtaining the final fine pose estimation.

Images