Pose estimation method and device based on feature points and semantic information, medium and equipment

Abstract

The application discloses a pose estimation method and device based on feature points and semantic information, a medium and equipment. The first step of the method optimizes the extraction of feature points based on the result of semantic segmentation, including eliminating dynamic feature points, accurately performing feature matching through a semantic descriptor, and restoring the real scale through the segmented ground feature points, thereby avoiding the scale uncertainty of traditional monocular SLAM. Finally, the inter-frame pose is calculated preliminarily through the data correlation between frames, and a pose estimation with real scale can be output accurately. The second step optimizes the SLAM backend optimization algorithm by fusing semantic information. The medium-term semantic constraint is established based on the semantic information, including the semantic general constraint and the semantic constraint established for the ground and the rod according to the semantic category; the graph structure is established in combination with the various constraints, and the optimization is performed through the g2o optimization library, so that the cumulative drift caused by long-term motion of the robot is effectively inhibited.

Description

Technical Field

[0001] This application relates to the field of robot vision positioning and navigation, and in particular to a pose estimation method, apparatus, medium and device based on feature points and semantic information. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is an important direction in mobile robot localization and navigation technology. It refers to a moving object achieving self-localization and simultaneously building a map of its surrounding environment using only its onboard sensors. Because cameras are cheaper and smaller than laser sensors, and capture richer images, vision-based SLAM (N-SLAM, or visual SLAM) has become a research hotspot. Compared to computationally intensive binocular cameras and RGB-D depth cameras, monocular cameras offer advantages such as lower power consumption, smaller size, and lighter weight, especially on small mobile robot platforms (such as micro-drones). Therefore, monocular SLAM technology can be applied to various mobile robot platforms and has significant research value.

[0003] SLAM front-end and back-end algorithms are categorized into direct methods, indirect methods, high-level feature-based methods, and semantic information-based methods. Direct methods assume grayscale invariance and estimate pose transformation by optimizing the photometric error of corresponding pixels during continuous camera observations. They can construct semi-dense or even dense maps. However, direct methods are non-convex, making them prone to failure under significant motion and susceptible to illumination changes. Indirect methods optimize by minimizing the reprojection error between 3D map points and their observed projections in the image. Because indirect methods estimate pose by detecting and matching local features, they are generally more robust to viewpoint and illumination changes. While indirect methods offer improvements in robustness and accuracy compared to direct methods, they are prone to failure in scenes with low texture or missing features. Furthermore, feature detectors and descriptors are not entirely invariant to such changes. Therefore, indirect methods still suffer from the same limitations as direct methods, failing to track feature points over long timeframes. Summary of the Invention

[0004] In view of this, this application provides a pose estimation method, apparatus, medium and device based on feature points and semantic information, which integrates semantic information into SLAM and solves the problems of insufficient localization and pose drift in existing methods.

[0005] According to one aspect of this application, a pose estimation method based on feature points and semantic information is provided, comprising:

[0006] Acquire images captured by a mobile device, and perform semantic segmentation on the images to obtain the semantic information corresponding to the images;

[0007] Multiple feature points are identified in the image, a target descriptor is determined for each feature point, and dynamic feature points are removed from the multiple feature points based on the semantic information.

[0008] Based on the target descriptor, a matching point is determined for each feature point to obtain multiple matching pairs, and the multiple matching pairs are then filtered.

[0009] Perform map initialization and determine ground points in the initialized map;

[0010] The true scale is estimated based on the ground points, and the pose estimate of the mobile device at the true scale is determined using a constant velocity model.

[0011] Optionally, determining the target descriptor for each of the feature points includes:

[0012] Based on the semantic information, a semantic descriptor for each feature point is determined, and the semantic descriptor and the BRIEF descriptor of the feature point are combined to obtain the target descriptor of the feature point.

[0013] Optionally, the step of determining matching points for each feature point based on the target descriptor to obtain multiple matching pairs includes:

[0014] Based on the current pose of the mobile device and the camera intrinsic parameters, the feature points are projected to obtain the projected coordinates of the feature points in the current frame.

[0015] The feature points are provided with scale information through an image pyramid, and the search range is determined based on the scale information. Within the search range, the candidate matching points corresponding to the feature points are determined based on the semantic descriptor.

[0016] The similarity between the candidate matching point and the feature point is verified by a similarity formula. If the verification passes, the candidate matching point is the matching point corresponding to the feature point. The feature point and the matching point form a matching pair, wherein each feature point corresponds to one or more matching points.

[0017] Optionally, filtering the plurality of matching pairs includes:

[0018] Calculate the distance between the target descriptor corresponding to the feature point and the target descriptor corresponding to the matching point. If the distance is greater than a first preset threshold, then discard the matching pair consisting of the feature point and the matching point.

[0019] Among the matching points corresponding to the feature points, the best matching point and the second best matching point are determined. If the best matching point and the second best matching point do not meet the preset advantage verification condition, the matching pair corresponding to the feature point is removed.

[0020] The directions of all matching pairs are counted to obtain a direction histogram. The rotation direction is determined based on the direction histogram, and matching pairs whose directions are inconsistent with the rotation direction are removed.

[0021] Optionally, determining ground points in the initialized map includes:

[0022] Keyframes are determined using nearest neighbor search and then inserted into the keyframes.

[0023] The pose of the mobile device is obtained by processing the matching pairs between the current frame and keyframes using epipolar set constraints.

[0024] In the initialized map, the coordinates of the map points corresponding to the matching pairs are determined based on the matching pairs between the current frame and keyframes and the pose of the mobile device.

[0025] Through semantic segmentation, feature points on the ground are separated from the map points and denoted as ground points;

[0026] Calculate the horizontal distance between each ground point and the origin of the camera coordinate system corresponding to the mobile device, and remove ground points whose horizontal distance is greater than a second preset threshold.

[0027] Optionally, estimating the true scale based on the ground points and determining the pose of the mobile device at the true scale using a constant velocity model includes:

[0028] The actual scale is determined based on the height of the ground point and the height of the mobile device;

[0029] The pose of the mobile device and the coordinates of the ground point are adjusted according to the actual scale.

[0030] The matching pairs between the current frame and keyframes, as well as the map points, are input into the constant velocity model, and the BA method is used to reduce the reprojection bias to obtain the pose estimate of the mobile device.

[0031] Optionally, the method further includes:

[0032] An error function is established based on the semantic category, where the semantic category is either road surface or pole.

[0033] Keyframes are inserted by tracking the number of shared feature points;

[0034] Construct a likelihood model, wherein the likelihood model is used for map initialization;

[0035] Adjust the likelihood model based on the error function;

[0036] And / or,

[0037] A graph structure is established using the current keyframe and the corresponding co-view keyframe, and the graph structure is optimized using the g2o optimization library;

[0038] The pose of the mobile device and the coordinates of the map points are updated using the optimized graph structure.

[0039] According to another aspect of this application, a pose estimation device based on feature points and semantic information is provided, the device comprising:

[0040] The semantic segmentation module is used to acquire images captured by a mobile device and perform semantic segmentation on the images to obtain the semantic information corresponding to the images;

[0041] The feature point extraction module is used to determine multiple feature points in the image, determine the target descriptor for each feature point, and remove dynamic feature points from the multiple feature points according to the semantic information.

[0042] The matching module is used to determine a matching point for each feature point based on the target descriptor, obtain multiple matching pairs, and filter the multiple matching pairs;

[0043] The pose estimation module is used to initialize the map and determine ground points in the initialized map; and to estimate the true scale based on the ground points and use a constant velocity model to determine the pose estimate of the mobile device at the true scale.

[0044] Optionally, the feature point extraction module is used for:

[0045] Based on the semantic information, a semantic descriptor for each feature point is determined, and the semantic descriptor and the BRIEF descriptor of the feature point are combined to obtain the target descriptor of the feature point.

[0046] Optionally, the matching module is used for:

[0047] Based on the current pose of the mobile device and the camera intrinsic parameters, the feature points are projected to obtain the projected coordinates of the feature points in the current frame.

[0048] The feature points are provided with scale information through an image pyramid, and the search range is determined based on the scale information. Within the search range, the candidate matching points corresponding to the feature points are determined based on the semantic descriptor.

[0049] The similarity between the candidate matching point and the feature point is verified by a similarity formula. If the verification passes, the candidate matching point is the matching point corresponding to the feature point. The feature point and the matching point form a matching pair, wherein each feature point corresponds to one or more matching points.

[0050] Optionally, the matching module is used for:

[0051] Calculate the distance between the target descriptor corresponding to the feature point and the target descriptor corresponding to the matching point. If the distance is greater than a first preset threshold, then discard the matching pair consisting of the feature point and the matching point.

[0052] Among the matching points corresponding to the feature points, the best matching point and the second best matching point are determined. If the best matching point and the second best matching point do not meet the preset advantage verification condition, the matching pair corresponding to the feature point is removed.

[0053] The directions of all matching pairs are counted to obtain a direction histogram. The rotation direction is determined based on the direction histogram, and matching pairs whose directions are inconsistent with the rotation direction are removed.

[0054] Optionally, the pose estimation module is used for:

[0055] Keyframes are determined using nearest neighbor search and then inserted into the keyframes.

[0056] The pose of the mobile device is obtained by processing the matching pairs between the current frame and keyframes using epipolar set constraints.

[0057] In the initialized map, the coordinates of the map points corresponding to the matching pairs are determined based on the matching pairs between the current frame and keyframes and the pose of the mobile device.

[0058] Through semantic segmentation, feature points on the ground are separated from the map points and denoted as ground points;

[0059] Calculate the horizontal distance between each ground point and the origin of the camera coordinate system corresponding to the mobile device, and remove ground points whose horizontal distance is greater than a second preset threshold.

[0060] Optionally, the pose estimation module is used for:

[0061] The actual scale is determined based on the height of the ground point and the height of the mobile device;

[0062] The pose of the mobile device and the coordinates of the ground point are adjusted according to the actual scale.

[0063] The matching pairs between the current frame and keyframes, as well as the map points, are input into the constant velocity model, and the BA method is used to reduce the reprojection bias to obtain the pose estimate of the mobile device.

[0064] Optionally, the device further includes a back-end optimization module for:

[0065] An error function is established based on the semantic category, where the semantic category is either road surface or pole.

[0066] Keyframes are inserted by tracking the number of shared feature points;

[0067] Construct a likelihood model, wherein the likelihood model is used for map initialization;

[0068] Adjust the likelihood model based on the error function;

[0069] And / or,

[0070] A graph structure is established using the current keyframe and the corresponding co-view keyframe, and the graph structure is optimized using the g2o optimization library;

[0071] The pose of the mobile device and the coordinates of the map points are updated using the optimized graph structure.

[0072] According to another aspect of this application, a medium is provided that stores a program or instructions thereon, which, when executed by a processor, implement the above-described pose estimation method based on feature points and semantic information.

[0073] According to another aspect of this application, an apparatus is provided, including a storage medium and a processor, the storage medium storing a computer program, the processor executing the computer program to implement the above-described pose estimation method based on feature points and semantic information.

[0074] By employing the above technical solution, this application achieves the following beneficial effects:

[0075] 1. This invention designs a visual odometry method based on feature point-semantic information. The extraction of feature points is optimized by fusing the results of semantic segmentation, including removing dynamic feature points and performing accurate feature matching through semantic descriptors. The true scale is recovered from the segmented ground feature points, avoiding the scale uncertainty inherent in traditional monocular SLAM. Finally, inter-frame pose is preliminarily calculated through inter-frame data association, which can output a relatively accurate pose estimate with true scale.

[0076] 2. This invention employs a SLAM backend optimization algorithm that integrates semantic information. Mid-term semantic constraints are established based on semantic information, including general semantic constraints and semantic constraints specifically for the semantic categories of ground and pole. A graph structure is built by combining these constraints and optimized using the g2o optimization library, thereby effectively suppressing the cumulative drift caused by the robot's long-term motion.

[0077] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0078] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0079] Figure 1 The diagram illustrates a pose estimation method based on feature points and semantic information provided in an embodiment of this application.

[0080] Figure 2 This illustration shows a schematic diagram of the implementation process of a pose estimation method based on feature points and semantic information provided in an embodiment of this application;

[0081] Figure 3 This illustration shows a schematic diagram of the region division of ground points when estimating the true scale using a pose estimation method based on feature points and semantic information provided in an embodiment of this application.

[0082] Figure 4 This illustration shows a comparison of trajectories recovered from ground points in the KITTI dataset 00 using a pose estimation method based on feature points and semantic information, as provided in an embodiment of this application.

[0083] Figure 5 This embodiment of the present application illustrates a pose estimation method based on feature points and semantic information, which combines various constraints to construct a g2o backend optimization graph structure.

[0084] Figure 6 The diagram shows a structural block diagram of a pose estimation device based on feature points and semantic information provided in an embodiment of this application. Detailed Implementation

[0085] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0086] This embodiment provides a pose estimation method based on feature points and semantic information, such as Figure 1 As shown, the method includes:

[0087] Step 101: Acquire an image captured by a mobile device and perform semantic segmentation on the image to obtain the semantic information corresponding to the image;

[0088] Step 102: Determine multiple feature points in the image, determine the target descriptor for each feature point, and remove dynamic feature points from the multiple feature points based on semantic information.

[0089] Step 103: Based on the target descriptor, determine the matching point for each feature point to obtain multiple matching pairs, and filter the multiple matching pairs;

[0090] Step 104: Initialize the map and determine the ground points in the initialized map;

[0091] Step 105: Estimate the true scale based on the ground points and use the constant velocity model to determine the pose estimate of the mobile device at the true scale.

[0092] The pose estimation method based on feature points and semantic information provided in this application combines semantic information and the data association between images to better estimate the camera pose.

[0093] First, the camera in this embodiment is mounted on a mobile device, such as a robot or a car. DeepLab v3+ is used to perform semantic segmentation on the image, extracting its semantic information. It can be understood that deeplab-v3+ is a semantic segmentation network that builds upon deeplab-v3, adding a simple and effective decoder to refine the segmentation results.

[0094] Then, multiple ORB feature points are determined and used as visual features required for SLAM odometry calculation. Dynamic feature points are removed by semantic information, and pose estimation is performed using the remaining feature points after removing dynamic feature points. This can improve the localization accuracy of SLAM in dynamic environments, as well as its robustness and stability.

[0095] As can be understood, feature points are representative points in an image that remain unchanged when the image changes, such as through rotation or scaling. ORB features consist of keypoints and descriptors, with the descriptor being BRIEF. In this embodiment, the BRIEF descriptor and the semantic descriptor corresponding to the semantic information can be combined to form the descriptor of the ORB feature points, i.e., the target descriptor. The target descriptor can be used for feature matching.

[0096] During feature matching, a matching point is determined for each feature point based on the target descriptor. Each feature point and its corresponding matching point form a matching pair. A similarity formula is used to determine the success or failure of the semantic descriptor matching, and these pairs are then filtered. In this process, semantic information is utilized to reduce the impact of mismatched feature points on the algorithm's accuracy.

[0097] After extracting feature points, map initialization can be performed, and ground points can be filtered in conjunction with semantic information. First, feature point matching is performed, with nearest neighbor search used for candidate keyframes. Epipolar geometry constraints are used to estimate camera motion based on two sets of 2D points, recovering the camera pose R and t. Then, Singular Value Decomposition (SVD) is performed to obtain the 3D points corresponding to the matched feature point pairs. Epipolar geometry, in this context, refers to a special geometric relationship between two images generated from two camera positions, serving as the fundamental model for 2D-2D solving of camera pose between two frames. The two sets of 2D points represent the 2D pixel coordinates of matched feature point pairs in both frames during monocular SLAM initialization, obtained through feature extraction and matching; hence the term 2D-2D.

[0098] Finally, scale estimation is performed based on ground points, and the ratio of the current map to the real map is calculated. The calculated pose and map points are adjusted to have a realistic scale, providing a realistic scale for subsequent tracking threads. A constant velocity model is used for initial pose estimation, and the Base Algorithm (BA) minimizes reprojection bias, outputting a relatively accurate pose estimate with a realistic scale. Regarding the constant velocity model (TrackWithMotionModel), it can be understood that SLAM has three tracking methods: reference keyframe tracking, constant velocity model tracking, and relocalization tracking. Constant velocity model tracking is used under normal conditions to track the current frame using the previous frame, estimating the pose of the current frame using the pose of the previous frame and the previously estimated velocity.

[0099] The embodiments of this application optimize the extraction of feature points by integrating the results of semantic segmentation, including removing dynamic feature points and performing accurate feature matching through semantic descriptors. The true scale is recovered through the segmented ground feature points, avoiding the disadvantage of scale uncertainty in traditional monocular SLAM. Finally, the inter-frame pose is initially calculated through inter-frame data association, which can output a pose estimate with a true scale relatively accurately.

[0100] Figure 2The figure illustrates a schematic diagram of the implementation process of a pose estimation method based on feature points and semantic information provided in an embodiment of this application. As shown, the method includes three steps: feature point data association, monocular vision initialization and scale estimation, and feature tracking. First, an image captured by a mobile device is input, and the image is semantically segmented. Feature point data association is performed using the image and its semantic segmentation results, sequentially performing image pyramid layering, feature point extraction, removal of dynamic feature points, and feature point matching to obtain matching pairs that meet the requirements. Then, monocular vision initialization and scale estimation are performed, sequentially initializing the pose, triangulating map points, weighted calculating ground point heights, and restoring the true scale based on the camera's installation height. Finally, feature tracking is performed, adjusting the pose according to the true scale and continuously tracking to obtain the pose estimate at the true scale.

[0101] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to fully illustrate the specific implementation process of this embodiment, another pose estimation method based on feature points and semantic information is provided. In this method, a target descriptor for each feature point is determined, including:

[0102] Based on semantic information, determine the semantic descriptor for each feature point, and combine the semantic descriptor with the BRIEF descriptor of the feature point to obtain the target descriptor of the feature point.

[0103] Specifically, the target descriptor is obtained by combining the semantic descriptor and the BRIEF descriptor, and the semantic descriptor can be calculated using the following formula:

[0104]

[0105] Where p(X) i =c) represents feature point X i The probability of belonging to category c, d(S) c ,X i ) represents feature point X i The distance to the nearest semantic contour classified as c. The semantic descriptor is combined with the BRIEF descriptor to form a descriptor for the feature point.

[0106] Optionally, in a pose estimation method based on feature points and semantic information, matching points are determined for each feature point based on the target descriptor, resulting in multiple matching pairs, including:

[0107] Step 201: Based on the current pose of the mobile device and the camera intrinsic parameters, project the feature points to obtain the projected coordinates of the feature points in the current frame.

[0108] Step 202: Provide scale information for feature points through image pyramid, determine the search range based on scale information, and determine the candidate matching points corresponding to feature points within the search range based on semantic descriptors;

[0109] Step 203: Verify the similarity between the candidate matching point and the feature point using the similarity formula. If the verification passes, the candidate matching point becomes the matching point corresponding to the feature point. The feature point and the matching point form a matching pair, where each feature point corresponds to one or more matching points.

[0110] In this embodiment, to achieve feature matching at different scales between image frames, an image pyramid is used to provide scale information for feature points. Based on the pyramid level of a feature point in the previous frame, the pyramid level of its corresponding feature point in the current frame is inferred. The search radius for the feature point is then selected based on the level. Finally, the map point is projected according to the current camera pose and camera intrinsic parameters to obtain its projected coordinates in the current frame's pixel plane. A similarity formula is used to determine the success of semantic descriptor matching. After semantic verification, the feature point with the smallest Hamming distance between BRIEF descriptors is the matching point of the feature point from the previous frame in the current frame.

[0111] Optionally, in a pose estimation method based on feature points and semantic information, multiple matching pairs are filtered, including:

[0112] Step 301: Calculate the distance between the target descriptor corresponding to the feature point and the target descriptor corresponding to the matching point. If the distance is greater than the first preset threshold, then discard the matching pair composed of the feature point and the matching point.

[0113] Step 302: Among the matching points corresponding to the feature points, determine the best matching point and the second best matching point. If the best matching point and the second best matching point do not meet the preset advantage verification condition, then remove the matching pair corresponding to the feature point.

[0114] Step 303: Count the directions of all matching pairs to obtain the direction histogram, determine the rotation direction based on the direction histogram, and remove matching pairs whose directions are inconsistent with the rotation direction.

[0115] In this embodiment, all obtained feature point matches are filtered. The specific process is as follows: First, matching pairs with a distance greater than a first preset threshold are removed. Then, for each feature point, its best matching point and second best matching point are recorded. When the advantage of the best matching point over the second best matching point is not obvious, the matching pair is removed. Finally, the directions of all feature point matching pairs are statistically analyzed to obtain a direction histogram, thereby filtering the directions of the matching pairs and removing matching pairs with inconsistent rotation directions.

[0116] The system allows for setting advantage verification conditions based on actual application scenarios. When the best matching point and the second-best matching point meet the preset advantage verification conditions, the best matching point is considered to have a significant advantage over the second-best matching point; otherwise, the advantage is considered not significant. For example, the following advantage verification condition can be set: if the distance corresponding to the best matching point is less than the distance corresponding to the second-best matching point × 2 × a preset coefficient, where the preset coefficient can be set to 0.9.

[0117] Optionally, in a pose estimation method based on feature points and semantic information, determining ground points in the initialized map includes:

[0118] Step 401: Use nearest neighbor search to determine keyframes and insert them;

[0119] Step 402: Use epipolar set constraints to process the matching pairs between the current frame and keyframes to obtain the pose of the mobile device.

[0120] Step 403: In the initialized map, determine the coordinates of the map points corresponding to the matching pairs based on the matching pairs between the current frame and keyframes and the pose of the mobile device.

[0121] Step 404: Separate the feature points of the ground from the map points through semantic segmentation, and denot them as ground points;

[0122] Step 405: Calculate the horizontal distance between each ground point and the origin of the camera coordinate system corresponding to the mobile device, and remove ground points whose horizontal distance is greater than the second preset threshold.

[0123] In this embodiment, local mapping can be performed, and ground points can be separated through semantic segmentation. Specifically, initial feature point matching is first performed, with the simplest nearest neighbor search used for candidate keyframes. Epipolar geometry constraints are used to estimate camera motion based on two sets of 2D points, recovering the camera pose R and t. Then, SVD is performed to obtain the 3D points corresponding to the matched feature point pairs. Next, ground feature points are separated through semantic segmentation, and the ratio of the current map to the real map is calculated. Specifically, the distance d between all ground points and the origin of the camera coordinate system in the horizontal direction is calculated sequentially, and a second preset threshold of 5a is set. If the distance d is greater than 5a, the ground point is discarded. The z-value of the closest ground points is used to ensure terrain consistency. Therefore, the ground points are segmented into five regions, such as... Figure 3 As shown in the diagram. Closer ground points are assigned higher weights, and finally, the z-values ​​of the remaining ground points are averaged. The specific formula is shown below:

[0124]

[0125] Where, ρ i z represents the weight of each ground point. iThis represents the z-value of each ground point. mean This represents the weighted average of the z-values ​​of ground points.

[0126] Optionally, in a pose estimation method based on feature points and semantic information, the true scale is estimated from ground points, and the pose of the mobile device at the true scale is determined using a constant velocity model, including:

[0127] Step 501: Determine the actual scale based on the height of the ground point and the height of the mobile device;

[0128] Step 502: Adjust the pose of the mobile device and the coordinates of the ground points according to the actual scale;

[0129] Step 503: Input the matching pairs between the current frame and keyframes, as well as the map points, into the constant velocity model, and use the BA method to reduce the reprojection bias to obtain the pose estimation value of the mobile device.

[0130] In this embodiment, according to the aforementioned z mean The formula for calculating the true scale is as follows:

[0131] s = h / z mean

[0132] Where h is the actual height of the camera above the ground, and s is the actual scale.

[0133] Finally, the calculated pose and map points are adjusted to a realistic scale, providing a realistic scale for subsequent tracking threads. For example... Figure 4 As shown, after recovering the true scale from ground points in the KITTI dataset 00, the scale of the monocular SLAM trajectory is close to the true scale.

[0134] Initial pose estimation is performed using a constant velocity model, and the pose is solved using Perspective-n-point (EPnP). Reprojection error will occur during this process. The formula for the reprojection error is as follows:

[0135]

[0136] Continue with Bundle Adjustment to minimize reprojection bias. Iterate the value of T to gradually reduce the reprojection error, ultimately minimizing all reprojection errors to obtain the camera pose corresponding to the current matching pair. Output a relatively accurate pose estimate with true scale.

[0137] Optionally, in a pose estimation method based on feature points and semantic information, backend optimization can also be performed. Specifically, the method further includes the following steps:

[0138] Step 601: Establish an error function based on the semantic category, where the semantic category is either road surface or pole;

[0139] Step 602: Insert keyframes using a method that tracks the number of shared feature points;

[0140] Step 603: Construct a likelihood model, which is used for map initialization;

[0141] Step 604: Adjust the likelihood model according to the error function.

[0142] In this embodiment, error functions are established using several semantic categories (road surface and poles) to further suppress pose estimation drift. Since feature points are low-level features affected by changes in viewpoint, scale, and lighting, only short-term constraints can be constructed in the back-end optimization algorithm, making it difficult to suppress long-term cumulative drift. Therefore, this chapter constructs medium-term constraints based on fused semantic information that is almost unaffected by the above factors, integrating semantic constraints and feature point constraints into pose and map optimization, thereby effectively suppressing the cumulative drift of visual odometry. For semantic constraints, a general semantic constraint is first constructed using the results of semantic segmentation, and then semantic error functions are established separately using the special properties of several semantic categories (road surface and poles) to more fully utilize semantic information and further suppress drift. Finally, the error function for each map point is established as shown in the following formula.

[0143]

[0144] Where E represents the overall error of the map points, and λ represents the weight of the semantic error, the value of which is adjusted according to the actual effect of semantic segmentation. seg_normal E Pole and E Ground These are semantic errors originating from general semantics, semantic errors originating from poles, and semantic errors originating from the ground, respectively. true For the correct observation of the feature point corresponding to P in the pixel plane, class represents the semantic information contained in the map point.

[0145] Keyframe insertion employs a mechanism that tracks the number of shared feature points, inserting a new keyframe only when leaving the current scene, thus avoiding keyframe redundancy. A likelihood model p(S) is defined using a probabilistic approach. k |T k ,X i Z i =c), semantic observation S k Current pose T k and map point P i Connecting them. Semantic point observation p(S) k |T k ,X i Zi =c) The probability should vary with the projection position π(T) k ,X i The likelihood decreases as the geometric distance to the nearest region contour labeled as semantic class c increases. The formula for the likelihood model is shown below:

[0146]

[0147] Based on the unique properties of the ground, ground point constraints are introduced to correct the scale in real time. For points whose semantic information is the ground, the error function is:

[0148] E=(1-λ)E base +λ[(1-γ)E seg_normal +γE Ground ]

[0149] Where λ represents the proportion of ground points constrained in height, the proportions from the first to the fifth layer are 75%, 60%, 45%, 30% and 15% respectively, and λ is zero for the remaining ground points.

[0150] In the real world, poles are typically approximately perpendicular to the ground and have a slender shape. By creating line constraints using poles, a more reasonable error function is established for map points whose semantic information is poles. For points whose semantic information is poles, the error function is:

[0151] E=λE Pole +(1-λ)E Base

[0152] The above formula is used for calculation, thus providing better initial values ​​for optimization.

[0153] Optionally, a pose estimation method based on feature points and semantic information may further include the following steps:

[0154] Step 701: Build a graph structure using the current keyframe and the co-view keyframes corresponding to the current keyframe, and optimize the graph structure using the g2o optimization library;

[0155] Step 702: Update the pose of the mobile device and the coordinates of map points using the optimized graph structure.

[0156] In this embodiment, to suppress the impact of cumulative drift, the back-end optimization algorithm establishes constraints among multiple image observations to jointly optimize the pose and map. An optimized graph structure is established based on g2o to solve the back-end optimization problem. The specific process is as follows:

[0157] 1) Add the initial poses of the current keyframe and its co-view keyframes as non-fixed (fixed initial frame) nodes to the graph optimization. The co-view keyframe is obtained by the number of jointly observed map points; that is, when the number of jointly observed map points is greater than a certain threshold, it is considered a co-view keyframe.

[0158] 2) Add the map point coordinates of the current keyframe and its co-view keyframes as non-fixed nodes to the graph optimization.

[0159] 3) The initial poses of the remaining keyframes of the map points in 2) that can be observed are added as fixed nodes to the graph optimization.

[0160] 4) Add edges to connect the map point nodes in 2) to the pose of the observation keyframe of that map point to optimize the reprojection error (E). Base ).

[0161] 5) If the map point projected back to the common-view keyframe does not exceed the image range and satisfies the correct optimization direction described in Section 3.2.1, then add an edge to connect the map point and the pose of the keyframe to construct a general semantic constraint. Note that if an edge with excessive projection error has already been added between the map point and the keyframe, then it is no longer necessary to add such an edge.

[0162] 6) If the current motion is stable and the map point is projected back to the common view keyframe without exceeding the image range, add an edge to connect the map point and the pose of the keyframe to construct ground point-based constraints.

[0163] 7) If the map point is projected back to the common view keyframe without exceeding the image range, add an edge to connect the map point and the pose of the keyframe to construct a rod-based constraint.

[0164] 8) The final constructed graph structure is as follows: Figure 5 As shown, iterative optimization begins. After each iteration, the keyframe pose and map point coordinates are updated.

[0165] A graph structure was established by combining the various constraints and optimized using the g2o optimization library, thereby effectively suppressing the cumulative drift caused by the robot's long-term motion.

[0166] This application employs a SLAM backend optimization algorithm that integrates semantic information. Mid-term semantic constraints are established based on semantic information, including general semantic constraints and semantic constraints specifically for the semantic categories of ground and pole. A graph structure is built by combining these constraints and optimized using the g2o optimization library, thereby effectively suppressing the cumulative drift caused by the robot's long-term motion.

[0167] Furthermore, as a specific implementation of the above-mentioned pose estimation method based on feature points and semantic information, embodiments of this application provide a pose estimation device based on feature points and semantic information, such as... Figure 6 As shown, the device includes: a semantic segmentation module, a feature point extraction module, a matching module, and a pose estimation module.

[0168] The semantic segmentation module is used to acquire images captured by mobile devices and perform semantic segmentation on the images to obtain the semantic information corresponding to the images;

[0169] The feature point extraction module is used to identify multiple feature points in an image, determine the target descriptor for each feature point, and remove dynamic feature points from the multiple feature points based on semantic information.

[0170] The matching module is used to determine matching points for each feature point based on the target descriptor, obtain multiple matching pairs, and filter the multiple matching pairs;

[0171] The pose estimation module is used to initialize the map and determine the ground points in the initialized map; and to estimate the real scale based on the ground points and use a constant velocity model to determine the pose estimate of the mobile device at the real scale.

[0172] Optionally, the feature point extraction module is used for:

[0173] Based on semantic information, determine the semantic descriptor for each feature point, and combine the semantic descriptor with the BRIEF descriptor of the feature point to obtain the target descriptor of the feature point.

[0174] Optionally, the matching module is used for:

[0175] Based on the current pose of the mobile device and the camera intrinsic parameters, the feature points are projected to obtain the projected coordinates of the feature points in the current frame.

[0176] The image pyramid provides scale information for feature points, and the search range is determined based on the scale information. Within the search range, the candidate matching points corresponding to the feature points are determined based on the semantic descriptors.

[0177] The similarity between candidate matching points and feature points is verified using a similarity formula. If the verification passes, the candidate matching point becomes the matching point corresponding to the feature point. The feature point and the matching point form a matching pair, where each feature point corresponds to one or more matching points.

[0178] Optionally, the matching module is used for:

[0179] Calculate the distance between the target descriptor corresponding to the feature point and the target descriptor corresponding to the matching point. If the distance is greater than the first preset threshold, then discard the matching pair consisting of the feature point and the matching point.

[0180] Among the matching points corresponding to the feature points, the best matching point and the second best matching point are determined. If the best matching point and the second best matching point do not meet the preset advantage verification condition, the matching pair corresponding to the feature point is removed.

[0181] The orientations of all matching pairs are counted to obtain an orientation histogram. The rotation direction is determined based on the orientation histogram, and matching pairs whose orientations do not match the rotation direction are removed.

[0182] Optionally, the pose estimation module is used for:

[0183] Nearest neighbor search is used to determine keyframes, and then keyframes are inserted.

[0184] The pose of the mobile device is obtained by processing the matching pairs between the current frame and keyframes using epipolar set constraints.

[0185] In the initialized map, the coordinates of the map points corresponding to the matching pairs are determined based on the matching pairs between the current frame and keyframes, as well as the pose of the mobile device.

[0186] By semantic segmentation, feature points on the ground are separated from map points and denoted as ground points;

[0187] Calculate the horizontal distance between each ground point and the origin of the camera coordinate system corresponding to the mobile device, and remove ground points whose horizontal distance is greater than the second preset threshold.

[0188] Optionally, the pose estimation module is used for:

[0189] Determine the actual scale based on the height of the ground points and the height of the mobile equipment;

[0190] Adjust the pose of the mobile device and the coordinates of the ground points according to the actual scale;

[0191] The matching pairs between the current frame and keyframes, as well as map points, are input into the constant velocity model. The BA method is used to reduce reprojection bias, and the pose estimate of the mobile device is obtained.

[0192] Optionally, the device also includes a back-end optimization module for:

[0193] An error function is established based on semantic categories, where the semantic categories are road surface or pole.

[0194] Keyframes are inserted by tracking the number of shared feature points;

[0195] Construct a likelihood model, which is used for map initialization;

[0196] Adjust the likelihood model based on the error function;

[0197] And / or,

[0198] Build a graph structure using the current keyframe and the corresponding co-view keyframe, and optimize the graph structure using the g2o optimization library;

[0199] The optimized graph structure is used to update the pose of the mobile device and the coordinates of map points.

[0200] It should be noted that other corresponding descriptions of the functional modules involved in the pose estimation device based on feature points and semantic information provided in this application embodiment can be found in the corresponding descriptions in the above method, and will not be repeated here.

[0201] Based on the above method, the present application also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described pose estimation method based on feature points and semantic information.

[0202] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause an electronic device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0203] Based on the above, Figures 1 to 5 The method shown, and Figure 6 To achieve the above objectives, the present application also provides a device, specifically a personal computer, server, network device, etc., as shown in the virtual device embodiment. This electronic device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figures 1 to 5 The pose estimation method shown is based on feature points and semantic information.

[0204] Optionally, the electronic device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0205] Those skilled in the art will understand that the electronic device structure provided in this embodiment does not constitute a limitation on the electronic device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0206] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of the electronic device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between various components within the storage medium, as well as communication with other hardware and software within the physical device.

[0207] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware.

[0208] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the units or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the units in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be located in one or more apparatuses different from this embodiment, with corresponding changes. The units of the above-described embodiment can be combined into one unit, or further divided into multiple sub-units.

[0209] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A pose estimation method based on feature points and semantic information, characterized in that, The method includes: Acquire images captured by a mobile device, and perform semantic segmentation on the images to obtain the semantic information corresponding to the images; Multiple feature points are identified in the image. Based on the semantic information, a semantic descriptor for each feature point is determined. The semantic descriptor and the BRIEF descriptor of the feature point are combined to obtain a target descriptor for each feature point. Dynamic feature points are then removed from the multiple feature points based on the semantic information. Based on the target descriptor, a matching point is determined for each feature point to obtain multiple matching pairs, and the multiple matching pairs are then filtered. Map initialization is performed, and keyframes are determined and inserted using nearest neighbor search. Epipolar set constraints are used to process matching pairs between the current frame and keyframes to obtain the pose of the mobile device. In the initialized map, the coordinates of map points corresponding to the matching pairs are determined based on the current frame, keyframes, and the pose of the mobile device. Through semantic segmentation, ground feature points are separated from the map points and denoted as ground points. The horizontal distance between each ground point and the origin of the camera coordinate system corresponding to the mobile device is calculated, and ground points with a horizontal distance greater than a second preset threshold are removed. The true scale is estimated based on the ground points, and the pose estimate of the mobile device at the true scale is determined using a constant velocity model.

2. The method according to claim 1, characterized in that, Based on the target descriptor, a matching point is determined for each feature point to obtain multiple matching pairs, including: Based on the current pose of the mobile device and the camera intrinsic parameters, the feature points are projected to obtain the projected coordinates of the feature points in the current frame. The feature points are provided with scale information through an image pyramid, and the search range is determined based on the scale information. Within the search range, the candidate matching points corresponding to the feature points are determined based on the semantic descriptor. The similarity between the candidate matching point and the feature point is verified by a similarity formula. If the verification passes, the candidate matching point is the matching point corresponding to the feature point. The feature point and the matching point form a matching pair, wherein each feature point corresponds to one or more matching points.

3. The method according to claim 2, characterized in that, The filtering of the plurality of matching pairs includes: Calculate the distance between the target descriptor corresponding to the feature point and the target descriptor corresponding to the matching point. If the distance is greater than a first preset threshold, then discard the matching pair consisting of the feature point and the matching point. Among the matching points corresponding to the feature points, the best matching point and the second best matching point are determined. If the best matching point and the second best matching point do not meet the preset advantage verification condition, the matching pair corresponding to the feature point is removed. The directions of all matching pairs are counted to obtain a direction histogram. The rotation direction is determined based on the direction histogram, and matching pairs whose directions are inconsistent with the rotation direction are removed.

4. The method according to claim 1, characterized in that, The step of estimating the true scale based on the ground points and determining the pose of the mobile device at the true scale using a constant velocity model includes: The actual scale is determined based on the height of the ground point and the height of the mobile device; The pose of the mobile device and the coordinates of the ground point are adjusted according to the actual scale. The matching pairs between the current frame and keyframes, as well as the map points, are input into the constant velocity model, and the BA method is used to reduce the reprojection bias to obtain the pose estimate of the mobile device.

5. The method according to claim 1, characterized in that, The method further includes: An error function is established based on the semantic category, where the semantic category is either road surface or pole. Keyframes are inserted by tracking the number of shared feature points; Construct a likelihood model, wherein the likelihood model is used for map initialization; Adjust the likelihood model based on the error function; And / or, A graph structure is established using the current keyframe and the corresponding co-view keyframe, and the graph structure is optimized using the g2o optimization library; The pose of the mobile device and the coordinates of the map points are updated using the optimized graph structure.

6. A pose estimation device based on feature points and semantic information, characterized in that, The device includes: The semantic segmentation module is used to acquire images captured by a mobile device and perform semantic segmentation on the images to obtain the semantic information corresponding to the images; The feature point extraction module is used to determine multiple feature points in the image, determine a semantic descriptor for each feature point based on the semantic information, and combine the semantic descriptor and the BRIEF descriptor of the feature point to obtain a target descriptor for each feature point; and remove dynamic feature points from the multiple feature points based on the semantic information. The matching module is used to determine a matching point for each feature point based on the target descriptor, obtain multiple matching pairs, and filter the multiple matching pairs; The pose estimation module is used to initialize the map, determine keyframes using nearest neighbor search, and insert the keyframes; process the matching pairs between the current frame and keyframes using epipolar set constraints to obtain the pose of the mobile device; in the initialized map, determine the coordinates of the map points corresponding to the matching pairs based on the current frame, keyframes, and the pose of the mobile device; separate ground feature points from the map points through semantic segmentation, denoted as ground points, and calculate the horizontal distance between each ground point and the origin of the camera coordinate system corresponding to the mobile device, discarding ground points whose horizontal distance is greater than a second preset threshold; and estimate the true scale based on the ground points, and determine the pose estimate of the mobile device at the true scale using a constant velocity model.

7. A storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the method as described in any one of claims 1 to 5.

8. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the method of any one of claims 1 to 5.

Images