Monocular vision positioning method based on point-line characteristics, chip and robot

By extracting and combining monocular visual positioning methods of point and line features, using SIFT descriptor and KD tree database, the problem of insufficient positioning accuracy in low texture environments is solved, efficient pose estimation and optimization are achieved, and calculations are reduced Complexity.

CN119027496APending Publication Date: 2024-11-26AMICRO SEMICONDUCTOR CO LTD

Patent Information

Application Number
CN202310588500.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing visual SLAM algorithm based on point features has the problem of insufficient positioning accuracy in low texture environments and light source transformation scenarios, and the calculation complexity is high, making it difficult to effectively combine point features and line features for pose estimation.

Method used

Using a monocular visual positioning method based on point-line features, point features and line features are extracted and line features are characterized by the same SIFT descriptor, and the candidate keyframe updates and selections are performed by combining the KD tree database, and the PnP algorithm and random sampling consistency algorithm are used for Position estimation and optimization through least squares method model, the precise calculation of the pose is finally realized.

Benefits of technology

Improve positioning accuracy and reduce calculation complexity in low-texture environments and light source transformation scenarios. By combining point features and line features, we can effectively make up for the limitations of point features and improve the accuracy and efficiency of pose estimation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027496A_ABST
    Figure CN119027496A_ABST
Patent Text Reader

Abstract

The invention provides a monocular vision positioning method based on point-line features, a chip and a robot. The monocular vision positioning method based on the point-line features comprises the following steps: extracting point features and line features corresponding to a current frame visual image from the current frame visual image; representing the point features and the line features corresponding to the current frame visual image by using the same SIFT descriptors; on the basis of a KD tree database corresponding to the key frame visual image, updating and selecting candidate key frame visual images in combination with the current frame visual image; performing pose estimation on the current frame visual image based on the candidate key frame visual image to obtain a first pose corresponding to the current frame visual image; and performing pose optimization on the first pose based on the point feature and the line feature corresponding to the current frame visual image and the point feature and the line feature corresponding to the candidate key frame visual image to obtain a final pose corresponding to the current frame visual image. The positioning precision is improved by combining the point features and the line features, and the application scene is expanded at low cost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to a monocular vision positioning method, chip and robot based on point and line features. Background Art

[0002] Visual SLAM technology based on point features in visual images has always been a hot research topic in the field of simultaneous positioning and mapping. The SIFT algorithm uses feature points on visual images as a basis to perform feature tracking, matching, positioning, etc. in real time, and can complete the entire process of simultaneous positioning and mapping. It is one of the most advanced positioning algorithms in the field of visual SLAM. It has stable features, invariance to rotation, scale transformation, and brightness, and a certain degree of stability to perspective transformation and noise. However, the SIFT algorithm has poor real-time performance and poor feature point extraction capabilities for targets with smooth edges. Especially in low-texture environments and scenes with light source changes, the small number of effective feature points seriously affects the visual positioning capability. Compared with point features, line features are more stable, so SLAM algorithms based on line features have also attracted much attention. At present, the method of combining point features and line features for visual positioning has the problems of high computational complexity and poor posture accuracy. Summary of the invention

[0003] This application provides a monocular vision positioning method, chip and robot based on point and line features. The specific technical solution is as follows:

[0004] A monocular visual positioning method based on point and line features specifically includes: extracting point features and line features corresponding to the current frame visual image from the current frame visual image; using the same SIFT descriptor to represent the point features and line features corresponding to the current frame visual image; updating and selecting candidate key frame visual images based on a KD tree database corresponding to the key frame visual image in combination with the current frame visual image; estimating the pose of the current frame visual image based on the candidate key frame visual image to obtain the first pose corresponding to the current frame visual image; optimizing the pose of the first pose based on the point features and line features corresponding to the current frame visual image and the point features and line features corresponding to the candidate key frame visual image to obtain the final pose corresponding to the current frame visual image.

[0005] Furthermore, the extracting of point features and line features corresponding to the current frame visual image from the current frame visual image specifically includes: establishing a Gaussian pyramid for the current frame visual image; using a SIFT algorithm to detect and extract point features in the current frame visual image from the Gaussian pyramid corresponding to the current frame visual image, and using a SIFT descriptor to characterize the point features corresponding to the current frame visual image; using an LSD algorithm to detect and extract line features in the current frame visual image from the Gaussian pyramid corresponding to the current frame visual image, and using an LBD descriptor to characterize the line features corresponding to the current frame visual image.

[0006] Furthermore, the method of using the same SIFT descriptor to represent the point features and line features corresponding to the current frame visual image specifically includes: extracting the mean part of the line features corresponding to the current frame visual image represented by the LBD descriptor as the SIFT descriptor of the line features corresponding to the current frame visual image, so that the point features and line features corresponding to the current frame visual image are represented by the same SIFT descriptor.

[0007] Furthermore, the monocular vision positioning method based on point and line features also includes: when the line feature corresponding to the current frame visual image is stored, the pixel coordinates and normalized plane coordinates of the starting end point and the ending end point of the line feature in the current frame visual image are recorded.

[0008] Furthermore, the KD tree database corresponding to the key frame visual image includes: a first KD tree database recording all corresponding point features in all historical key frame visual images and a second KD tree database recording all corresponding line features in all historical key frame visual images.

[0009] Furthermore, the KD tree database corresponding to the key-frame visual image is combined with the current frame visual image to update and select the candidate key-frame visual image, specifically including: performing KD tree searches in the first KD tree database based on all point features corresponding to the current frame visual image, so as to match the closest key-frame feature points respectively; performing KD tree searches in the second KD tree database based on all line features corresponding to the current frame visual image, so as to match the closest key-frame feature lines respectively; traversing and counting the number of key-frame feature points and key-frame feature lines that are closest to the current frame visual image in each frame of key-frame visual images; based on the number of key-frame feature points and key-frame feature lines that are closest to the current frame visual image in each frame of key-frame visual images, selecting a frame of key-frame visual image from all key-frame visual images to update and select it as a candidate key-frame visual image.

[0010] Furthermore, based on the number of key frame feature points and key frame feature lines in each key frame visual image that are closest to the current frame visual image, a key frame visual image is selected from all key frame visual images to be updated and selected as a candidate key frame visual image, specifically including: traversing and calculating the product of the number of key frame feature points in each key frame visual image that are closest to the current frame visual image and a first weight, as the first product corresponding to each key frame visual image; traversing and calculating the product of the number of key frame feature lines in each key frame visual image that are closest to the current frame visual image and a second weight, as the second product corresponding to each key frame visual image; traversing and calculating the sum of the first product and the second product corresponding to each key frame visual image as the candidate key frame selection information corresponding to each key frame visual image; traversing and comparing the candidate key frame selection value corresponding to each key frame visual image, and selecting a key frame visual image with the largest candidate key frame selection value from all key frame visual images to be updated as the candidate key frame visual image; wherein the second weight is greater than the first weight.

[0011] Furthermore, a key frame visual image is selected from all key frame visual images and updated as a candidate key frame visual image, which also includes: constructing a third KD tree database that records all corresponding point features in the one frame of candidate key frame visual image based on the updated selected candidate key frame visual image; and constructing a fourth KD tree database that records all corresponding line features in the one frame of candidate key frame visual image based on the updated selected candidate key frame visual image.

[0012] Furthermore, the pose estimation of the current frame visual image based on the candidate key frame visual image to obtain the first pose corresponding to the current frame visual image specifically includes: traversing all the corresponding point features in the current frame visual image in the third KD tree database to perform KD tree search to find the first matching feature point pairs respectively; traversing all the corresponding line features in the current frame visual image in the fourth KD tree database to perform KD tree search to find the first matching feature line pairs respectively; using a random sampling consistency algorithm to perform a first mismatch feature filtering on the obtained first matching feature point pairs and first matching feature line pairs to obtain first valid matching feature point pairs and first valid matching feature line pairs; using a PnP algorithm to perform pose estimation on the first valid matching feature point pairs and the first valid matching feature line pairs to obtain an initial estimated pose; and using reprojection error calculation to further optimize the initial estimated pose to obtain the first pose corresponding to the current frame visual image.

[0013] Furthermore, the use of the PnP algorithm to perform pose estimation on the first effective matching feature point pair and the first effective matching line pair specifically includes: taking the starting endpoint pair corresponding to each pair of effective matching line pairs in the first effective matching line pair as a supplementary effective matching feature point pair; and using the PnP algorithm to perform pose estimation on the first effective matching feature point pair and the supplementary effective matching feature point pair.

[0014] Furthermore, the pose optimization of the first pose based on the point features and line features corresponding to the current frame visual image and the point features and line features corresponding to the candidate key frame visual image to obtain the final pose corresponding to the current frame visual image specifically includes: using the SIFT feature matching algorithm to match the corresponding point features and line features in the current frame visual image with the corresponding point features and line features in the candidate key frame visual image, respectively, to obtain second matching feature point pairs and second matching feature line pairs; calculating the reprojection error between the current frame visual image and the candidate key frame visual image based on the first pose, the second matching feature point pairs and the second matching feature line pairs; performing second mismatch feature filtering on the second matching feature point pairs and second matching feature line pairs obtained based on the reprojection error between the current frame visual image and the candidate key frame visual image, to obtain second valid matching feature point pairs and second valid matching feature line pairs; constructing a least squares model based on the reprojection error between the current frame visual image and the candidate key frame visual image; and iteratively solving using the LM algorithm based on the least squares model, the second valid matching feature point pairs and the second valid matching feature line pairs to obtain the final pose corresponding to the current frame visual image.

[0015] The present application also discloses a chip having a computer program stored therein. When the computer program stored therein is executed by a processor, the monocular vision positioning method based on point and line features as described above is executed.

[0016] The present application also discloses a robot, comprising: a monocular camera, a chip, and a processor; wherein the monocular camera is used to collect visual images; the processor is used to run the chip; when the chip is run by the processor, the monocular visual positioning method based on point and line features as described above is executed based on the visual image collected by the monocular camera to obtain the robot's posture based on the visual image.

[0017] The present application discloses a monocular vision positioning method, chip and robot based on point and line features. By combining point features and line features, line features can be used as supplementary features in low-texture environments or light source change scenes where effective point feature extraction is limited, thereby making up for the limitation of insufficient point features and effectively improving positioning accuracy. Point features and line features are represented by the same SIFT descriptor, so that the same code and architecture can be used for feature storage and matching of point features and line features, thereby reducing computational complexity and realizing positioning calculation at low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic flow chart of a monocular vision positioning method based on point and line features according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to more clearly illustrate the present invention, specific embodiments are given below for further explanation. It should be understood that when used in the present application, the term "comprising" indicates the existence of described features, integral bodies, steps, operations, elements and / or components, but does not exclude the existence or addition of one or more other features, integral bodies, steps, operations, elements, components and / or their sets. It should also be understood that the term "and / or" used in the present application refers to any combination and all possible combinations of one or more of the items listed in association, and includes these combinations.

[0020] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. The reference to "one embodiment" or "some embodiments" described in the specification of the present application means that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in the differences in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0021] In one embodiment of the present application, a monocular vision positioning method based on point and line features is provided, which aims to solve the limitations of scene positioning caused by insufficient point features due to low texture and light source changes, and combines point features and line features to improve the monocular positioning posture accuracy.

[0022] Specifically, Figure 1 As shown, the monocular vision positioning method based on point and line features includes:

[0023] Step 1: Extract point features and line features corresponding to the current frame visual image from the current frame visual image; wherein the point features include object edge points, corner points, line intersection points, etc., and the line features refer to line segment features in the image, that is, a very narrow line segment area between two adjacent boundaries in the image, and the pixels in the area have similar or identical grayscale values; the point feature extraction method may be but is not limited to curvature-based extraction method, grayscale-based extraction method, edge-based extraction method, etc.; the line feature extraction method may be but is not limited to line feature detection algorithm, differential operator, gradient operator, directional differential operator, etc.

[0024] Step 2: Use the same SIFT descriptor to represent the point features and line features corresponding to the current frame visual image. In this step, the point features and line features are represented by the same SIFT descriptor so that the feature storage and matching of the point features and line features can use the same code and architecture, thereby reducing the computational complexity and achieving low-cost positioning calculations.

[0025] Step 3: Update and select candidate key frame visual images based on the KD tree database corresponding to the key frame visual image and the current frame visual image; wherein the KD tree database corresponding to the key frame visual image is used to record and store all corresponding point features and all line features in the key frame visual image, and the KD tree database can be understood as a visual dictionary for comparison and matching with the current frame visual image. It should be noted that the update selection rule of the key frame visual image described in this application is to select a key frame visual image with the highest matching degree with the current frame visual image from all key frame visual images.

[0026] Step 4: perform pose estimation on the current frame visual image based on the candidate key frame visual image to obtain the first pose corresponding to the current frame visual image; specifically, since the candidate key frame visual image is a key frame visual image with the highest matching degree with the current frame visual image, the pose corresponding to the candidate key frame visual image can be combined with the matching error between the two to achieve a preliminary pose estimation for the current frame visual image, and obtain the first pose corresponding to the current frame visual image with average accuracy. It should be noted that the key frame visual image is output through a sliding window optimization method, and the pose corresponding to the key frame visual image, all the corresponding point feature information and all the line feature information in the key frame visual image can be obtained through the sliding window optimization method, and the corresponding point feature information in the key frame visual image at least includes: pixel coordinates, normalized plane coordinates and depth information of the point; the corresponding line feature information in the key frame visual image at least includes: pixel coordinates of the endpoints of the line feature, normalized plane coordinates of the endpoints of the line feature, and Plücker coordinates of the line feature and its orthogonal representation.

[0027] Step 5: Optimize the first pose based on the point features and line features corresponding to the current frame visual image and the point features and line features corresponding to the candidate key frame visual image to obtain the final pose corresponding to the current frame visual image; This step further optimizes the first pose based on the candidate key frame, and performs multi-dimensional synchronous optimization on both point features and line features, greatly improving the pose optimization accuracy, and effectively making up for the limitations of point feature extraction in low-texture environments or light source change scenes.

[0028] As one implementation manner, step 1 of extracting point features and line features corresponding to the current frame visual image from the current frame visual image specifically includes:

[0029] Step 1.1: Establish a Gaussian pyramid for the visual image of the current frame; specifically, the construction process of the Gaussian pyramid includes: doubling the visual image of the current frame as the first group of first layer images of the Gaussian pyramid, and performing Gaussian convolution on the first group of first layer images as the first group of second layer images; using the smoothing factor corresponding to the first group of second layer images to perform a smoothing process on the first group of second layer images to obtain the first group of third layer images; cyclically using the corresponding smoothing factor to perform a smoothing process until the first group obtains L layer images; in the same group of the Gaussian pyramid, the size of each image is the same, but the smoothing coefficient is different. Downsample the third-to-last layer image of the first group with a scale factor of 2, that is, the size is halved, and the obtained image is used as the second group of first layer images of the Gaussian pyramid; similarly, using the corresponding smoothing factor to perform a smoothing process on the second group of first layer images to obtain the second group of second layer images; cyclically using the corresponding smoothing factor to perform a smoothing process until the second group obtains the L layer image. Repeat the above steps until O groups of L layers of images are obtained to form a Gaussian pyramid. The characteristics of the Gaussian pyramid are: in the same group, the sizes of different layers of images are the same, and the smoothing factor corresponding to the next layer of images is k times the smoothing factor corresponding to the previous image; in different groups, the size of the first image in the next group is half the size of the third to last image in the previous group. Based on the changes in image blur at different scales in the Gaussian pyramid, it is possible to simulate the visual process of the camera when the distance from the target changes from near to far.

[0030] Step 1.2: Use the SIFT algorithm to detect and extract point features in the current frame visual image from the Gaussian pyramid corresponding to the current frame visual image, and use the SIFT descriptor to characterize the point features corresponding to the current frame visual image; wherein, the SIFT algorithm refers to the Scale Invariant Feature Transform (SIFT) algorithm, which is used to extract local features of the visual image, find extreme points in the scale space, and extract the position, scale, direction and other information of the points, and use this information to describe the feature points. Specifically, the SIFT algorithm detects point features by using a neighborhood processing method, comparing each pixel point with all its adjacent points to determine the size of the image domain and scale domain of the pixel point compared with the adjacent pixel points, thereby detecting extreme points, and then screening the extreme points to remove unstable and incorrectly detected extreme points to ensure the scale invariance and accuracy of the extracted point features.

[0031] Step 1.3: Use the LSD algorithm to detect and extract the line features in the current frame visual image from the Gaussian pyramid corresponding to the current frame visual image, and use the LBD descriptor to characterize the line features corresponding to the current frame visual image; wherein, the LSD algorithm refers to the line segment detection algorithm (LineSegmentDetector, LSD), which is used to obtain sub-pixel level precision detection results in linear time, and obtain high-precision straight line detection results by merging pixels with similar gradients. The specific detection method is to determine the length of the baseline by the angle between the pixel point and the baseline in the image, and then use the region growing algorithm to merge the approximately consistent pixel areas to obtain the support domain. Each line segment support domain corresponds to at least one line segment. The density of the pixels in the support domain determines whether to divide the support domain into multiple rectangular areas. Finally, all the generated rectangular support domains are improved and screened, and the rectangular support domains that meet the conditions are retained as line feature detection results. This step uses the LSD algorithm with the advantages of fast calculation speed, the extracted line segments have directions, can reduce the mismatch rate during image matching, good real-time performance and high precision, so as to achieve high-precision line feature extraction. It should be noted that the execution order of the above steps 1.2 and 1.3 may be, but is not limited to, first executing step 1.2 and then executing step 1.3, or first executing step 1.3 and then executing step 1.2, or synchronously executing step 1.2 and step 1.3.

[0032] The point features and line features obtained by the SIFT algorithm and LSD algorithm used in this embodiment have rotation invariance, scale invariance, brightness invariance, and a certain degree of stability to perspective change, noise, etc. Among them, since the line feature has a direction and is composed of continuous points with consistent gradients, the line feature has rotation invariance like the point feature; since both the line feature and the point feature are extracted based on the Gaussian pyramid visual image, and the scale invariance and stability to noise are both achieved based on the Gaussian pyramid, the line feature and the point feature have scale invariance and stability to noise; since the point feature achieves brightness invariance based on the normalization of the descriptor amplitude, and the line feature also normalizes the mean part of the LBD descriptor, the line feature and the point feature have brightness invariance. It can be understood that the characteristics of the point features and line features obtained by the SIFT algorithm and the LSD algorithm used in this embodiment are basically the same, and the combination of the two algorithms will not lead to a decrease in performance, but will bring about the beneficial effect of improving performance.

[0033] Preferably, the SIFT algorithm and the LSD algorithm can be implemented by the OpenCV open source algorithm, so for the same visual image, the two algorithms can share the same Gaussian pyramid. Before the execution of step 1.2 and step 1.3, the Gaussian pyramid is first calculated based on the visual image of the current frame, and then the corresponding point features and line features are extracted in step 1.2 and step 1.3 based on the same Gaussian pyramid. In this embodiment, the extraction of line features and point features can share the same Gaussian pyramid, optimize the image processing steps, and improve the extraction efficiency of point features and line features.

[0034] As one of the implementation methods, step 2 uses the same SIFT descriptor to represent the point features and line features corresponding to the visual image of the current frame, specifically including: extracting the mean part of the line features corresponding to the visual image of the current frame represented by the LBD descriptor as the SIFT descriptor of the line features corresponding to the visual image of the current frame, so that the point features and line features corresponding to the visual image of the current frame are represented by the same SIFT descriptor; wherein the LBD descriptor refers to (The Line Band Descriptor), which represents the line features by strips, is more robust to small position changes in the direction of the line features, most of the content of the image in the strip remains unchanged, only the strip boundary changes slightly, and since there is no overlap between each strip in the direction perpendicular to the line feature, the strip calculation efficiency is higher, and the Gaussian weight can be directly applied to each row of strips instead of each pixel. Specifically, strips are usually used to represent the line segment support domain (i.e., line features) in the visual image, and a line segment support domain is divided into a group of strips, each strip is parallel to each other, the strip length is equal to the line segment length, and the descriptor of each strip is calculated by the strip descriptors of its two adjacent rows. The stripe descriptor is composed of the mean variance and standard deviation of the stripe description matrix, such as: LBD = (M1 T , S1 T , M2 T , S2 T , ..., M m T , S m T ) T ; Where LBD is the strip descriptor, M m is the mean vector of the strip description matrix, S m is the standard deviation of the strip description matrix, and m represents the number of strip rows; in this embodiment, extracting the mean part of the LBD descriptor of the line feature actually refers to extracting the mean variance of the strip description matrix from the strip descriptor. In this embodiment, the mean part of the line feature is extracted and represented by the SIFT descriptor, which is equivalent to using the mean part of the line feature to compensate for the point feature, so that the feature storage and matching of the point feature and the line feature can use the same code and architecture, reduce the computational complexity, and realize the positioning calculation at a low cost.

[0035] As one of the implementation methods, the monocular visual positioning method based on point and line features also includes: when the line feature corresponding to the current frame visual image is stored, the pixel coordinates and normalized plane coordinates of the starting endpoint and the ending endpoint of the line feature in the current frame visual image are recorded. This implementation stores and records the coordinate information corresponding to the starting endpoint and the ending endpoint of the line feature, so that the endpoint features in the line feature can be taken into consideration as point features. When the number of point features is insufficient, the endpoints of the line features can be used as supplementary point features. Since the starting endpoint of the line feature is the seed point with the largest gradient in the line feature, the matching accuracy is higher, so it has the characteristics of matching as a point feature, which can effectively solve the problem of difficulty in extracting enough point features in low-texture environments and environments with large light source changes, and expand the applicable environmental scenarios of the positioning method.

[0036] As one of the implementation modes, the KD tree database corresponding to the key frame visual image includes: a first KD tree database that records all the corresponding point features in all historical key frame visual images and a second KD tree database that records all the corresponding line features in all historical key frame visual images. The KD tree database is a tree data structure that stores instance points in a K-dimensional space for fast retrieval. Based on the KD tree database, K nearest neighbor search can be implemented to improve the matching efficiency of point and line features. The KD tree database provided in this implementation mode divides point features and line features into two groups of KD tree databases, so that the KD tree search process can perform targeted search and matching for point features and line features, thereby improving matching accuracy.

[0037] As one implementation method, the updating and selecting of the candidate key frame visual images based on the KD tree database corresponding to the key frame visual image in combination with the current frame visual image in step 3 specifically includes:

[0038] Step 3.1: Based on all the point features corresponding to the current frame visual image, KD tree searches are performed in the first KD tree database respectively to match the closest key frame feature points respectively; specifically, the method for matching the closest key frame feature points is: traverse and calculate the distance between the current point feature and each key frame feature point in the first KD tree database in the first KD tree database, and update the shortest point feature distance from the calculated point feature distance. When the distances between all the key frame feature points in the first KD tree database and the current point feature have been calculated, the key frame feature point corresponding to the shortest point feature distance is selected as the key frame feature point that matches the current point feature to the closest distance, and the above steps are repeated until all point features in the current frame visual image are matched to the closest key frame feature point.

[0039] Step 3.2: Based on all the line features corresponding to the current frame visual image, KD tree searches are performed in the second KD tree database respectively to match the key frame feature lines with the closest distance respectively; specifically, the method for matching the key frame feature lines with the closest distance is: traverse and calculate the distance between the current line feature and each key frame feature line in the second KD tree database in the second KD tree database, and update the shortest line feature distance from the calculated line feature distance. When the distances between all the key frame feature lines in the second KD tree database and the current line feature have been calculated, the key frame feature line corresponding to the shortest line feature distance is selected as the key frame feature line with the closest distance to the current line feature, and the above steps are repeated until all line features in the current frame visual image are matched to the key frame feature line with the closest distance.

[0040] Step 3.3: Traverse and count the number of key-frame feature points and key-frame feature lines in each key-frame visual image that are closest to the current frame visual image; it should be noted that each key-frame feature point has a one-to-one corresponding key-frame visual image, and each key-frame feature line has a one-to-one corresponding key-frame visual image. There are multiple key-frame feature points and multiple key-frame feature lines in a key-frame visual image.

[0041] Step 3.4: Based on the number of key frame feature points and key frame feature lines in each key frame visual image that are closest to the current frame visual image, select a key frame visual image from all key frame visual images to update and select it as a candidate key frame visual image. This embodiment determines a key frame visual image that has the highest degree of match with the current frame visual image as a candidate key frame visual image update by counting the number of key frame feature points and key frame feature lines in the key frame visual image that are closest to the current frame visual image, so as to estimate the pose of the current frame visual image based on the pose information corresponding to the candidate key frame visual image, thereby reducing the initial pose estimation error.

[0042] As one of the implementation modes, step 3.4 selects a key frame visual image from all key frame visual images based on the number of key frame feature points and key frame feature lines in each key frame visual image that are closest to the current frame visual image and updates it as a candidate key frame visual image, specifically including: traversing and calculating the product of the number of key frame feature points in each key frame visual image that are closest to the current frame visual image and a first weight, as the first product corresponding to each key frame visual image; traversing and calculating the product of the number of key frame feature lines in each key frame visual image that are closest to the current frame visual image and a second weight, as the second product corresponding to each key frame visual image; traversing and calculating the sum of the first product and the second product corresponding to each key frame visual image as the candidate key frame selection information corresponding to each key frame visual image; traversing and comparing the candidate key frame selection values ​​corresponding to each key frame visual image, and selecting a key frame visual image with the largest candidate key frame selection value from all key frame visual images to update it as the candidate key frame visual image; wherein the second weight is greater than the first weight. This embodiment adjusts the influence weights of the number of key frame feature points and the number of key frame feature lines in the candidate key frame selection information through the first weight and the second weight. In particular, based on the fact that the information contained in the line features is richer and can contribute more reference information in the process of selecting the key frame visual image with the highest degree of matching, the second weight corresponding to the number of key frame feature lines is set to be greater than the first weight corresponding to the number of key frame feature points, thereby effectively optimizing the rationality of the selection of candidate key frames.

[0043] Preferably, in some embodiments of the present application, the second weight is set to twice the first weight to amplify the influence weight of the line feature in the candidate key frame selection information. For example, the first weight is set to 1 and the second weight is set to 2. In this case, the second weight is twice the first weight. When the candidate key frame selection information is calculated, the influence of the line feature is amplified to twice the influence of the point feature, which makes up for the defect that the reference information contributed by the point feature is less than that of the line feature in a low-texture environment or a scene with a changing light source.

[0044] As one of the implementation methods, a key frame visual image is selected from all key frame visual images and updated as a candidate key frame visual image, and further includes: constructing a third KD tree database that records all corresponding point features in the candidate key frame visual image based on the updated selected candidate key frame visual image; constructing a fourth KD tree database that records all corresponding line features in the candidate key frame visual image based on the updated selected candidate key frame visual image. This implementation method specifically constructs the third KD tree database and the fourth KD tree database corresponding to the updated selected candidate key frame visual image, so that when the current frame visual image is matched with the candidate key frame visual image, it only needs to be performed in the third KD tree database and the fourth KD tree database with smaller data volume, thereby reducing the resources required for matching operations and improving matching efficiency.

[0045] As one implementation method, step 4 estimates the pose of the current frame visual image based on the candidate key frame visual image to obtain the first pose corresponding to the current frame visual image, specifically including:

[0046] Step 4.1: Traverse the third KD tree database for all the corresponding point features in the current frame visual image and perform a KD tree search to find the first matching feature point pairs respectively; traverse the fourth KD tree database for all the corresponding line features in the current frame visual image and perform a KD tree search to find the first matching feature line pairs respectively; it should be noted that the first matching feature point pair may include one or more matching feature point pairs, and the first matching feature line pair may include one or more feature line pairs, each pair of matching feature point pairs includes a point feature in the current frame visual image and a point feature in the candidate key frame visual image, and each pair of matching feature line pairs includes a line feature in the current frame visual image and a line feature in the candidate key frame visual image.

[0047] Step 4.2: Use a random sampling consensus algorithm to perform first mismatch feature filtering on the first matching feature point pair and the first matching feature line pair, so as to obtain a first valid matching feature point pair and a first valid matching feature line pair; specifically, the random sampling consensus algorithm refers to the RANSAC algorithm (Random Sample Consensus), which can iteratively train the optimal parameter module from a set of observation data containing "outliers" to remove the outliers. This step uses a random sampling consensus algorithm to perform first error matching feature filtering on the first matching feature point pair and the first matching feature line pair, respectively, in order to filter out the mismatched feature point pairs or feature line pairs in the first matching feature point pair and the first matching feature line pair, and improve the matching effectiveness and accuracy.

[0048] Step 4.3: Use the PnP algorithm to estimate the pose of the first valid matching feature point pair and the first valid matching feature line pair to obtain the initial estimated pose; specifically, the PnP algorithm refers to (Perspective-n-Points algorithm), which is a three-dimensional object pose estimation method used to solve the transformation relationship between camera coordinates and world coordinates based on n known point pairs in space. Since the use of the PnP algorithm for pose estimation is a commonly used technical means in this field, the specific solution process will not be repeated here. Step 4.4: Use the reprojection error calculation to further optimize the initial estimated pose to obtain the first pose corresponding to the current frame visual image. In this embodiment, the random sampling consistency algorithm, the PnP algorithm, and the reprojection error algorithms are superimposed to obtain more accurate pose information to ensure the accuracy of pose estimation.

[0049] As one of the implementation modes, the use of the PnP algorithm to perform pose estimation on the first effective matching feature point pair and the first effective matching line pair specifically includes: using the starting endpoint pair corresponding to each pair of effective matching line pairs in the first effective matching line pair as a supplementary effective matching feature point pair; and using the PnP algorithm to perform pose estimation on the first effective matching feature point pair and the supplementary effective matching feature point pair. In this implementation mode, the starting endpoint of the line feature is supplemented as a point feature, which makes up for the defect of insufficient point feature extraction in some scenes. In addition, considering that the line feature is insensitive to length, resulting in instability of the end segment point of the line feature, and the starting endpoint of the line feature is usually the seed point with the largest gradient, only the starting endpoint of the line feature is used as a point feature supplement, thereby increasing the amount of point features that can be matched while ensuring the effectiveness of the match.

[0050] As one of the implementation modes, step 5 performs posture optimization on the first posture based on the point features and line features corresponding to the current frame visual image and the point features and line features corresponding to the candidate key frame visual image to obtain the final posture corresponding to the current frame visual image, specifically including: using the SIFT feature matching algorithm to match the corresponding point features and line features in the current frame visual image with the corresponding point features and line features in the candidate key frame visual image, respectively, to obtain second matching feature point pairs and second matching feature line pairs; specifically, the SIFT feature matching algorithm obtains the second matching feature point pairs by comparing the corresponding point features in the current frame visual image with the corresponding point feature descriptors in the candidate key frame visual image. Similarly, the SIFT feature matching algorithm obtains the second matching feature line pairs by comparing the corresponding line features in the current frame visual image with the corresponding line feature descriptors in the candidate key frame visual image, and the similarity measure of the descriptors is calculated using Euclidean distance. Based on the first pose, the second matching feature point pair and the second matching feature line pair, the reprojection error between the current frame visual image and the candidate key frame visual image is calculated; based on the reprojection error between the current frame visual image and the candidate key frame visual image, the second matching feature point pair and the second matching feature line pair are filtered out for the second mismatched feature to obtain the second valid matching feature point pair and the second valid matching feature line pair; based on the reprojection error between the current frame visual image and the candidate key frame visual image, a least squares model is constructed; based on the least squares model, the second valid matching feature point pair and the second valid matching feature line pair, the LM algorithm is used to iteratively solve to obtain the final pose corresponding to the current frame visual image. Among them, the LM algorithm refers to the Levenberg-Marquardt algorithm, which is one of the optimization algorithms. The algorithm uses extraction to solve the optimal value, has the advantages of fast optimization speed, can effectively deal with redundant parameter problems, and has a small chance of the cost function falling into the local minimum; the optimization refers to finding the parameter vector that minimizes the function value. In this embodiment, the LM algorithm is combined with the least squares model to achieve least squares optimization to solve the optimal pose. In this step, the least squares model is established with the minimum reprojection error as the criterion, and then the LM algorithm is used for optimization iteration to solve the optimal final pose.

[0051] As an implementation mode, a chip is provided in the implementation mode, wherein a computer program is stored inside the chip, and when the computer program stored inside the chip is executed by a processor, the monocular vision positioning method based on point and line features as described in any of the previous implementation modes is executed.

[0052] As an embodiment, a robot is provided in this embodiment, wherein the robot includes: a monocular camera, a chip and a processor; wherein the monocular camera is used to collect visual images; the processor is used to run the chip; when the chip is run by the processor, the monocular visual positioning method based on point and line features as described above is performed based on the visual image collected by the monocular camera to obtain the robot posture based on the visual image.

[0053] Those skilled in the art will understand that all or part of the steps in the above-mentioned implementation methods can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions for making a single-chip microcomputer, a chip or a processor (processor) perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0054] It should be noted that any process or method description in the flowchart or otherwise described herein may be understood to represent a module, fragment or portion of a code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and that the scope of the preferred embodiments of the present invention includes alternative implementations in which the functions may not be performed in the order described or discussed, including performing the functions in a substantially simultaneous manner or in the reverse order according to the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention. The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

Claims

1. A monocular vision positioning method based on point and line features, characterized in that: The monocular vision positioning method based on point and line features specifically includes: Extracting point features and line features corresponding to the current frame visual image from the current frame visual image; The point features and line features corresponding to the visual image of the current frame are represented by the same SIFT descriptor; Based on the KD tree database corresponding to the key frame visual image and the current frame visual image, the candidate key frame visual image is updated and selected; Based on the candidate key frame visual image, the pose of the current frame visual image is estimated to obtain the first pose corresponding to the current frame visual image; The first pose is optimized based on the point features and line features corresponding to the current frame visual image and the point features and line features corresponding to the candidate key frame visual image to obtain the final pose corresponding to the current frame visual image.

2. The monocular vision positioning method based on point and line features according to claim 1, characterized in that: The step of extracting point features and line features corresponding to the current frame visual image from the current frame visual image specifically includes: Establish a Gaussian pyramid for the current frame visual image; The SIFT algorithm is used to detect and extract point features in the current frame visual image from the Gaussian pyramid corresponding to the current frame visual image, and the SIFT descriptor is used to characterize the point features corresponding to the current frame visual image; The LSD algorithm is used to detect and extract the line features in the current frame visual image from the Gaussian pyramid corresponding to the current frame visual image, and the LBD descriptor is used to characterize the line features corresponding to the current frame visual image.

3. The monocular vision positioning method based on point and line features according to claim 2 is characterized in that: The method of using the same SIFT descriptor to represent the point features and line features corresponding to the current frame visual image specifically includes: extracting the mean part of the line features corresponding to the current frame visual image represented by the LBD descriptor as the SIFT descriptor of the line features corresponding to the current frame visual image, so that the point features and line features corresponding to the current frame visual image are represented by the same SIFT descriptor.

4. The monocular vision positioning method based on point and line features according to claim 3 is characterized in that: The monocular vision positioning method based on point and line features also includes: when the line feature corresponding to the current frame visual image is stored, the pixel coordinates and normalized plane coordinates of the starting endpoint and the ending endpoint of the line feature in the current frame visual image are recorded.

5. The monocular vision positioning method based on point and line features according to claim 4 is characterized in that: The KD tree database corresponding to the key frame visual image includes: a first KD tree database recording all corresponding point features in all historical key frame visual images and a second KD tree database recording all corresponding line features in all historical key frame visual images.

6. The monocular vision positioning method based on point and line features according to claim 5, characterized in that: The updating and selecting of the candidate key frame visual images based on the KD tree database corresponding to the key frame visual images in combination with the current frame visual images specifically includes: Based on all point features corresponding to the visual image of the current frame, KD tree searches are performed in the first KD tree database to match the closest key frame feature points respectively; Based on all line features corresponding to the visual image of the current frame, KD tree searches are performed in the second KD tree database to match the closest key frame feature lines respectively; Traverse and count the number of key frame feature points and key frame feature lines in each key frame visual image that are closest to the current frame visual image; Based on the number of key frame feature points and key frame feature lines in each key frame visual image that are closest to the current frame visual image, a key frame visual image is selected from all key frame visual images to be updated and selected as a candidate key frame visual image.

7. The monocular vision positioning method based on point and line features according to claim 6, characterized in that: The method of selecting a key frame visual image from all key frame visual images based on the number of key frame feature points and key frame feature lines in each key frame visual image that are closest to the current frame visual image and updating and selecting the key frame visual image as the candidate key frame visual image specifically includes: Traversing and calculating the product of the number of key frame feature points closest to the current frame visual image in each key frame visual image and the first weight as the first product corresponding to each key frame visual image; Traversing and calculating the product of the number of key frame feature lines closest to the current frame visual image in each key frame visual image and the second weight as the second product corresponding to each key frame visual image; Traversing and calculating the sum of the first product and the second product corresponding to each key frame visual image as candidate key frame selection information corresponding to each key frame visual image; Traverse and compare the candidate key frame selection values ​​corresponding to each key frame visual image, and select a key frame visual image with the largest candidate key frame selection value from all key frame visual images and update it as the candidate key frame visual image; The second weight is greater than the first weight.

8. The monocular vision positioning method based on point and line features according to claim 7, characterized in that: A key frame visual image is selected from all key frame visual images and updated as a candidate key frame visual image, which also includes: constructing a third KD tree database that records all corresponding point features in the one frame of candidate key frame visual image based on the updated selected candidate key frame visual image; and constructing a fourth KD tree database that records all corresponding line features in the one frame of candidate key frame visual image based on the updated selected candidate key frame visual image.

9. The monocular vision positioning method based on point and line features according to claim 8, characterized in that: The step of estimating the pose of the current frame visual image based on the candidate key frame visual image to obtain the first pose corresponding to the current frame visual image specifically includes: Traversing all the corresponding point features in the visual image of the current frame in the third KD tree database to perform KD tree search to find the first matching feature point pairs respectively; Traversing all corresponding line features in the visual image of the current frame in the fourth KD tree database to perform KD tree search to find the first matching feature line pairs respectively; Using a random sampling consistency algorithm to perform first mismatch feature filtering on the acquired first matching feature point pairs and first matching feature line pairs, so as to obtain first valid matching feature point pairs and first valid matching feature line pairs; The PnP algorithm is used to estimate the pose of the first valid matching feature point pair and the first valid matching feature line pair to obtain an initial estimated pose; The reprojection error calculation is used to further optimize the initial estimated pose to obtain the first pose corresponding to the visual image of the current frame.

10. The monocular vision positioning method based on point and line features according to claim 9, characterized in that: The method of using the PnP algorithm to estimate the pose of the first effective matching feature point pair and the first effective matching line pair specifically includes: taking the starting endpoint pair corresponding to each pair of effective matching line pairs in the first effective matching line pair as a supplementary effective matching feature point pair; and using the PnP algorithm to estimate the pose of the first effective matching feature point pair and the supplementary effective matching feature point pair.

11. The monocular vision positioning method based on point and line features according to claim 10, characterized in that: The step of optimizing the first pose based on the point features and line features corresponding to the visual image of the current frame and the point features and line features corresponding to the visual image of the candidate key frame to obtain the final pose corresponding to the visual image of the current frame specifically includes: The corresponding point features and line features in the current frame visual image are matched with the corresponding point features and line features in the candidate key frame visual image by using the SIFT feature matching algorithm to obtain a second matching feature point pair and a second matching feature line pair; Calculate the reprojection error between the current frame visual image and the candidate key frame visual image based on the first pose, the second matching feature point pair and the second matching feature line pair; Performing second mismatch feature filtering on the acquired second matching feature point pairs and second matching feature line pairs based on the reprojection error between the current frame visual image and the candidate key frame visual image to obtain second valid matching feature point pairs and second valid matching feature line pairs; A least squares model is constructed based on the reprojection error between the current frame visual image and the candidate key frame visual image; Based on the least squares model, the second effective matching feature point pairs and the second effective matching feature line pairs are iteratively solved using the LM algorithm to obtain the final pose corresponding to the current frame visual image.

12. A chip having a computer program stored therein, characterized in that: When the computer program stored in the chip is executed by the processor, the monocular vision positioning method based on point and line features as described in claims 1 to 11 is executed.

13. A robot, characterized in that: The robot includes: a monocular camera, a chip as claimed in claim 12, and a processor; wherein the monocular camera is used to collect visual images; the processor is used to run the chip; when the chip is run by the processor, the monocular visual positioning method based on point and line features as claimed in claims 1 to 11 is executed based on the visual image collected by the monocular camera to obtain the robot posture based on the visual image.

Citation Information

Patent Citations

  • Downward-looking visual compass for fusing point features and line features

    CN108921896A

  • A texture-free metal part image pose detection method based on a CAD sparse template

    CN109887030A

  • Unmanned aerial vehicle visual odometer method based on depth point-line characteristics

    CN111047620A

  • Equipment pose calculation method, device and apparatus

    CN111461141A

  • Simultaneous positioning and mapping method for autonomous mobile platform in rescue scene

    CN111583136A

Cited By

  • Positioning method, storage medium, electronic equipment and program product

    CN120890440A