A visual positioning method, device, storage medium and electronic equipment
By fusing feature points and feature lines in visual localization and constructing reprojection error and optimization functions, the problem of insufficient visual localization accuracy and robustness in weak texture environments is solved, achieving higher localization accuracy and stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2026-03-02
- Publication Date
- 2026-05-08
AI Technical Summary
In weakly textured environments, existing feature-point-based visual localization methods suffer from poor localization accuracy and robustness due to the lack of stable feature points.
By combining feature points and feature lines, and constructing point reprojection error and line reprojection error, a target optimization function is built to solve the relative pose, thereby improving the accuracy and robustness of visual positioning.
By fusing point and line features, the stability and robustness of feature extraction are enhanced, overcoming the limitations of feature point extraction in weak texture environments and improving the accuracy and robustness of visual localization.
Smart Images
Figure CN121746472B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information technology, and in particular to a visual positioning method, device, storage medium, and electronic device. Background Technology
[0002] With the continuous evolution of robotics technology, visual SLAM (Simultaneous Localization and Mapping) has become an indispensable core technology for positioning and navigation in fields such as autonomous driving, industrial robots, and drones. This technology does not rely on traditional positioning methods such as GPS, demonstrating irreplaceable practical value in scenarios with no satellite signal coverage or high-precision positioning requirements. Although visual SLAM technology has made breakthroughs in many fields, it still faces many technical bottlenecks that urgently need to be addressed in complex real-world conditions, and its positioning performance is easily affected by special environmental factors.
[0003] Currently, visual localization based on feature points is commonly used. However, in weakly textured environments such as open corridors, large flat surfaces, or areas with sparse textures, the lack of stable feature points means that this existing visual localization method, which relies solely on feature points, can affect the accuracy and robustness of localization. Summary of the Invention
[0004] In view of this, this application provides a visual positioning method, apparatus, storage medium and electronic device, which mainly overcomes the limitations of feature point extraction in weak texture environments, thereby improving the accuracy and robustness of visual positioning.
[0005] According to a first aspect of this application, a visual positioning method is provided, the method comprising:
[0006] Obtain the current frame image and the previous frame image corresponding to the current frame image;
[0007] Extract the feature points and feature lines corresponding to the previous frame image and the feature points and feature lines corresponding to the current frame image, and perform feature point matching and feature line matching on the previous frame image and the current frame image to obtain feature point matching pairs and feature line matching pairs;
[0008] Based on the feature point matching pairs and the feature line matching pairs, the point reprojection error and the line reprojection error between the previous frame image and the current frame image are respectively constructed.
[0009] Based on the point reprojection error and the line reprojection error, construct the target optimization function;
[0010] Based on the objective optimization function, the relative pose between the current frame image and the previous frame image is calculated.
[0011] Based on the relative pose, the target object is visually located to obtain the positioning information of the target object based on the current frame image.
[0012] According to a second aspect of this application, a visual positioning device is provided, the device comprising:
[0013] The acquisition unit is used to acquire the current frame image and the previous frame image corresponding to the current frame image;
[0014] The extraction unit is used to extract the feature points and feature lines corresponding to the previous frame image and the feature points and feature lines corresponding to the current frame image, and to perform feature point matching and feature line matching on the previous frame image and the current frame image to obtain feature point matching pairs and feature line matching pairs.
[0015] The construction unit is used to construct the point reprojection error and the line reprojection error between the previous frame image and the current frame image based on the feature point matching pair and the feature line matching pair, respectively.
[0016] The construction unit is further configured to construct a target optimization function based on the point reprojection error and the line reprojection error;
[0017] The calculation unit is used to calculate the relative pose between the current frame image and the previous frame image based on the target optimization function;
[0018] The positioning unit is used to perform visual positioning of the target object based on the relative pose, and obtain the positioning information of the target object based on the current frame image.
[0019] According to a third aspect of this application, a storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described visual positioning method.
[0020] According to a fourth aspect of this application, an electronic device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described visual positioning method.
[0021] By employing the above technical solutions, this application provides a visual positioning method, apparatus, storage medium, and electronic device. Compared with existing technologies, it first constructs the point reprojection error and line reprojection error between the previous frame image and the current frame image based on feature point matching pairs and feature line matching pairs, respectively. Then, based on the point reprojection error and line reprojection error, it constructs a target optimization function. Finally, based on the target optimization function, it calculates the relative pose between the current frame image and the previous frame image, thereby achieving visual positioning. The point features extracted in this application are suitable for regions with obvious textures, while line features provide additional geometric constraints for regions with sparse textures. The fusion of point features and line features significantly enhances the image feature extraction capability, improves the stability and robustness of feature extraction, thereby overcoming the limitations of feature point extraction in weak texture environments and improving the accuracy and robustness of visual positioning.
[0022] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description
[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0024] Figure 1 A flowchart illustrating a visual positioning method provided in an embodiment of this application is shown;
[0025] Figure 2 The reprojection error map of the point features provided in the embodiments of this application is shown;
[0026] Figure 3 A flowchart illustrating the line reprojection error calculation method provided in an embodiment of this application is shown.
[0027] Figure 4 The diagram shows the reprojection error of a line feature provided in an embodiment of this application;
[0028] Figure 5 This document illustrates the overall process of visual positioning provided in an embodiment of this application.
[0029] Figure 6 A schematic diagram of the structure of a visual positioning device provided in an embodiment of this application is shown. Detailed Implementation
[0030] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.
[0031] Existing technologies lack stable feature points in weakly textured environments, limiting their feature extraction capabilities and thus affecting the accuracy and robustness of visual localization.
[0032] To address the above problems, embodiments of the present invention provide a visual positioning method, such as... Figure 1 As shown, the method includes:
[0033] Step 10: Obtain the current frame image and the previous frame image corresponding to the current frame image.
[0034] The current frame and the previous frame are captured by a camera on the target object. The camera can continuously capture images to form a sequence of image frames. The current frame and the previous frame are any two adjacent frames in the image frame sequence. The target object can be a robot with autonomous navigation capabilities, a vehicle with autonomous driving capabilities, etc. It should be noted that the target object is not limited to robots and vehicles, but can also be other objects with autonomous navigation and positioning capabilities.
[0035] The embodiments of the present invention are mainly applicable to visual positioning scenarios. The executing entity of the embodiments of the present invention is a device or equipment capable of visual positioning.
[0036] In this embodiment of the invention, in order to visually locate the target object during its movement, the camera will capture images, and the visual location of the target object can be achieved based on the captured image frames.
[0037] In low-light environments, image details are difficult to capture, and the quality of feature point extraction decreases significantly, affecting the accuracy and robustness of visual localization. Simultaneously, the reduced number of feature points and increased image noise also impact the matching quality of feature points. Therefore, this embodiment of the invention performs image enhancement processing on the current frame and the previous frame before extracting feature points and feature lines. Specifically, it uses a low-light enhancement network to improve image quality, ensuring the accuracy of subsequent feature extraction. The low-light enhancement network specifically includes RetinexNet, EnlightenGAN, and LLFormer.
[0038] Specifically, the low-light enhancement algorithm incorporates physical and unsupervised loss functions during the training phase, enabling it to adapt to diverse and complex scenes while exhibiting operational insensitivity and model-independent generalization capabilities. This low-light enhancement algorithm demonstrates excellent performance in visual quality, computational efficiency, and downstream tasks (such as low-light object detection and nighttime semantic segmentation), exhibiting particularly strong robustness when handling unknown real-world scenes.
[0039] Step 20: Extract the feature points and feature lines corresponding to the previous frame image and the feature points and feature lines corresponding to the current frame image, and perform feature point matching and feature line matching on the previous frame image and the current frame image to obtain feature point matching pairs and feature line matching pairs.
[0040] In this embodiment of the invention, after acquiring the previous frame image and the current frame image, feature points and feature lines in the current frame image, as well as feature points and feature lines in the previous frame image, are extracted respectively. Specifically, step 20 includes the following steps for feature point and feature line extraction: using a directional fast rotation binary descriptor algorithm to extract feature points and descriptors corresponding to the previous frame image and the current frame image respectively; performing feature point matching based on the feature points and descriptors corresponding to the previous frame image and the current frame image to obtain feature point matching pairs; using a real-time line segment detection algorithm to extract feature lines corresponding to the previous frame image and the current frame image respectively; calculating the direction difference, length ratio, and midpoint distance between the feature lines corresponding to the previous frame image and the feature lines corresponding to the current frame image, and performing feature line matching based on the direction difference, length ratio, and midpoint distance to obtain feature line matching pairs.
[0041] Specifically, the Oriented Fast Rotation Binary Descriptor Algorithm includes the ORB feature extraction algorithm, and the Real-Time Line Segment Detection Algorithm includes the EDlines algorithm.
[0042] Specifically, the ORB feature extraction algorithm is used to extract ORB feature points from the current frame and the previous frame, respectively. Then, feature point matching is performed based on the descriptor operators corresponding to the ORB feature points to obtain matching feature point pairs. Simultaneously, the EDlines algorithm is used to extract feature lines from the current frame and the previous frame, respectively, and the direction difference, length ratio, and midpoint distance between the feature lines corresponding to the previous and current frames are calculated. The length ratio is the ratio of the length of the shortest to the length of the longest feature line between any two feature lines, where the two feature lines are the feature lines in the current and previous frames, respectively. If the direction difference is less than a preset angle, the length ratio is greater than a preset length, and the midpoint distance is less than a preset midpoint distance, then the two feature lines are determined to be a matched pair.
[0043] Step 30: Based on the feature point matching pair and the feature line matching pair, construct the point reprojection error and line reprojection error between the previous frame image and the current frame image, respectively.
[0044] In embodiments of the present invention, when the environment is open or the texture is sparse, traditional feature-point-based visual localization algorithms often face significant limitations, mainly manifested in the scarcity and low quality of feature points, making it difficult to provide stable pose constraints and affecting the accuracy of visual localization. To address this technical problem, embodiments of the present invention employ a multimodal feature extraction method based on point-line feature fusion. This method combines point features and line features, using ORB feature points and descriptors, along with line features extracted by the EDlines algorithm, to improve the robustness of the system in low-texture environments. ORB features provide effective point features suitable for regions with obvious texture, while the line features extracted by the EDlines algorithm provide additional geometric constraints for regions with sparse texture. The fusion of these two significantly enhances the feature extraction capability, ensuring stability and accuracy during the localization process.
[0045] After extracting and matching feature points and feature lines in the current frame and the previous frame, the camera pose can be estimated using these feature point pairs and feature line pairs. However, since the camera pose itself is an estimate, and the observed data is inevitably affected by noise, it is necessary to construct and calculate the reprojection error for both point features and line features separately, thereby continuously optimizing the estimated pose.
[0046] The method for calculating the reprojection error of point features specifically includes: for any feature point pair in the feature point matching pair, based on the pixel coordinates of the first feature point in the previous frame image and the historical relative pose between the previous two frames corresponding to the previous frame image and the current frame image, calculating the three-dimensional coordinates of the first feature point in the world coordinate system; based on the three-dimensional coordinates of the first feature point in the world coordinate system, the camera intrinsic parameter matrix, and the scale factor, calculating the projection coordinates of the first feature point in the current frame image; based on the pixel coordinates of the second feature point in the current frame image and the projection coordinates of the first feature point in the current frame image, calculating the reprojection error corresponding to the arbitrary feature point pair; and determining the point reprojection error based on the reprojection error corresponding to the arbitrary feature point pair.
[0047] Specifically, such as Figure 2 As shown, I 1 represents the previous frame. I 2 represents the current frame image. p 1 represents the first feature point in the previous frame image. p2 is the second feature point in the current frame image. p 1 and p 2 represents a pair of feature points. Let L be the projection of the first feature point onto the current frame image, and L be the projection of the first feature point onto the current frame image. p 2 and The distance between them is represented by the solid red dots, which are the three-dimensional points of the first feature point in the world coordinate system. During calculation, the first feature point is used as the basis for... p 1. The pixel coordinates in the previous frame, and the historical relative pose between the previous frame and the two frames before that. T 1. Calculate the first feature point p 1. Three-dimensional coordinates in the world coordinate system, i.e. Figure 2 The three-dimensional coordinates of the solid red dot. The specific formula is as follows.
[0048] sU 1= KT 1 P
[0049] in, s The scaling factor. U 1 is the first feature point p 1. Pixel coordinates in the previous frame image. T 1 represents the historical relative pose between the previous frame and the two frames above, predicted at the previous time step. K Let be the camera intrinsic parameter matrix. The first feature point can be derived from the above formula. p 1. Three-dimensional coordinates in the world coordinate system, i.e., the three-dimensional coordinates of the solid circle in the world coordinate system. P .
[0050] After calculating the first feature point p 1. Three-dimensional coordinates in the world coordinate system P Next, the first feature point is derived. p 1. In the current frame image I The projected coordinates in 2, i.e. The pixel coordinates. The specific formula is as follows:
[0051] sU 2= KT 2 P
[0052] in, T 2 represents the relative pose between the current frame and the previous frame, and is the variable to be solved. U 2 is the first feature point p 1. In the current frame image I The projected coordinates in 2, i.e. pixel coordinates, The pixel coordinates can be represented based on the variable to be solved.T A function of 2.
[0053] Furthermore, based on the second feature point p 2. In the current frame image I Pixel coordinates in 2 U 3, and the first feature point p 1. In the current frame image I Projected coordinates in 2 ( (pixel coordinates), calculate the reprojection error corresponding to the feature point pair composed of the first and second feature points. This refers to the point reprojection error. The specific calculation formula is as follows:
[0054]
[0055] Through continuous iterative adjustments T A value of 2 will gradually reduce the reprojection error, eventually minimizing all reprojection errors and yielding the corresponding camera pose, i.e., the relative pose between the current frame image and the previous frame image.
[0056] The calculation of reprojection error for line features is similar to that for point features. The reprojection error of line features reflects the difference between the projection of a 3D line segment onto the image plane and the detected 2D line segment. Figure 3 As shown, it specifically includes:
[0057] Step 31: For any feature line pair in the feature line matching pair, perform two-dimensional reprojection calculation on the feature line pair to obtain the two-dimensional reprojection error corresponding to the feature line pair.
[0058] In this embodiment of the invention, when calculating the two-dimensional reprojection error, the three-dimensional coordinates of the first feature line in the previous frame image are determined based on the pixel coordinates of the first feature line in the arbitrary feature line pair; the projection coordinates of the first feature line in the current frame image are determined based on the three-dimensional coordinates of the first feature line in the world coordinate system; the perpendicular distance between the second feature line and the corresponding projection line segment of the first feature line is calculated based on the pixel coordinates of the second feature line in the current frame image and the projection coordinates of the first feature line in the current frame image; and the two-dimensional reprojection error is determined based on the perpendicular distance between the second feature line and the corresponding projection line segment of the first feature line.
[0059] Specifically, such as Figure 4 As shown, I k For the current frame image, and These are the two endpoints of the second feature line in the current frame image. p and q These are the projections of the two endpoints of the first feature line into the current frame image. P and Q These are the two endpoints of the first feature line in the world coordinate system. When calculating the two-dimensional reprojection error, since the first feature line is composed of multiple points (including endpoints and intermediate points), the three-dimensional coordinates of the first feature line in the world coordinate system can be calculated based on the pixel coordinates of the first feature line in the previous frame image, using the aforementioned formula for calculating point reprojection error. After determining the three-dimensional coordinates of the first feature line in the world coordinate system, the projection coordinates of the first feature line in the current frame image, i.e., the line segment (…), can be derived similarly using the aforementioned formula for calculating point reprojection error. p , q ) pixel coordinates, line segment ( p , q The pixel coordinates of () can be represented based on the variable to be solved. T The function is 2. Then, based on the pixel coordinates of the second feature line in the current frame image and the projection coordinates of the first feature line in the current frame image, the second feature line is calculated. The projection line segment corresponding to the first feature line ( p , q The vertical distance between the two is calculated as the projection line segment ( p , q ) to two-dimensional line segment The perpendicular distance, assuming the equation of the two-dimensional line segment is... ,in, It is the normal vector of a two-dimensional line segment. It is a two-dimensional line segment The distance to the origin and the perpendicular distance can be expressed as:
[0060]
[0061] in, and Three-dimensional points P and Q Projection and two-dimensional line segments in the current frame image The distance between them and It can be used to represent projected line segments ( p , q ) to two-dimensional line segment The vertical distance; k Indicates the first k In this embodiment of the invention, a frame image is equivalent to the current frame image. Indicates the first A pair of feature points.
[0062] Furthermore, the projected line segment ( p , q ) to two-dimensional line segment Projection error vector between Specifically, it can be expressed as:
[0063]
[0064] Furthermore, the two-dimensional reprojection error can be specifically defined as:
[0065]
[0066] in, This is the two-dimensional reprojection error. It is the covariance matrix of the linear characteristic. This represents a robust kernel function, which is used to suppress the impact of outliers on optimization.
[0067] Two-dimensional reprojection error measures the error between the projection of the two endpoints of a spatial line onto a planar image and the endpoints of the extracted feature line segment. By minimizing this error, the camera pose (relative pose between two frames) can be optimized, thereby making the projection of the line segment more consistent with the detection results in the image.
[0068] Step 32: Perform a three-dimensional reprojection calculation on any one of the feature line pairs to obtain the three-dimensional reprojection error corresponding to any one of the feature line pairs.
[0069] In this embodiment of the invention, the three-dimensional reprojection error of a line segment measures the error between the back projection of the image plane line segment into space and the spatial line segment. Specifically, step 32 includes the following calculation process for the three-dimensional reprojection error: Based on the pixel coordinates of the second feature line in the current frame image, backproject the second feature line into space to obtain the three-dimensional coordinates of the second feature line in the world coordinate system; based on the three-dimensional coordinates of the second feature line in the world coordinate system and the three-dimensional coordinates of the first feature line in the world coordinate system, calculate the perpendicular distance between the first spatial line segment corresponding to the first feature line and the second spatial line segment corresponding to the second feature line, as well as the endpoint distance between the first spatial line segment and the second spatial line segment; based on the perpendicular distance between the first spatial line segment and the second spatial line segment, and the endpoint distance between the first spatial line segment and the second spatial line segment, determine the three-dimensional reprojection error.
[0070] Specifically, such as Figure 4 As shown, line segment l This is the back projection of the second characteristic line in space, i.e., the second spatial line segment. and They are spatial line segmentsl The two endpoints, a three-dimensional line segment ( P , Q The first characteristic line is the spatial segment (first spatial segment) in the world coordinate system. The three-dimensional reprojection error between the first spatial segment and the second spatial segment can be specifically defined as:
[0071]
[0072] in, This is for 3D reprojection error. For three-dimensional line segments ( P , Q ) and line segment l The projection error vector between them This represents a robust kernel function, which is used to suppress the impact of outliers on optimization.
[0073] Furthermore, the projection error vector This can be specifically expressed as:
[0074]
[0075] in, and Representing three-dimensional points respectively P and Q With line segment l Distance in three-dimensional space, used to represent three-dimensional line segments ( P , Q ) and line segment l The vertical distance between them Represents the weight parameter. Representing three-dimensional points P With back projection point The distance between them Representing three-dimensional points Q With back projection point The distance between them. and Specifically, it can be defined as:
[0076]
[0077] in, and Representing three-dimensional points respectively P and Q The coordinates of a 3D point in the camera coordinate system can be converted from the world coordinate system to the camera coordinate system using camera intrinsic parameters. k Indicates the first k In this embodiment of the invention, a frame image is equivalent to the current frame image. Indicates the first A pair of characteristic lines.
[0078] Furthermore, and It can be specifically defined as:
[0079]
[0080] 3D reprojection error incorporates depth information, enabling a more accurate estimation of the positions of line segment endpoints. 2D reprojection error is also based on the variables to be solved. T The function 2 optimizes the camera pose (relative pose between two frames) by minimizing this error, thereby making the endpoints of the line segment more consistent with the back-projection point and improving the accuracy of visual positioning.
[0081] Step 33: Based on the two-dimensional reprojection error and three-dimensional reprojection error corresponding to any feature line pair, determine the line reprojection error between the previous frame image and the current frame image.
[0082] In this embodiment of the invention, after calculating the two-dimensional reprojection error and the three-dimensional reprojection error corresponding to each feature line pair, the two-dimensional reprojection error and the three-dimensional reprojection error are added together to obtain the line reprojection error corresponding to each feature line. Then, the line reprojection errors corresponding to all feature line pairs are added together to obtain the line reprojection error between the previous frame image and the current frame image.
[0083] Step 40: Construct the objective optimization function based on the point reprojection error and the line reprojection error.
[0084] In this embodiment of the invention, Local Bundle Adjustment (BA) is used to optimize the relative pose between the current frame image and the previous frame image. Specifically, a loss function (objective optimization function) is constructed based on the two-dimensional reprojection error and the three-dimensional reprojection error.
[0085] Step 50: Based on the target optimization function, calculate the relative pose between the current frame image and the previous frame image.
[0086] In this embodiment of the invention, the relative pose and 3D feature positions are adjusted using a nonlinear optimization method, and the optimal parameter values are finally solved using a sparse solution method, thereby improving visual positioning accuracy. The specific formula for the objective optimization function is as follows.
[0087]
[0088] in, Represents a set of image frames. This represents the total weight projection error. This represents the sum of reprojection errors for all feature point pairs. This represents the sum of reprojection errors for all feature line pairs. By solving this objective function, the optimal relative pose can be obtained. T 2.
[0089] In the specific optimization solution, the Jacobian matrix of point reprojection error and line reprojection error is calculated separately, and the relative pose is iteratively optimized step by step through the Levenberg-Marquardt (LM) algorithm to gradually reduce the total point reprojection error, and finally output the camera pose (relative pose) with the minimum total reprojection error.
[0090] Step 60: Based on the relative pose, perform visual localization of the target object to obtain the localization information of the target object based on the current frame image.
[0091] For embodiments of the present invention, Figure 2 and Figure 4 The feature points and feature lines in the image are all environmental features around the target object. After calculating the relative pose between the current frame image and the previous frame image, the three-dimensional coordinates of these surrounding environmental features in the world coordinate system can be determined based on the relative pose, thereby determining the three-dimensional coordinates of the target object itself in the world coordinate system and realizing visual positioning.
[0092] In the actual operating environment of robots, the movement of dynamic interference targets such as pedestrians and vehicles can cause positional shifts or false feature generation of feature points and feature lines in images, easily leading to incorrect matching of feature points and feature lines, and thus compromising the positioning stability of the visual positioning system. To eliminate the impact of dynamic objects on the accuracy of the positioning system, this invention also designs a dynamic feature point (line) elimination algorithm based on semantic and geometric constraints. This algorithm, combining semantic information and geometric constraints, can effectively eliminate dynamic feature points and dynamic feature lines in the environment, thereby reducing positioning errors caused by object interference.
[0093] Based on this, such as Figure 5As shown, the method further includes: performing target recognition on the current frame image and the previous frame image respectively to obtain semantic information corresponding to the current frame image and the previous frame image; determining potential dynamic feature points and potential dynamic feature lines in the current frame image, and potential dynamic feature points and potential dynamic feature lines in the previous frame image based on the semantic information corresponding to the current frame image and the semantic information corresponding to the previous frame image; determining potential dynamic feature point matching pairs and potential dynamic feature line matching pairs based on the potential dynamic feature points and potential dynamic feature lines in the current frame image and the potential dynamic feature points and potential dynamic feature lines in the previous frame image; determining target feature point matching pairs and target feature line matching pairs based on the potential dynamic feature point matching pairs and potential dynamic feature line matching pairs; and simultaneously constructing point reprojection error and line reprojection error between the previous frame image and the current frame image based on the target feature point matching pairs and target feature line matching pairs.
[0094] Specifically, firstly, target detection is performed on the current frame and the previous frame using target recognition networks such as YOLOv8 to obtain semantic information from these images. This semantic information includes the category information of the target object. Based on this semantic information, potential dynamic feature points and potential dynamic feature lines in the current frame and the previous frame can be identified. These potential dynamic feature points and lines are then excluded, retaining only feature points and lines in the static background environment. This yields static feature point matching pairs and static feature line matching pairs, which are then used as target feature point matching pairs and target feature line matching pairs, thus constructing point reprojection error and line reprojection error. Specifically, during the exclusion process, if the target object is determined to be a dynamic object based on its semantic information, all feature points and feature lines within the target object's detection box are identified as potential dynamic feature points and potential dynamic feature lines and are excluded.
[0095] For a feature point matching pair, if one of the feature points in a feature point matching pair is identified as a potential dynamic feature point, then the other feature point that matches it is also excluded from the corresponding image as a potential dynamic feature point. This feature point matching pair is a potential dynamic feature point matching pair. For a feature line matching pair, since the two endpoints of a line segment can be regarded as feature points, if one endpoint is identified as a potential dynamic feature point, then the line segment is identified as a potential dynamic feature line, and the other feature line that matches it is also excluded from the corresponding image as a potential dynamic feature line. This feature line matching pair is a potential dynamic feature line matching pair.
[0096] The above method removes all potential dynamic feature points and potential dynamic feature lines based on semantic information, retaining only feature points and feature lines in the background static environment. However, this method of removing feature points and feature lines based solely on semantic information can lead to the removal of many real static feature points and feature lines, i.e., the erroneous removal of many static feature points, resulting in low robustness of the system. Therefore, this embodiment of the invention will also perform a secondary screening of potential dynamic feature points and potential dynamic feature lines.
[0097] Based on this, the method further includes: when determining the target feature point matching pair and the target feature line matching pair, determining the static feature point matching pair and the static feature line matching pair based on the potential dynamic feature point matching pair and the potential dynamic feature line matching pair; calculating the fundamental matrix based on the static feature point matching pair, and determining the epipolar line between the current frame image and the previous frame image; calculating the epipolar line distances corresponding to the potential dynamic feature point matching pair and the potential dynamic feature line matching pair based on the potential dynamic feature point matching pair and the potential dynamic feature line matching pair, as well as the epipolar line and the fundamental matrix; determining the true static feature point matching pair in the potential dynamic feature point matching pair and the true static feature line matching pair in the potential dynamic feature line matching pair based on the epipolar line distance; and determining the target feature point matching pair and the target feature line matching pair based on the true static feature point matching pair and the static feature point matching pair, as well as the true static feature line matching pair and the static feature line matching pair.
[0098] Specifically, after excluding potential dynamic feature point matching pairs and potential dynamic feature line matching pairs, static feature point matching pairs and static feature line matching pairs are obtained. The fundamental matrix is then solved using eight pairs of static feature point matching pairs. F 0, F 0 describes the static geometric relationship between the current frame image and the previous frame image. This is used to obtain the fundamental matrix. F After 0, using the fundamental matrix F 0. A secondary screening is performed on the previously eliminated potential dynamic feature point matching pairs and potential dynamic feature line matching pairs. According to epipolar geometry constraints, static feature points will map to the vicinity of the epipolar line in the next frame image. Because dynamic feature points move in the environment, they have a certain offset distance from the epipolar line in the next frame image. The larger this distance, the greater the probability that the feature point belongs to a dynamic object. Based on this, for the eliminated potential dynamic feature point matching pairs, the distance from the potential dynamic feature point in the previous frame image to the epipolar line in the current frame image is calculated, i.e., the epipolar distance corresponding to the potential dynamic feature point pair. The specific formula for calculating the epipolar distance is as follows.
[0099]
[0100] in, Represents the polar distance. and It is a pair of potential dynamic feature points matched between the previous frame and the current frame. X and Y These are the components of the epipolar vector.
[0101] If the polar distance If the value exceeds a preset threshold, the potential dynamic feature point matching pair is determined to be a true static feature point matching pair and retained; otherwise, it is a dynamic feature point matching pair. Finally, all true static feature point matching pairs and static feature point matching pairs are combined to calculate the point reprojection error. Similarly, the endpoints of a potential dynamic feature line matching pair can be considered as two potential dynamic feature point matching pairs. The determination of whether two endpoint pairs are dynamic feature point matching pairs can be made using the above method. If one endpoint pair is a dynamic feature point matching pair, the potential dynamic feature line matching pair is determined to be a dynamic feature line matching pair. Finally, all true static feature line matching pairs and static feature line matching pairs are combined to calculate the line reprojection error.
[0102] Therefore, by following the above method, the influence of dynamic objects can be removed, while retaining as many static feature points as possible, effectively improving the visual positioning accuracy and robustness of the system in dynamic scenes.
[0103] This invention provides a visual localization method that significantly enhances image feature extraction capabilities by fusing point and line features, improving the stability and robustness of feature extraction. This overcomes the limitations of feature point extraction in weakly textured environments, thereby improving the accuracy and robustness of visual localization. Simultaneously, this invention utilizes a low-light enhancement network to effectively improve image quality in low-light environments, providing a more reliable foundation for subsequent feature extraction. Furthermore, this invention designs a dynamic feature removal strategy that accurately identifies and removes interference caused by dynamic objects, further improving the accuracy and stability of the system's visual localization.
[0104] Furthermore, as Figure 1 and Figure 3 To specifically implement the method shown, this embodiment provides a visual positioning device, such as... Figure 6 As shown, the device includes: an acquisition unit 101, an extraction unit 102, a construction unit 103, a calculation unit 104, and a positioning unit 105.
[0105] The acquisition unit 101 can be used to acquire the current frame image and the previous frame image corresponding to the current frame image.
[0106] The extraction unit 102 can be used to extract the feature points and feature lines corresponding to the previous frame image and the feature points and feature lines corresponding to the current frame image, and perform feature point matching and feature line matching on the previous frame image and the current frame image to obtain feature point matching pairs and feature line matching pairs.
[0107] The construction unit 103 can be used to construct the point reprojection error and line reprojection error between the previous frame image and the current frame image based on the feature point matching pair and the feature line matching pair.
[0108] The construction unit 103 can also be used to construct a target optimization function based on the point reprojection error and the line reprojection error.
[0109] The calculation unit 104 can be used to calculate the relative pose between the current frame image and the previous frame image based on the target optimization function.
[0110] The positioning unit 105 can be used to perform visual positioning of the target object based on the relative pose, and obtain the positioning information of the target object based on the current frame image.
[0111] In some embodiments, the construction unit 103 may be specifically configured to, for any feature point pair in the feature point matching pair, calculate the three-dimensional coordinates of the first feature point in the world coordinate system based on the pixel coordinates of the first feature point in the previous frame image and the historical relative pose between the previous two frames corresponding to the previous frame image and the current frame image; calculate the projection coordinates of the first feature point in the current frame image based on the three-dimensional coordinates of the first feature point in the world coordinate system, the camera intrinsic parameter matrix, and the scale factor; calculate the reprojection error corresponding to the arbitrary feature point pair based on the pixel coordinates of the second feature point in the current frame image and the projection coordinates of the first feature point in the current frame image; and determine the point reprojection error based on the reprojection error corresponding to the arbitrary feature point pair.
[0112] In some embodiments, the construction unit 103 includes: a first calculation module, a second calculation module, and a determination module.
[0113] The first calculation module can be used to perform two-dimensional reprojection calculation on any one of the feature line pairs in the feature line matching pair to obtain the two-dimensional reprojection error corresponding to the feature line pair.
[0114] The second calculation module can be used to perform three-dimensional reprojection calculation on any one of the feature line pairs to obtain the three-dimensional reprojection error corresponding to the one or more feature line pairs.
[0115] The determining module can be used to determine the line reprojection error between the previous frame image and the current frame image based on the two-dimensional reprojection error and the three-dimensional reprojection error corresponding to any feature line pair.
[0116] In some embodiments, the first calculation module may be specifically configured to: determine the three-dimensional coordinates of the first feature line in the world coordinate system based on the pixel coordinates of the first feature line in the previous frame image of any one feature line pair; determine the projection coordinates of the first feature line in the current frame image based on the three-dimensional coordinates of the first feature line in the world coordinate system; calculate the perpendicular distance between the second feature line and the corresponding projection line segment of the first feature line based on the pixel coordinates of the second feature line in the current frame image of any one feature line pair and the projection coordinates of the first feature line in the current frame image; and determine the two-dimensional reprojection error based on the perpendicular distance between the second feature line and the corresponding projection line segment of the first feature line.
[0117] In some embodiments, the second calculation module may be specifically used to back-project the second feature line into space based on the pixel coordinates of the second feature line in the current frame image to obtain the three-dimensional coordinates of the second feature line in the world coordinate system; based on the three-dimensional coordinates of the second feature line in the world coordinate system and the three-dimensional coordinates of the first feature line in the world coordinate system, calculate the perpendicular distance between the first spatial line segment corresponding to the first feature line and the second spatial line segment corresponding to the second feature line, and the endpoint distance between the first spatial line segment and the second spatial line segment; and based on the perpendicular distance between the first spatial line segment and the second spatial line segment, and the endpoint distance between the first spatial line segment and the second spatial line segment, determine the three-dimensional reprojection error.
[0118] In some embodiments, the apparatus further includes an identification unit and a determination unit.
[0119] The recognition unit can be used to perform target recognition on the current frame image and the previous frame image respectively, and obtain the semantic information corresponding to the current frame image and the semantic information corresponding to the previous frame image.
[0120] The determining unit can be used to determine potential dynamic feature points and potential dynamic feature lines in the current frame image, and potential dynamic feature points and potential dynamic feature lines in the previous frame image, based on the semantic information corresponding to the current frame image and the semantic information corresponding to the previous frame image.
[0121] In some embodiments, the determining unit may also be used to determine potential dynamic feature point matching pairs and potential dynamic feature line matching pairs based on potential dynamic feature points and potential dynamic feature lines in the current frame image and potential dynamic feature points and potential dynamic feature lines in the previous frame image.
[0122] The determining unit can also be used to determine the target feature point matching pair and the target feature line matching pair based on the potential dynamic feature point matching pair and the potential dynamic feature line matching pair.
[0123] In some embodiments, the construction unit 103 may also be specifically used to construct the point reprojection error and line reprojection error between the previous frame image and the current frame image based on the target feature point matching pair and the target feature line matching pair.
[0124] In some embodiments, the determining unit may be specifically configured to: determine static feature point matching pairs and static feature line matching pairs based on the potential dynamic feature point matching pairs and the potential dynamic feature line matching pairs; calculate the fundamental matrix based on the static feature point matching pairs and determine the epipolar line between the current frame image and the previous frame image; calculate the epipolar line distances corresponding to the potential dynamic feature point matching pairs and the potential dynamic feature line matching pairs based on the potential dynamic feature point matching pairs and the potential dynamic feature line matching pairs, as well as the epipolar line and the fundamental matrix; determine the true static feature point matching pairs and the true static feature line matching pairs in the potential dynamic feature point matching pairs based on the epipolar line distances; and determine the target feature point matching pairs and the target feature line matching pairs based on the true static feature point matching pairs and the static feature point matching pairs, as well as the true static feature line matching pairs and the static feature line matching pairs.
[0125] It should be noted that other corresponding descriptions of the functional units involved in the visual positioning device provided in this embodiment can be found in [reference needed]. Figure 1 and Figure 3 The corresponding descriptions in [the document] will not be repeated here.
[0126] Based on the above, Figure 1 and Figure 3 Accordingly, this embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method. Figure 1 and Figure 3 The visual positioning method shown.
[0127] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause an electronic device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.
[0128] Based on the above, Figure 1 and Figure 3 The method shown, and Figure 6 To achieve the above objectives, the present application also provides an electronic device, specifically a personal computer, tablet computer, server, or other network device, as shown in the virtual device embodiment. This device includes a storage medium and a processor; the storage medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figure 1 and Figure 3 The visual positioning method shown.
[0129] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.
[0130] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.
[0131] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.
[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platform, or it can be implemented by hardware.
[0133] This invention significantly enhances image feature extraction capabilities by fusing point and line features, improving the stability and robustness of feature extraction. This overcomes the limitations of feature point extraction in weakly textured environments, thereby improving the accuracy and robustness of visual localization. Simultaneously, this invention utilizes a low-light enhancement network to effectively improve image quality in low-light environments, providing a more reliable foundation for subsequent feature extraction. Furthermore, this invention designs a dynamic feature removal strategy that accurately identifies and removes interference caused by dynamic objects, further improving the accuracy and stability of the system's visual localization.
[0134] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.
[0135] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.
Claims
1. A visual positioning method, characterized in that, include: Obtain the current frame image and the previous frame image corresponding to the current frame image; The feature points and feature lines corresponding to the previous frame image and the current frame image are extracted. Feature point matching and feature line matching are performed on the previous frame image and the current frame image to obtain feature point matching pairs and feature line matching pairs. When performing feature line matching, the direction difference, length ratio and midpoint distance between the feature line corresponding to the previous frame image and the feature line corresponding to the current frame image are calculated respectively. If the direction difference is less than a preset angle, the length ratio is greater than a preset length, and the midpoint distance is less than a preset midpoint distance, then it is determined that the feature line corresponding to the previous frame image matches the feature line corresponding to the current frame image. Based on the feature point matching pairs and the feature line matching pairs, the point reprojection error and line reprojection error between the previous frame image and the current frame image are constructed respectively. The line reprojection error includes two-dimensional reprojection error and three-dimensional reprojection error. When constructing the three-dimensional reprojection error, based on the pixel coordinates of the second feature line in any feature line pair in the current frame image, the second feature line is back-projected into space to obtain the three-dimensional coordinates of the second feature line in the world coordinate system. Based on the three-dimensional coordinates of the second feature line in the world coordinate system and the three-dimensional coordinates of the first feature line in any feature line pair in the world coordinate system, the perpendicular distance between the first spatial line segment corresponding to the first feature line and the second spatial line segment corresponding to the second feature line, as well as the endpoint distance between the first spatial line segment and the second spatial line segment, are calculated. Based on the perpendicular distance between the first spatial line segment and the second spatial line segment, and the endpoint distance between the first spatial line segment and the second spatial line segment, the three-dimensional reprojection error is determined. Based on the point reprojection error and the line reprojection error, construct the objective optimization function; Based on the objective optimization function, the relative pose between the current frame image and the previous frame image is calculated. Based on the relative pose, the target object is visually located to obtain the positioning information of the target object based on the current frame image; The method further includes, after obtaining the current frame image and the corresponding previous frame image, the method further includes: The potential dynamic feature points and potential dynamic feature lines in the current frame image, and the potential dynamic feature points and potential dynamic feature lines in the previous frame image are determined respectively. After performing feature point matching and feature line matching on the previous frame image and the current frame image to obtain feature point matching pairs and feature line matching pairs, the method further includes: Based on the potential dynamic feature points and potential dynamic feature lines in the current frame image, and the potential dynamic feature points and potential dynamic feature lines in the previous frame image, determine the potential dynamic feature point matching pairs and potential dynamic feature line matching pairs. Based on the potential dynamic feature point matching pairs and the potential dynamic feature line matching pairs, static feature point matching pairs and static feature line matching pairs are determined. Based on the static feature point matching pairs, the fundamental matrix is calculated, and the epipolar line between the current frame image and the previous frame image is determined; Based on the potential dynamic feature point matching pair and the potential dynamic feature line matching pair, as well as the epipolar line and the fundamental matrix, calculate the epipolar distances corresponding to the potential dynamic feature point matching pair and the potential dynamic feature line matching pair, respectively. Based on the epipolar distance, the true static feature point matching pairs in the potential dynamic feature point matching pairs and the true static feature line matching pairs in the potential dynamic feature line matching pairs are determined. Based on the real static feature point matching pair and the static feature point matching pair, and the real static feature line matching pair and the static feature line matching pair, the target feature point matching pair and the target feature line matching pair are determined respectively; The step of constructing the point reprojection error and line reprojection error between the previous frame image and the current frame image based on the feature point matching pair and the feature line matching pair includes: Based on the target feature point matching pair and the target feature line matching pair, the point reprojection error and line reprojection error between the previous frame image and the current frame image are respectively constructed.
2. The method according to claim 1, characterized in that, Based on the feature point matching pairs, the point reprojection error between the previous frame image and the current frame image is constructed, including: For any feature point pair in the feature point matching pair, based on the pixel coordinates of the first feature point in the previous frame image and the historical relative pose between the previous two frames corresponding to the previous frame image and the current frame image, calculate the three-dimensional coordinates of the first feature point in the world coordinate system. Based on the three-dimensional coordinates of the first feature point in the world coordinate system, the camera intrinsic parameter matrix, and the scale factor, calculate the projection coordinates of the first feature point in the current frame image; Based on the pixel coordinates of the second feature point in the current frame image of any feature point pair, and the projection coordinates of the first feature point in the current frame image, calculate the reprojection error corresponding to the arbitrary feature point pair. The point reprojection error is determined based on the reprojection error corresponding to any one of the feature points.
3. The method according to claim 1, characterized in that, Based on the feature line matching pairs, the line reprojection error between the previous frame image and the current frame image is constructed, including: For any feature line pair in the feature line matching pair, perform two-dimensional reprojection calculation on the feature line pair to obtain the two-dimensional reprojection error corresponding to the feature line pair. Perform a 3D reprojection calculation on any one of the feature line pairs to obtain the 3D reprojection error corresponding to any one of the feature line pairs; Based on the two-dimensional reprojection error and three-dimensional reprojection error corresponding to any one of the feature line pairs, the line reprojection error between the previous frame image and the current frame image is determined.
4. The method according to claim 3, characterized in that, The step of performing a two-dimensional reprojection calculation on any one of the feature line pairs to obtain the two-dimensional reprojection error corresponding to the any one of the feature line pairs includes: Based on the pixel coordinates of the first feature line in the previous frame image of any one of the feature line pairs, the three-dimensional coordinates of the first feature line in the world coordinate system are determined. Based on the three-dimensional coordinates of the first feature line in the world coordinate system, determine the projection coordinates of the first feature line in the current frame image; Based on the pixel coordinates of the second feature line in the current frame image of any feature line pair, and the projection coordinates of the first feature line in the current frame image, calculate the vertical distance between the second feature line and the projection line segment corresponding to the first feature line. The two-dimensional reprojection error is determined based on the vertical distance between the projection line segments corresponding to the second feature line and the first feature line.
5. The method according to claim 1, characterized in that, The step of determining the potential dynamic feature points and potential dynamic feature lines in the current frame image, and the potential dynamic feature points and potential dynamic feature lines in the previous frame image, includes: Target recognition is performed on the current frame image and the previous frame image respectively to obtain the semantic information corresponding to the current frame image and the semantic information corresponding to the previous frame image; Based on the semantic information corresponding to the current frame image and the semantic information corresponding to the previous frame image, potential dynamic feature points and potential dynamic feature lines in the current frame image, as well as potential dynamic feature points and potential dynamic feature lines in the previous frame image, are determined respectively.
6. A visual positioning device, characterized in that, include: The acquisition unit is used to acquire the current frame image and the previous frame image corresponding to the current frame image; An extraction unit is used to extract feature points and feature lines corresponding to the previous frame image and the current frame image, and to perform feature point matching and feature line matching on the previous frame image and the current frame image to obtain feature point matching pairs and feature line matching pairs. When performing feature line matching, the unit calculates the direction difference, length ratio and midpoint distance between the feature line corresponding to the previous frame image and the feature line corresponding to the current frame image. If the direction difference is less than a preset angle, the length ratio is greater than a preset length, and the midpoint distance is less than a preset midpoint distance, then it is determined that the feature line corresponding to the previous frame image matches the feature line corresponding to the current frame image. A construction unit is configured to construct, based on the feature point matching pairs and the feature line matching pairs, the point reprojection error and the line reprojection error between the previous frame image and the current frame image, respectively. The line reprojection error includes two-dimensional reprojection error and three-dimensional reprojection error. When constructing the three-dimensional reprojection error, based on the pixel coordinates of the second feature line in any feature line pair in the current frame image, the second feature line is back-projected into space to obtain its three-dimensional coordinates in the world coordinate system. Based on the three-dimensional coordinates of the second feature line in the world coordinate system and the three-dimensional coordinates of the first feature line in any feature line pair in the world coordinate system, the perpendicular distance between the first spatial line segment corresponding to the first feature line and the second spatial line segment corresponding to the second feature line, as well as the endpoint distance between the first spatial line segment and the second spatial line segment, are calculated. Based on the perpendicular distance between the first spatial line segment and the second spatial line segment, and the endpoint distance between the first spatial line segment and the second spatial line segment, the three-dimensional reprojection error is determined. The construction unit is further configured to construct a target optimization function based on the point reprojection error and the line reprojection error; The calculation unit is used to calculate the relative pose between the current frame image and the previous frame image based on the target optimization function; The positioning unit is used to perform visual positioning of the target object based on the relative pose, and obtain the positioning information of the target object based on the current frame image; The determining unit is used to determine the potential dynamic feature points and potential dynamic feature lines in the current frame image, and the potential dynamic feature points and potential dynamic feature lines in the previous frame image, respectively. The determining unit is further configured to: determine potential dynamic feature point matching pairs and potential dynamic feature line matching pairs based on potential dynamic feature points and potential dynamic feature lines in the current frame image, and potential dynamic feature points and potential dynamic feature lines in the previous frame image; determine static feature point matching pairs and static feature line matching pairs based on the potential dynamic feature point matching pairs and potential dynamic feature line matching pairs; calculate the fundamental matrix based on the static feature point matching pairs and determine the epipolar line between the current frame image and the previous frame image; calculate the epipolar line distances corresponding to the potential dynamic feature point matching pairs and potential dynamic feature line matching pairs, as well as the epipolar line and the fundamental matrix, based on the epipolar line distances; determine the true static feature point matching pairs and true static feature line matching pairs in the potential dynamic feature point matching pairs and potential dynamic feature line matching pairs; and determine the target feature point matching pairs and target feature line matching pairs based on the true static feature point matching pairs and the static feature point matching pairs, as well as the true static feature line matching pairs and the static feature line matching pairs. The construction unit is specifically used to construct the point reprojection error and line reprojection error between the previous frame image and the current frame image based on the target feature point matching pair and the target feature line matching pair.
7. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.
8. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 5.
Citation Information
Patent Citations
Point-line fusion robot SLAM method and system in dynamic environment
CN117036472A
Downhole positioning method and system based on comprehensive point-line characteristics
CN118031963A