A vslam method and apparatus for indoor low-texture environments

Abstract

The present application relates to the field of machine vision and image processing, and more particularly to a VSLAM method and device for indoor low-texture environment, by acquiring original color images, converting into gray images containing only single-channel information, performing Shi-Tomasi corner feature extraction to obtain point features, performing line feature extraction, and performing near-line merging and short-line rejection on the extracted line features, based on the extracted point features and line features, based on the relative rotation and translation optimization process of the hamilton world assumption, a new pose matrix is constructed through the optimized rotation matrix and optimal translation vector, the obtained pose matrix is used for point feature and line feature reprojection error optimization, positioning and mapping in indoor low-texture environment are realized, not only can high-quality extraction and matching of feature points and feature lines be ensured in indoor low-texture environment, but also structural information can be more efficiently utilized in indoor artificial rule environment to improve the accuracy and robustness of the system.

Description

Technical Field

[0001] This invention relates to the field of machine vision and image processing technology, and in particular to a VSLAM method and apparatus for indoor low-texture environments. Background Technology

[0002] In recent years, with the rapid development of SLAM (Simultaneous Localization and Mapping) technology, camera-sensor-based Visual SLAM (VSLAM) systems have attracted increasing attention from researchers due to their lower hardware costs compared to Lidar SLAM and laser SLAM systems, and the fact that camera sensors can acquire richer information. VSLAM systems primarily rely on multi-view geometric information from the scene, combined with computer vision algorithms, to accurately estimate the robot's position and simultaneously generate a 3D map of the environment. Therefore, it is widely used in 3D visual reconstruction, mobile robotics, 3D holography, visual positioning, AR / VR applications, and autonomous driving technologies such as micro air vehicles (MAVs).

[0003] Currently, researchers have developed VSLAM methods based on various visual sensors, including monocular cameras, stereo binocular cameras, RGB-D cameras, and event cameras. Existing VSLAM methods are mainly divided into feature-based methods and direct methods. Among them, feature-based methods have always been a research focus because they rely on classic computer vision algorithms to extract geometric features and are more robust to changes in lighting compared to direct methods that rely on image pixel values. In addition, feature-based methods require less computational resources than direct methods, as the assumptions required by direct methods are often difficult to meet in general camera sensors and environments, making feature-based methods more popular among researchers. However, in some low-texture scenes of artificial indoor environments, feature-based methods often suffer from uneven distribution or insufficient number of extracted point features, leading to anomalies in VSLAM systems during tracking and potentially causing localization and mapping failures.

[0004] To address this challenge, many studies have introduced other geometric elements from multi-view geometry (such as line segments, planes, or vanishing points) into VSLAM systems, significantly improving system robustness. These geometric elements can provide additional information for feature-sparse scenes, thereby improving the system's performance in low-texture environments. Typically, many indoor environments are man-made, regular structures from which three mutually orthogonal line segments can be extracted, validating the general Manhattan world assumption. However, most existing structured VSLAM methods rely on the global Manhattan world assumption, requiring vanishing points to be estimated for each frame. This assumption is not always applicable in real-world indoor environments, potentially leading to computational overhead and poor optimization results. Furthermore, global Manhattan world optimization often depends on initial frame estimation, making it susceptible to initial errors in subsequent optimizations. These methods do not optimize or fully utilize point features in low-texture environments, often relying on traditional processing methods or discarding them entirely. In most scenarios, the number and matching success rate of line features are typically much lower than those of point features; therefore, line features are primarily used as supplementary means to address the lack of sufficient point features. Summary of the Invention

[0005] In view of this, the purpose of this invention is to propose a VSLAM method and apparatus for indoor low-texture environments, so as to solve the problems of limitations and insufficient robustness of traditional feature-based visual SLAM in indoor low-texture environments.

[0006] To achieve the above objectives, the present invention provides a VSLAM method for indoor low-texture environments, comprising the following steps:

[0007] S1. Obtain the original color image and convert it into a grayscale image containing only single-channel information;

[0008] S2. Extract Shi-Tomasi corner features from the grayscale image to obtain point features;

[0009] S3. Extract line features from the grayscale image, and merge near lines and remove short lines from the extracted line features;

[0010] S4. Based on the extracted point and line features, and the relative rotation and translation optimization process based on the Hammanton world hypothesis, a new pose matrix is ​​constructed using the optimized rotation matrix and the optimal translation vector.

[0011] S5. The obtained pose matrix is ​​used to optimize the reprojection error of point features and line features to achieve localization and mapping in indoor low-texture environments.

[0012] Preferably, the grayscale image is subjected to Shi-Tomasi corner feature extraction to obtain point features including:

[0013] Calculate the horizontal gradient I of the grayscale image. x Image gradient I in the vertical direction y The autocorrelation matrix Q of the grayscale image is calculated through the image gradient, and the minimum eigenvalue is calculated for the autocorrelation matrix Q.

[0014]

[0015] `window` refers to the size of a local window, and the response function `R` of the Shi-Tomasi corner detection algorithm is used. Shi-Tomasi The minimum eigenvalue is used to determine whether a corner point at a certain location in a grayscale image is a valid Shi-Tomasi corner point. The response function R... Shi-Tomasi Typically, a specific threshold is used, and λ1 and λ2 are the eigenvalues ​​calculated from the autocorrelation matrix Q;

[0016] R Shi-Tomasi =min(λ1,λ2).

[0017] Preferably, the Shi-Tomasi corner feature extraction of the grayscale image employs a gridding and pyramid layering extraction strategy. Compared to the baseline version ORB-SLAM3, this extraction strategy increases the grid size, halves the number of pyramid layers, and halves the maximum number of corner points to extract.

[0018] Preferably, the extraction of line features involves merging near lines and removing short lines, including:

[0019] A near-line merging and short-line elimination algorithm based on the LSD line segment extractor is used to perform near-line merging and short-line elimination.

[0020] Preferably, before the relative rotation and translation optimization process based on the Harmanton world assumption, this method further includes a relative rotation optimization pre-step execution logic, specifically including:

[0021] First, determine if the current frame ID is a multiple of a certain integer to achieve intermittent Manhattan vanishing point estimation. If Manhattan vanishing point estimation is performed on the current frame, then determine if the line clusters clustered according to the Manhattan vanishing point in the current frame meet the condition that all three types of line clusters exist and each line cluster contains at least three line segments. If this condition is met, calculate the main direction curdk of the line cluster in the current frame and determine if the main direction curdk of the line cluster in the current frame and the main direction lastdk of the line cluster in the previous frame both exist, and the frame ID interval between the frames containing these two main directions of the line cluster cannot be greater than 15 frames. If all the above conditions are met, perform relative rotation optimization.

[0022] Preferably, in step S4, the relative rotation process includes:

[0023] For each line cluster clustered through the Manhattan vanishing point, first calculate the normal vector of each line segment in the current camera coordinate system. Then, calculate the direction vector l of each line segment by performing a cross product operation on the start and end points in the aligned sub-coordinate form. Normalize this direction vector l, and then pass the processed line segment direction vector l through the transpose K of the camera intrinsic parameter matrix K. T The transformation is performed to obtain the normal vector s of the line segment in the current camera coordinate system. i Then, normalize it to construct a matrix S containing the normal vectors of all line segments in the three line clusters;

[0024] Transpose matrix S and perform SVD singular value decomposition to obtain three directions that are orthogonal to the normal vectors of all line segments in the three line clusters, namely the main directions of the three line clusters, and obtain the main direction of the line cluster curdk of the current frame.

[0025] After each calculation of curdk, the system saves and updates it to construct a pose-relative rotation optimization method between two frames.

[0026] Preferably, the pose-relative rotation optimization method between two frames includes:

[0027] For the previous main direction of the line cluster, lastdk, ​​calculate its transformation direction in the current frame:

[0028]

[0029] Where R is the rotation matrix of the current frame, and k represents the main direction index of the three different line clusters. It is the inverse of the rotation matrix of the frame containing the main direction of the previous line cluster. Let lastdk be the principal direction of the previous line cluster. This is calculated... The relative rotation angle error is defined by the change in angle relative to the current Manhattan vanishing point direction δ. A prior method based on angle weights and distance weights is adopted to adjust the internal arrangement order of curdk.

[0030] Based on the cost function for minimizing the overall relative rotation error, the corresponding formula for calculating the Jacobian matrix is ​​as follows:

[0031]

[0032] Since the derivative of the translation part is 0, the final Jacobian matrix is ​​equal to the derivative of the translation part for each error term θ. i It is a 1×6 matrix, where the first 3 columns correspond to the translation part and the last 3 columns correspond to the rotation part. The final Jacobian matrix is ​​expressed as:

[0033]

[0034] The complete Jacobian matrix representing the rotation error term under each distinct pairwise orthogonal direction is expressed as:

[0035]

[0036] The relative rotation error is defined as an edge in the Levenberg-Marquardt optimization algorithm within G2O and added to the vertex defined by the current frame pose. This indicates that the relative rotation error imposes an optimization constraint on the current frame pose. The three main directions of the line clusters used in the optimization have the same weight, indicating that they have equal importance in the optimization process, thus obtaining the optimized rotation matrix R. init .

[0037] Preferably, in step S4, the translation optimization process includes:

[0038] Define a random number generator to randomly select feature point indices in each iteration, and set an appropriate maximum number of iterations and an interior point threshold. In each iteration, query whether the randomly selected feature point index is a feature point that is associated with a valid graph point.

[0039] After selecting feature points that are effectively associated with the image points, a linear least squares problem is constructed. First, the image coordinates k of randomly selected feature points are obtained. p and the world coordinates X of its associated map points. w Then, based on the principal point coordinates (c) in the camera intrinsic parameter matrix x ,c y The coordinates (u, v) of the random feature point relative to the camera optical center are calculated as follows:

[0040] u=k p .xc x v=k p .xc y

[0041] Next, based on the projection model, the matrix M and vector n required for solving the least squares problem are constructed:

[0042]

[0043] Among them, M T f is the transpose of matrix M. x and f y σ is the focal length of the camera. -2 X represents the weights of the image pyramid corresponding to random feature points. w Y w and Z w Given the world coordinates of map points associated with random feature points, solve the following equations using the least squares method to obtain the optimal translation vector t.init ;

[0044] t init =(M T M) -1 M T n.

[0045] Preferably, step S5 further includes:

[0046] This invention employs bundle adjustment technique to globally optimize the poses of keyframes, map points, and map lines. It calculates the three-dimensional coordinates of map points and map lines through triangulation and uses global bundle adjustment technique to optimize the poses of keyframes, map points, and map lines, thereby improving the accuracy of the map.

[0047] The present invention also provides a VSLAM apparatus for indoor low-texture environments, for performing the VSLAM method for indoor low-texture environments described above.

[0048] The beneficial effects of this invention are:

[0049] Compared to the benchmark ORB-SLAM3, the method of this invention achieved an average 29.89% improvement in APE (RMSE) trajectory accuracy across several experimental EuRoC sequences. Furthermore, for indoor low-texture environments such as V201, the method of this invention also achieved an average 59.85% improvement in trajectory accuracy compared to state-of-the-art algorithms. Compared to other VSLAM methods, this invention not only ensures high-quality extraction and matching of feature points and feature lines in indoor low-texture environments but also more efficiently utilizes structural information to improve the system's accuracy and robustness in indoor artificially constructed rule environments. The method of this invention demonstrates superior performance in indoor low-texture environments, making a significant contribution to the development of VSLAM systems applied in this scenario. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the overall framework of an embodiment of the present invention;

[0052] Figure 2 This is a comparison image of the improved FAST corner extraction and Shi-Tomasi corner extraction according to an embodiment of the present invention;

[0053] Figure 3This is a schematic diagram of the nearline merging operation logic according to an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram illustrating the use of the Manhattan world vanishing point for relative rotation optimization of camera frame pose in an embodiment of the present invention.

[0055] Figure 5 This is a flowchart of the preliminary steps for relative rotation optimization according to an embodiment of the present invention;

[0056] Figure 6 A visual comparison chart showing relative rotation optimization performed when the conditions of an embodiment of the present invention are met;

[0057] Figure 7 These are partial indoor low-texture environment images from embodiments of the present invention;

[0058] Figure 8 The above are line graphs and trajectory diagrams showing the APE accuracy error compared to the actual trajectory in this embodiment of the invention.

[0059] Figure 9 This is a comparison result of all point features between two frames in an embodiment of the present invention.

[0060] Figure 10 This is a comparison result of high-quality point feature matching between two frames in an embodiment of the present invention.

[0061] Figure 11 The image shows a comparison of the improved line feature extraction results in this embodiment of the invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0063] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0064] This specification provides an embodiment of a VSLAM method for indoor low-texture environments, the overall process of which is as follows: Figure 1 As shown, it includes:

[0065] 1. Input image

[0066] First, the VSLAM system captures the original color image from the stereo camera and converts it into a grayscale image containing only single-channel information.

[0067] 2. Extract Shi-Tomasi point features from the input image using the VSLAM system.

[0068] To address the issue that the improved VSLAM system's point feature extraction module may fail to balance robustness and real-time performance when extracting point features in low-texture indoor environments, this invention proposes a method for extracting front-end point features of a meshed multi-layer pyramid based on the Shi-Tomasi corner detection algorithm.

[0069] First, for the input grayscale image, the front-end point feature extraction part of the VSLAM system of this invention calculates the image gradient I in the horizontal direction. x Image gradient I in the vertical direction y The autocorrelation matrix Q of the grayscale image is calculated through the image gradient. The minimum eigenvalue is calculated for the autocorrelation matrix Q. The window is the size of the local window (usually 3×3 pixels).

[0070]

[0071] λ1 and λ2 are the eigenvalues ​​calculated using the autocorrelation matrix Q, and are expressed by the response function R of the Shi-Tomasi corner detection algorithm. Shi-Tomasi The minimum eigenvalue is used to determine whether a corner point at a certain location in a grayscale image is a valid Shi-Tomasi corner point. The response function R... Shi-Tomasi It is usually a defined threshold.

[0072] R Shi-Tomasi =min(λ1,λ2) (2)

[0073] To ensure the uniform distribution and real-time performance of the improved foreground feature extraction corner points in the VSLAM system of this invention, a gridding and pyramid layering extraction strategy is introduced. This ensures that a sufficient number of Shi-Tomasi corner points can be uniformly extracted within the grayscale image, thereby reducing matching failures caused by drastic camera movement. In ORB-SLAM3, ORB feature point extraction relies on the detection of FAST corner points in the image. Although FAST corner detection is known for its fast detection speed and ability to extract a large number of corner points in a short time, making it very suitable for real-time applications, its performance in low-texture environments is often unsatisfactory. In low-texture environments, due to the scarcity and uneven distribution of distinguishable feature points in the image, FAST corner points are easily affected by noise, leading to unstable extracted feature points and even a large number of false matches, which can significantly affect the robustness and accuracy of the system. While FAST corner points can play an important role in applications with high real-time requirements, their dependence on feature points and limitations become apparent in low-texture environments.

[0074] In contrast, the Shi-Tomasi corner detection algorithm is an improvement on the classic Harris corner detection algorithm[]. It primarily works by weighting the gradient information of local image regions, thereby more accurately detecting stable corners in the image. In low-texture environments, although there are fewer detectable feature points in the image, the Shi-Tomasi algorithm exhibits stronger feature stability. This means it can effectively avoid FAST corner detection, which extracts incorrect and noisy point features in areas with limited texture. More importantly, the stability of Shi-Tomasi corners ensures that the system maintains relatively stable tracking and localization even in environments with sparse feature points, thus reducing matching failures and trajectory drift caused by low-quality feature points in low-texture environments.

[0075] To improve feature point stability while ensuring the real-time performance of the VSLAM system, this invention optimizes the Shi-Tomasi corner extraction algorithm and simplifies the meshing and pyramid layering strategies in ORB-SLAM3. Specifically, this invention appropriately increases the mesh size, reducing the computational burden by decreasing the number of mesh cells, thereby improving the speed of feature point extraction. Simultaneously, this invention reduces the number of pyramid layers, reducing the overhead of multi-scale image processing. These simplifications aim to ensure stable system operation in low-texture environments by reducing computational complexity. Although Shi-Tomasi corner extraction consumes more computational resources than the traditional FAST corner extraction method, its high-quality feature points allow this invention to reduce the maximum number of feature points extracted, further improving the quality of extracted points and maintaining a uniform distribution, while ensuring real-time performance requirements.

[0076] Through these optimizations, this invention effectively improves localization accuracy in low-texture environments while maintaining computational overhead close to the baseline version. These adjustments not only enhance the robustness of the VSLAM system, enabling it to cope with the challenges of low-texture environments, but also ensure that the improved system maintains good real-time performance and reliability while providing higher localization accuracy. Therefore, the introduction and related optimizations of Shi-Tomasi corner points successfully address the challenges of point feature extraction in low-texture environments, effectively improving the overall performance of the VSLAM system.

[0077] like Figure 2 As shown, the number of point features extracted and the number of pyramid layers in the feature extraction pyramid of the present invention are half that of the original ORB-SLAM3 version.

[0078] 3. Perform near-line merging and short-line removal on the line features extracted from the VSLAM system.

[0079] This invention's VSLAM method adds line feature extraction and matching processes to the benchmark VSLAM method. Addressing the problem that traditional LSD detectors easily cause mismatches and increased computational overhead when processing scenes with rich line textures, redundant line segments in this invention can lead to mismatches of line features. This invention proposes a near-line merging and short-line removal algorithm based on an LSD line segment extractor. The aim is to filter out redundant line features through post-processing steps, thereby reducing mismatches of line features in the VSLAM system and reducing the time required for the VSLAM system to compute LBD descriptors for line features. This algorithm strategy implements near-line merging and short-line removal operations through several steps as shown in Table 1.

[0080] Table 1Algorithm:LineFeatureProcessing

[0081]

[0082]

[0083] The logic of this invention for merging near-line features extracted from the front end can be achieved through... Figure 3 To express.

[0084] exist Figure 3 In the diagram, line segments A and B represent the line segments that this invention will determine whether to merge. L1, L2, L3, and L4 are the distances between the endpoints of line segments A and B, d is the distance between the midpoint of line segment A and line segment B, and θ is the angle between the line segments calculated using the direction vector.

[0085] The VSLAM system front-end feature extraction process is completed in two and three steps.

[0086] This invention selects the LSD (Line Segment Detector) as the method for extracting line segment features. LSD is a parameter-free, high-precision line segment detection method that not only boasts strong real-time performance but also effectively preserves the details and geometric features of line segments in an image, making it the most widely used line segment extractor in current VSLAM methods. This invention selects the LBD (Line Digitization Descriptor) as the descriptor for line segment features. The LBD descriptor is specifically designed for line segment features and possesses characteristics such as rotation invariance, high computational efficiency, and strong robustness, making it suitable for line segment matching and feature description in VSLAM systems.

[0087] However, in low-texture environments, VSLAM systems often face problems of sparse features and inaccurate matching, especially in the extraction and matching of online features. Although the Line Segment Extractor (LSD) can effectively extract line segment features from images, in low-texture environments, images often lack sufficient explicit features, resulting in potentially few and low-quality extracted line segments. Furthermore, line segments in low-texture environments often exhibit irregular distributions, and the LSD detector may detect redundant line segments or mismatches, thus impacting system performance.

[0088] To address the aforementioned issues, the near-line merging and short-line culling algorithm proposed in this invention plays a crucial role. The core of this algorithm is to filter detected line segments through post-processing steps, eliminating redundant features and improving the matching quality of line features. Through near-line merging, this invention can merge adjacent line segments with similar directions into a longer and more stable line segment, thereby reducing matching errors caused by short and discontinuous line segments. Simultaneously, short-line culling effectively removes unrepresentative short-line features, which is particularly important in low-texture environments because short-line features are often more susceptible to noise and edge effects. These processing steps significantly improve the robustness and localization accuracy of the system by reducing useless redundant features and false matches. Specifically, near-line merging reduces the possibility of false matches, enabling more stable matching of line segment features in low-texture environments, while short-line culling reduces the time required to compute LBD descriptors, thus improving the real-time performance of the system.

[0089] 4. Constructing an optimization method for relative rotation and translation based on the Manhattan world.

[0090] After completing the front-end feature extraction process of the VSLAM system, a relative rotation and translation optimization process based on the Manhattan world hypothesis will be carried out.

[0091] To ensure the real-time performance and effectiveness of the VSLAM system, this invention employs a fast Manhattan vanishing point estimation method based on two line segments. This method quickly and accurately estimates the Manhattan vanishing point in space by extracting two random line segments from the input image, and obtains the globally optimal set of Manhattan vanishing points through detailed search.

[0092] Based on the vanishing point estimation method of the Manhattan world hypothesis mentioned above, this invention proposes the following relative rotation and translation optimization method based on the Manhattan world.

[0093] Figure 4 This is a schematic diagram for optimizing the relative rotation of camera frames using the Manhattan World Vanishing Point.

[0094] This invention integrates the Manhattan vanishing point estimation method into the frame construction process of the front-end feature extraction part of the proposed VSLAM system, i.e., a subsequent process after feature extraction. Simultaneously, to ensure system real-time performance and avoid forcibly using the relative rotation and relative translation optimization method based on the Manhattan vanishing point in non-Manhattan structure environments, this invention designs a relative rotation optimization pre-step execution logic, the flowchart of which is shown below. Figure 5 As shown.

[0095] In the execution logic of the pre-step relative rotation optimization, it is first determined whether the current frame ID is a multiple of a certain integer to achieve intermittent Manhattan vanishing point estimation. If Manhattan vanishing point estimation is performed on the current frame, it is determined whether the line clusters clustered according to the Manhattan vanishing point in the current frame meet the condition that all three types of line clusters exist and each line cluster contains at least three line segments. If this condition is met, the main direction curdk of the line cluster in the current frame is calculated, and it is determined whether both the main direction curdk of the current frame and the main direction lastdk of the previous line cluster exist, and the frame ID interval between the frames containing these two main directions of the line cluster cannot be greater than 15 frames. Finally, only when all the above conditions are met is the relative rotation optimization within this invention confirmed to be executed.

[0096] When the system of this invention performs relative rotation optimization, three different colored line segments will appear in the visualization window interface, representing line segments in different line clusters. Figure 6 The visualization windows show the windows with and without relative rotation optimization.

[0097] The following section details the process of calculating the principal direction curdk for each line cluster. For each line cluster clustered through the Manhattan vanishing point, this invention first calculates the normal vector of each line segment in the line cluster in the current camera coordinate system. Specifically, for each line segment starting point p... s and the endpoint p e The present invention converts the pixel coordinates in the image into homogeneous coordinates.

[0098]

[0099] Where, x s and y s Let x be the two-dimensional pixel coordinate of the starting point of the line segment. e and y e Let be the two-dimensional pixel coordinates of the endpoint of the line segment. Next, the direction vector l of each line segment is calculated by performing a cross product operation on the start and end points in the form of aligned coordinates, and then the direction vector l is normalized.

[0100]

[0101] The processed line segment direction vector l is passed through the transpose K of the camera intrinsic parameter matrix K. T The transformation is performed to obtain the normal vector s of the line segment in the current camera coordinate system. i And normalize it.

[0102]

[0103] This invention will construct a matrix S containing the normal vectors of all line segments in 3 line clusters. Assuming that there are n line segments in each line cluster, matrix S can be represented as a 3×n matrix.

[0104]

[0105] Among them, s nn Let S be the normal vector of the nth line segment in the nth type of line cluster. Then, by transposing the matrix S and performing SVD singular value decomposition, we can obtain three directions that are orthogonal to the normal vectors of all line segments in the three line clusters, which are the main directions of the three line clusters. Thus, this invention can obtain the main direction of the line cluster curdk in the current frame.

[0106] Each time the curdk is calculated, the system saves and updates it, retaining at most the last line cluster main direction lastdk. When this invention obtains the current Manhattan vanishing point direction vps, the current line cluster main direction curdk, the last line cluster main direction lastdk, ​​and the inverse of the camera pose rotation matrix of the frame containing the last line cluster main direction, this invention can construct a pose relative rotation optimization method between two frames.

[0107] Specifically, this invention defines the error of the relative rotation optimization method as a nonlinear least squares problem based on the Levenberg-Marquardt optimization framework. For the previous line cluster principal direction lastdk, ​​this invention first calculates its transformation direction in the current frame:

[0108]

[0109] Where R is the rotation matrix of the current frame, and k represents the main direction index of the three different line clusters. It is the inverse of the rotation matrix of the frame containing the main direction of the previous line cluster. Let lastdk be the main direction of the previous line cluster. This invention can calculate using formula (8). The relative rotation angle error is defined by the change in angle with the current Manhattan vanishing point direction δ. Since the Manhattan vanishing point estimation in this invention is random, the arrangement within the Manhattan vanishing point direction δ is prone to randomness, which leads to randomness in the arrangement within curdk when calculating the main direction of the line cluster, which is not conducive to error calculation. Therefore, a prior method based on angle weight and distance weight is adopted to adjust the arrangement order within curdk. This prior method calculates the angle difference of the Manhattan vanishing point vector and the Euclidean distance of the Manhattan vanishing point between the currently estimated Manhattan vanishing point direction vps and the previous Manhattan vanishing point direction lastvps. By continuously sorting the vectors within the currently estimated Manhattan vanishing point direction vps, the combination with the smallest change is obtained, which becomes the new Manhattan vanishing point direction vps. Formula (9) is also introduced to a certain extent to avoid large angle errors caused by errors in the prior method adjustment. Formula (10) is the embodiment of the cost function for minimizing the overall angle error.

[0110]

[0111] Where δ represents the current direction of the Manhattan vanishing point. This represents the transpose of the direction vector of the i-th Manhattan vanishing point, where i represents the indices of the three distinct Manhattan vanishing point directions. θ represents the main direction of the line cluster with the same index as the Manhattan vanishing point. i This represents the relative rotation angle error for each pair of distinct mutually orthogonal directions. Representative to Calculate its antisymmetric matrix. Based on the cost function of minimizing the overall relative rotation error, the corresponding Jacobian matrix calculation formula can be derived in this invention.

[0112]

[0113] Since the derivative of the translation part is 0, the final Jacobian matrix is ​​equal to the derivative of the translation part for each error term θ. i It is a 1×6 matrix, where the first 3 columns correspond to the translation part (the zero vector), and the last 3 columns correspond to the rotation part. The following is the final Jacobian matrix.

[0114]

[0115] The complete Jacobian matrix representing the rotation error term under each distinct pairwise orthogonal direction can be expressed as:

[0116]

[0117] The relative rotation error is defined as an edge in the Levenberg-Marquardt optimization algorithm within G2O and added to the vertex defined by the current frame pose. This indicates that the relative rotation error imposes an optimization constraint on the current frame pose. The three main directions of the line clusters used in the optimization have the same weight, indicating that they have equal importance in the optimization process. Based on the above method, the optimized rotation matrix R can be obtained. init .

[0118] Obtain the optimized rotation matrix R init Furthermore, this invention will define a strategy for optimizing translation vectors based on the Random Sample Consensus Algorithm (RANSAC) framework. First, a random number generator is defined to randomly select feature point indices in each iteration, with appropriate maximum iteration counts and interior point thresholds set. In each iteration, the randomly selected feature point index is used to query whether it is a feature point that is associated with a valid graph point.

[0119] Once feature points that are effectively associated with the graph points are selected, this invention will construct a linear least squares problem. First, obtain the image coordinates k of randomly selected feature points. p and the world coordinates X of its associated map points. w Then, based on the principal point coordinates (c) in the camera intrinsic parameter matrix... x ,c y This invention can calculate the coordinates (u,v) of a random feature point relative to the camera's optical center, as shown in the following formula:

[0120] u=k p .xc x v=k p .xc y (14)

[0121] Next, this invention will construct the matrix M and vector n required for solving the least squares problem based on the projection model.

[0122]

[0123] Among them, M T f is the transpose of matrix M. x and f y σ is the focal length of the camera. -2 X represents the weights of the image pyramid corresponding to random feature points. w Y w and Z wLet t be the world coordinates of the map points associated with the random feature points. This invention will solve the following equation using the least squares method to obtain the optimal translation vector t. init .

[0124] t init =(M T M) -1 M T n (17)

[0125] For the selected valid feature points in the current frame, i represents the index of the current valid feature point. For the world coordinates of the map point associated with the feature point at index i, this invention will use the rotation matrix R obtained based on the above method. init Translation vector t init To calculate the coordinates of the map points associated with these feature points in the current camera coordinate system:

[0126]

[0127] This invention will also calculate the projection error and statistically analyze the current rotation matrix R based on the projection error. init Translation vector t init The number of interior points under u i and v i This represents the coordinates of the feature point with index i relative to the camera's optical center. The projection error is calculated using the following formula:

[0128]

[0129] The sum of squares of the overall projection errors is:

[0130]

[0131] If the error is less than a set threshold, the feature point is considered an interior point. If the total number of interior points in the current iteration exceeds the previous optimal result, the translation vector t calculated using the least squares method is updated. init This is the optimal translation vector.

[0132] Optimized rotation matrix R init and the optimal translation vector t init A new pose matrix T will be constructed. init The pose matrix T init In the VSLAM system of this invention, it will be used as the initial pose for subsequent reprojection error optimization, so as to reduce the adverse effects of the Manhattan world's mandatory constraints and improve the accuracy and real-time performance of the reprojection error.

[0133] 5. Motion Estimation and Map Optimization

[0134] The pose matrix T from the above steps init For optimizing reprojection errors of point and line features, the VSLAM system can quickly acquire accurate current camera pose and determine keyframes. This invention employs bundle adjustment technology to globally optimize the poses of keyframes, map points, and map lines. Furthermore, the system calculates the 3D coordinates of map points and lines through triangulation and uses global bundle adjustment to optimize the poses of keyframes, map points, and map lines, thereby improving map accuracy. When the system detects loop closures, it utilizes previous keyframes and inactive maps to optimize the poses of the current keyframe and active map, thus reducing accumulated errors. Even if keyframes are lost during tracking, the system can quickly recover and continue tracking through relocalization and map fusion techniques. The combined application of these technologies enables the VSLAM system of this invention to achieve efficient and stable localization and mapping in indoor low-texture environments.

[0135] To test the performance of the proposed VSLAM system in terms of accuracy and real-time performance, this invention uses the EuRoC dataset (European Robotics Challenge Dataset) released by the Micro-UAV Laboratory (ASL) at ETH Zurich to compare the system with the ORB-SLAM3 system. The EuRoC dataset is a widely used dataset for evaluating VSLAM systems and visual inertial odometry (VIO) algorithms. It contains high-quality datasets collected by micro-unmanned aerial vehicles (MAVs) equipped with cameras and IMU sensors during flight in indoor and industrial environments, including indoor factories and rooms. These environments possess the complexity of real-world scenes, such as varying lighting, low-texture areas, and rapid movement.

[0136] The experimental environment for this embodiment was a laptop computer equipped with an Intel(R) Core(TM) i9-10980HK@2.4GHz 16-core CPU. All experiments were conducted on an 8-core Ubuntu 18.04 system within a VMware virtual machine. In evaluating trajectory accuracy, this invention used the EVO tool to perform an index-based evaluation of the accuracy of ORB-SLAM3 and the proposed method. This invention selected the root mean square error (RMSE) of the absolute trajectory error (APE) as the standard for judging the trajectory accuracy of this invention. The absolute trajectory error (APE) can be expressed by the following formula.

[0137]

[0138] in, Let T represent the estimated camera pose at time i. i The error e represents the actual camera pose at time i. i This represents the difference between the estimated camera pose and the actual camera pose. To obtain a measure of the overall error, this invention uses the root mean square (RMSE) value of the APE, as shown in the following formula.

[0139]

[0140] Where n is the total number of poses, RMSE evaluates the overall error by averaging the squares of the errors at each time step and then taking the square root. The smaller the value, the more accurate the estimated trajectory.

[0141] 1. Comparison of trajectory accuracy

[0142] This invention selects a portion of sequences from the well-known EuRoC dataset for experimental testing and comparison. The selected sequences include grayscale images of the left and right eyes and IMU data. Each image has a size of (752,480), an image frame rate of 20Hz, an IMU frame rate of 200Hz, and contains real trajectories. Figure 7 This demonstrates the challenging low-texture portions of the experimental scenes within the selected sequence.

[0143] Table 2 summarizes the APE (RMSE) trajectory accuracy of various methods for the selected sequences. The methods in this embodiment, OpenVINS, and ORB-SLAM3 were run in binocular IMU mode, while UV-SLAM and UL-SLAM were run in monocular IMU mode. The best experimental results are highlighted in bold in the table.

[0144] Table 2 Comparison of APE (RMSE) Accuracy

[0145]

[0146]

[0147] The data in Table 2 show that, compared to ORB-SLAM3, the method of this invention achieves an average overall accuracy improvement of 29.89%. Even when evaluating different improved modules individually, the trajectory accuracy of each module is improved compared to ORB-SLAM3. In dark environments where point features are difficult to extract sufficiently, such as the MH05 dataset, even though the trajectory accuracy of the method of this invention is not as good as the state-of-the-art UL-SLAM, it still achieves an accuracy improvement of 48.53% compared to ORB-SLAM3. In indoor environments, due to factors such as white walls and specular reflections, low-texture scenes often occur. The method of this invention, by combining local Manhattan vanishing point information, can also provide more accurate initial pose estimation for reprojection error optimization in complex or atypical Manhattan structure indoor environments. Therefore, the trajectory accuracy also achieves improvements of 18.91% and 27.29% respectively compared to the suboptimal ORB-SLAM3 algorithm in the indoor V201 and V203 datasets. These results verify the adaptability and accuracy improvement of the method of this invention in different scenarios.

[0148] Furthermore, this invention also utilizes the EVO tool to compare the estimated trajectories of ORB-SLAM3 and the method of this embodiment with the actual trajectories on the MH05 dataset, such as... Figure 8 As shown in the figure. The figure includes an APE accuracy error line graph and a trajectory graph, with the dashed line representing the true value.

[0149] Depend on Figure 8 As can be seen from (a) and (b), the absolute trajectory error of ORB-SLAM3 is mainly concentrated between 0.1 and 0.15, while the method of this invention further reduces the error to between 0.06 and 0.08. Furthermore, Figure 8 (c) and (d) further verify the accuracy advantage of the method of the present invention in low-light scenes. Compared with the ground truth, the trajectory of ORB-SLAM3 shows a significant error, which may reach about 0.2, while the maximum trajectory error of the method of the present invention remains at about 0.1, and the error of some paths is reduced by about 50%. These results fully demonstrate the accuracy improvement of the method of the present invention under low-light conditions.

[0150] 2. Comparison of point feature matching performance and line feature extraction performance

[0151] This invention conducted experiments on improved point feature matching using two adjacent frames of images selected from the EuRoC dataset MH01. Figure 9-10 The image above shows all point matching images and high-quality point matching images under the improved point feature matching method used in ORB-SLAM3 and this embodiment. The maximum number of extracted feature points is set to 600, and the threshold settings are the same as those in ORB-SLAM3 and this embodiment.

[0152] exist Figure 9-10 The improvement in feature matching performance is very significant. Specifically, in all point matching results, the FAST corner extraction method yielded 11 matched points, while the Shi-Tomasi corner extraction method significantly increased the number of matched points to 130, demonstrating the significant advantage of Shi-Tomasi corners in low-texture environments, representing an improvement of approximately 12 times. For high-quality point matching, the FAST corner extraction method yielded 6 matched points, while the Shi-Tomasi method increased the number of matched points to 97, an improvement of approximately 16 times, showing a significant improvement in feature point matching quality. We determine which feature point matches are high-quality by calculating the Hamming distance between the descriptors of the matched point features and comparing it with the minimum distance. More high-quality point feature matches help improve the stability of the VSLAM system, thereby improving tracking accuracy and landmark estimation accuracy.

[0153] Figure 11 This invention demonstrates the effectiveness of improved line feature extraction. The method of this invention filters out many redundant short line segments and near-line segments, retains long line segments that are more conducive to inter-frame matching, reduces the computational overhead of the line feature part, and improves the stability of line feature matching.

[0154] 3. Real-time analysis

[0155] This invention also conducted real-time experiments and analyses on ORB-SLAM3 and the method of this invention. By running ORB-SLAM3 and the method of this invention on the EuRoC MH01 dataset and statistically analyzing the time consumed by each module, this invention demonstrates the real-time performance of the proposed method by comparing the overall time consumption with ORB-SLAM3 and the time consumption between each module.

[0156] Table 3 Comparison of average time consumption per frame

[0157]

[0158]

[0159] In Table 3, the time consumption unit is ms / f, which means the average time consumed per frame. The addition of line features and Manhattan optimization has a certain impact on the time consumption of the method in this invention, but compared with ORB-SLAM3, the system of this invention can still meet the real-time requirements overall.

[0160] The method of this invention consumes an average of 71.40 ms per frame in the tracking module, compared to 31.99 ms for ORB-SLAM3, showing a slight increase. This is because the present invention introduces line feature matching and vanishing point estimation during the tracking process. However, the time consumption of the tracking module is still within an acceptable range and meets the requirements of a real-time vSLAM system. In the local mapping module, the method of this invention consumes 377.56 ms, an improvement compared to 278.74 ms for ORB-SLAM3. The main reason for the increased time is that the present invention employs more complex line feature processing and Manhattan optimization, increasing the accuracy and complexity of the local map. The performance improvement brought by these enhancement strategies is sufficient to compensate for the additional time overhead. The time consumption of the loop closure detection module is relatively low, with the method of this invention averaging only 1.19 ms per frame, compared to 4.35 ms for ORB-SLAM3. The method of this invention demonstrates higher efficiency in loop closure detection.

[0161] Overall, ORB-SLAM3 consumes an average of 315.08 ms per frame, while the method of this invention consumes 450.15 ms. Despite the increase of nearly 135 ms in computation time, the system of this invention, by introducing more geometric information and optimization strategies, effectively improves positioning accuracy and robustness, while still maintaining relatively good real-time performance.

[0162] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention is limited to these examples; within the framework of the invention, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity. Any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A VSLAM method for indoor low-texture environments, characterized in that, Includes the following steps: S1. Obtain the original color image and convert it into a grayscale image containing only single-channel information; S2. Extract Shi-Tomasi corner features from the grayscale image to obtain point features; S3. Extract line features from the grayscale image, and merge near lines and remove short lines from the extracted line features; S4. Based on the extracted point and line features, and the relative rotation and translation optimization process based on the Manhattan world hypothesis, a new pose matrix is ​​constructed using the optimized rotation matrix and the optimal translation vector. S5. The obtained pose matrix is ​​used to optimize the reprojection error of point features and line features to achieve localization and mapping in indoor low-texture environments. The Shi-Tomasi corner feature extraction of the grayscale image adopts a gridding and pyramid layer extraction strategy. Compared with the baseline version ORB-SLAM3, the extraction strategy increases the grid size, halves the number of pyramid layers, and halves the maximum number of corner points to be extracted. Before the relative rotation and translation optimization process based on the Manhattan world hypothesis, this method also includes the execution logic for the relative rotation optimization pre-step, specifically including: First, determine if the current frame ID is a multiple of a certain integer to achieve intermittent Manhattan vanishing point estimation. If Manhattan vanishing point estimation is performed on the current frame, then determine if the line clusters clustered according to the Manhattan vanishing point in the current frame meet the requirement that all three types of line clusters exist and each line cluster contains at least three line segments. If they meet the requirement, calculate the main direction curdk of the line cluster in the current frame and determine if the main direction curdk of the line cluster in the current frame and the main direction lastdk of the line cluster in the previous frame both exist, and the frame ID interval between the frames containing these two main directions of the line cluster cannot be greater than 15 frames. If all the above conditions are met, then perform relative rotation optimization.

2. The VSLAM method for indoor low-texture environments according to claim 1, characterized in that, The Shi-Tomasi corner feature extraction of the grayscale image yields the following point features: Calculate the image gradient in the horizontal direction of a grayscale image. Image gradient in the vertical direction The autocorrelation matrix Q of the grayscale image is calculated using the image gradient. The minimum eigenvalue of the autocorrelation matrix Q is then calculated. , In the formula This refers to the size of a local window. And through the response function of the Shi-Tomasi corner detection algorithm The response function is used to determine whether a corner point at a certain location in a grayscale image is a valid Shi-Tomasi corner point based on the calculated minimum eigenvalue. Typically, it is a defined threshold. ,in and The eigenvalues ​​are calculated using the autocorrelation matrix Q.

3. The VSLAM method for indoor low-texture environments according to claim 1, characterized in that, The process of merging near-line features and removing short lines from the extracted line features includes: A near-line merging and short-line elimination algorithm based on the LSD line segment extractor is used to perform near-line merging and short-line elimination.

4. The VSLAM method for indoor low-texture environments according to claim 1, characterized in that, In step S4, the relative rotation process includes: For each line cluster clustered through the Manhattan vanishing point, first calculate the normal vector of each line segment in the current camera coordinate system, and then calculate the direction vector of each line segment by performing a cross product operation on the start and end points in the aligned sub-coordinate form. and the direction vector Normalization is performed, and the processed line segment direction vectors are... Through camera intrinsic parameter matrix transpose Perform the transformation to obtain the normal vector of the line segment in the current camera coordinate system. Then, normalize it to construct a matrix S containing the normal vectors of all line segments in the three line clusters; Transpose matrix S and perform SVD singular value decomposition to obtain three directions that are orthogonal to the normal vectors of all line segments in the three line clusters, namely the main directions of the three line clusters, and obtain the main direction of the line cluster curdk of the current frame. After each calculation of curdk, the system saves and updates it to construct a pose-relative rotation optimization method between two frames.

5. The VSLAM method for indoor low-texture environments according to claim 4, characterized in that, The pose-relative rotation optimization method between the two frames includes: For the previous main direction of the line cluster, lastdk, ​​calculate its transformation direction in the current frame: ； in, The rotation matrix for the current frame. These represent the main direction numbers of three different line clusters. It is the inverse of the rotation matrix of the frame containing the main direction of the previous line cluster. The last main direction of the previous line cluster, lastdk, ​​is calculated. Direction relative to the current vanishing point of Manhattan The relative rotation angle error is defined by the change in angle, and a prior method based on angle weight and distance weight is adopted to adjust the internal arrangement order of curdk; Based on the cost function for minimizing the overall relative rotation error, the corresponding formula for calculating the Jacobian matrix is ​​as follows: ； Since the derivative of the translation part is 0, the final Jacobian matrix is ​​for each error term. It is The matrix is ​​given by the first three columns corresponding to the translation part and the last three columns corresponding to the rotation part. The final Jacobian matrix is ​​expressed as: ， in, These represent the three different Manhattan vanishing point direction indices, with the superscript T indicating transpose, [·]. × For vector antisymmetric matrix operators; The complete Jacobian matrix representing the rotation error term under each distinct pairwise orthogonal direction is expressed as: ； The relative rotation error is defined as an edge in the Levenberg-Marquardt optimization algorithm within G2O and added to the vertex defined by the current frame pose. This indicates that the relative rotation error imposes an optimization constraint on the current frame pose. The three main directions of the line clusters used in the optimization have the same weight, indicating that they have equal importance in the optimization process, thus obtaining the optimized rotation matrix. .

6. The VSLAM method for indoor low-texture environments according to claim 1, characterized in that, A strategy for optimizing the translation vector is defined using a random sampling consensus algorithm framework. In step S4, the translation optimization process includes: Define a random number generator to randomly select feature point indices in each iteration, and set an appropriate maximum number of iterations and an interior point threshold. In each iteration, query whether the randomly selected feature point index is a feature point that is associated with a valid graph point. After selecting feature points that are effectively associated with the image points, a linear least squares problem is constructed. First, the image coordinates of randomly selected feature points are obtained. and the world coordinates of its associated map points ( , , Then, based on the principal point coordinates in the camera intrinsic parameter matrix... The coordinates of the random feature points relative to the camera optical center are calculated. The specific formula is as follows: ； Next, based on the projection model, the matrix M and vector n required for solving the least squares problem are constructed: ； in, and For the camera's focal length, To determine the weights of the image pyramid corresponding to random feature points, solve the following equations using the least squares method to obtain the optimal translation vector. ; ； in, Let M be the transpose of matrix M.

7. The VSLAM method for indoor low-texture environments according to claim 1, characterized in that, Step S5 also includes: This invention employs bundle adjustment technique to globally optimize the poses of keyframes, map points, and map lines. It calculates the three-dimensional coordinates of map points and map lines through triangulation and uses global bundle adjustment technique to optimize the poses of keyframes, map points, and map lines, thereby improving the accuracy of the map.

8. A VSLAM device for indoor low-texture environments, characterized in that, The apparatus is used to perform the VSLAM method for indoor low-texture environments as described in any one of claims 1-7.

Images