VSLAM feature point matching method and system based on semantic topology constraint

Abstract

The invention belongs to the technical field of computer vision, and relates to a VSLAM feature point matching method and system based on semantic topology constraint, and the method comprises the following steps: generating a current feature point set containing a preliminary semantic tag and a semantic recognition result; generating an initial candidate matching pool; generating a dynamic geometric check threshold for each current feature point; performing forward filtering on the initial candidate matching pool by applying a dynamic geometric verification threshold, and performing initial round matching based on appearance description similarity to generate a high-potential pre-matching point pair list; the geometric structure consistency of the high-potential pre-matching point pair list is analyzed, and geometric structure confidence scores are generated; performing reverse calibration and confirmation on the high-potential pre-matching point pair list to generate a final matching result; and generating a semantic tag correction instruction according to the reverse calibration process. According to the method, the problem that adopted geometric correction is generally based on a fixed fault-tolerant threshold and lacks self-adaptive adjustment capability for credibility of different scenes and feature points is solved.

Description

A VSLAM Feature Point Matching Method and System Based on Semantic Topological Constraints Technical Field

[0001] This invention belongs to the technical field of computer vision and relates to a VSLAM feature point matching method and system based on semantic topological constraints. Background Technology

[0002] Visual Simultaneous Localization and Mapping (VSLAM) systems heavily rely on the quality of feature point matching, and achieving rapid feature point matching is a core challenge in this field. When ambient lighting changes, a large number of dynamic objects are present, or large areas of weak or repetitive texture exist in the scene, the appearance description of feature points becomes unstable or ambiguous, leading to numerous matching errors or ambiguities. These erroneous matches directly and severely interfere with the system's pose estimation and may even cause localization failure. Ensuring the robustness of feature point matching in complex and ever-changing real-world environments is a key issue determining the reliable application of VSLAM systems.

[0003] The solutions used in the industry rely on matching based on the appearance descriptors of feature points. The system extracts feature points from the current image and calculates their descriptors. Then, it searches for candidates with the closest descriptor distance in the entire map point cloud. Typically, a distance ratio test between the nearest and second nearest neighbors is used to initially filter out some fuzzy matches. To eliminate mismatches that are similar in appearance but have incorrect geometric positions, geometric verification algorithms such as Random Sample Consensus (RANSAC) are usually used. This involves iteratively searching for a camera pose model that can explain the most matching point pairs, and point pairs that conform to this model are considered the final correct matches.

[0004] Based on the above problems, the geometric verification method used is usually based on a fixed fault tolerance threshold, which lacks the ability to adaptively adjust the confidence of different scenarios and feature points. It appears rigid when dealing with matching of different confidence levels and cannot make full use of the high-level information in the scene to assist in the judgment. Summary of the Invention

[0005] In a first aspect, the present invention provides a VSLAM feature point matching method based on semantic topology constraints, employing the following technical solution: The VSLAM feature point matching method based on semantic topology constraints includes the following steps: S1, acquiring real-time image frames, performing feature point extraction and semantic segmentation on the real-time image frames, and generating a current feature point set containing preliminary semantic labels and semantic recognition results; S2, based on the preliminary semantic labels in the current feature point set, performing target indexing from a preset map point cloud to generate an initial candidate matching pool; S3, combining the semantic recognition results in the current feature point set, matching each current feature point with a specific target point. S4. Apply the dynamic geometric verification threshold to the initial candidate matching pool for forward filtering, and perform initial matching based on appearance description similarity to generate a list of high-potential pre-matching point pairs; S5. Analyze the geometric structure consistency of the high-potential pre-matching point pair list and generate a geometric structure confidence score; S6. Based on the geometric structure confidence score, perform reverse calibration and confirmation on the high-potential pre-matching point pair list to generate the final matching result; S7. Fuse the final accurate matching result to calculate and update the current pose, and generate semantic label correction instructions based on the reverse calibration process.

[0006] A further aspect of this invention involves generating a current feature point set, comprising the following steps: performing parallel feature point extraction on real-time image frames to obtain feature point coordinates and appearance descriptions, and performing semantic segmentation to obtain a semantic map containing the category attribution of each pixel; for each feature point, querying the semantic map using its feature point coordinates, using the queried category attribution as a preliminary semantic label, and using the classification certainty level output by the semantic segmentation network as a semantic recognition result, which, together with the feature point coordinates and appearance descriptions, constitutes one piece of data in the current feature point set.

[0007] A further aspect of the present invention generates an initial candidate matching pool, comprising the following steps: for any feature point in the current feature point set, extract its preliminary semantic label; using the extracted preliminary semantic label as a query index, select a subset of map points with the same semantic label from a preset map point cloud that has been classified and organized according to the semantic label; merge all the selected map point subsets to form the initial candidate matching pool.

[0008] A further aspect of this invention involves generating a dynamic geometric verification threshold for each current feature point, comprising the following steps: obtaining a confidence score from the semantic recognition results associated with the current feature point; applying a preset function that is positively correlated with the confidence score to map the confidence score to a geometric tolerance parameter, wherein a higher confidence score corresponds to a more lenient geometric tolerance parameter; and setting specific quantitative evaluation criteria based on the geometric tolerance parameter to form a dynamic geometric verification threshold.

[0009] A further aspect of this invention involves generating a list of high-potential pre-matching point pairs, comprising the following steps: traversing the initial candidate matching pool and calculating the geometric attribute difference value between each candidate point in the pool and the current feature point; comparing the geometric attribute difference value with a dynamic geometric verification threshold, retaining candidate points with difference values ​​below the threshold to form a list of remaining candidate points; sorting the remaining candidate points according to the similarity of their appearance descriptions, selecting the candidate point with the highest similarity to form a pre-matching point pair with the current feature point, and aggregating all generated pre-matching point pairs into a list of high-potential pre-matching point pairs.

[0010] A further aspect of the present invention generates a geometric structure confidence score, comprising the following steps: identifying whether a stable geometric configuration consisting of multiple point pairs exists in a list of high-potential pre-matched point pairs; wherein the relative spatial relationship of the current feature points of these point pairs in a real-time image frame is consistent with the relative spatial relationship of the matched map points in three-dimensional space; and quantifying and calculating the geometric structure confidence score based on the quality of the identified stable geometric configuration.

[0011] A further aspect of the present invention generates the final matching result by including the following steps: comparing the geometric structure confidence score with a preset high confidence threshold; if the geometric structure confidence score exceeds the high confidence threshold, then all point pairs constituting a stable geometric configuration are confirmed as the final accurate matching result, even if there are point pairs with inconsistent initial semantic labels.

[0012] A further aspect of the present invention generates a semantic label correction instruction based on the reverse calibration process, comprising the following steps: during the reverse calibration process, recording point pair information that is forcibly confirmed due to belonging to a stable geometric configuration and whose initial semantic labels are inconsistent; based on the recorded point pair information, generating a semantic label correction instruction that includes map point identifiers, original semantic labels, and new semantic labels suggested by the semantic recognition results of the current frame.

[0013] Secondly, this invention provides a VSLAM feature point matching system based on semantic topological constraints, employing the following technical solution: The VSLAM feature point matching system based on semantic topological constraints includes the following modules: a feature extraction and semantic segmentation module, used to acquire real-time image frames, perform feature point extraction and semantic segmentation on the real-time image frames, and generate a current feature point set containing preliminary semantic labels and semantic recognition results; a candidate matching pool generation module, based on the preliminary semantic labels in the current feature point set, performs targeted indexing from a preset map point cloud to generate an initial candidate matching pool; and a dynamic geometric threshold generation module, combining the semantic recognition results in the current feature point set, generates a dynamic geometric threshold for each current feature point. The system comprises the following modules: a dynamic geometric verification threshold for point generation; an initial matching and filtering module that applies the dynamic geometric verification threshold to perform forward filtering on the initial candidate matching pool and performs initial matching based on appearance description similarity to generate a list of high-potential pre-matching point pairs; a geometric consistency analysis module that analyzes the geometric structure consistency of the list of high-potential pre-matching point pairs and generates a geometric structure confidence score; a reverse calibration and confirmation module that performs reverse calibration and confirmation on the list of high-potential pre-matching point pairs based on the geometric structure confidence score to generate the final matching result; and a pose update and semantic correction module that integrates the final accurate matching result to calculate and update the current pose and generates semantic label correction instructions based on the reverse calibration process.

[0014] In summary, this invention offers the following beneficial technical effects: 1. By introducing semantic information to pre-screen the matching search space, the efficiency of feature point matching is significantly improved and the ambiguity of initial matching is reduced. By utilizing the semantic labels of feature points to target and index the map point cloud, the system can drastically narrow the search range from the entire map to a candidate subset that is semantically highly relevant to the current scene. This approach not only significantly reduces the computational complexity and time consumption of the matching process but also avoids invalid comparisons between semantically unrelated objects from the outset, thereby effectively reducing the number of initial mismatches caused by appearance similarity and laying a good foundation for subsequent matching.

[0015] 2. A dynamic geometric verification mechanism based on semantic confidence was established, enabling the matching process to have stronger adaptability and robustness. The system no longer uses a fixed geometric tolerance threshold, but dynamically adjusts the strictness of geometric constraints according to the determinism of semantic segmentation. When semantic information is highly reliable, the requirements for geometric accuracy can be relaxed, tolerating a certain amount of observation error; while when semantic information is ambiguous, geometric constraints are tightened to prevent the introduction of erroneous matches. This intelligent adjustment mechanism allows the system to flexibly balance the weights of semantic and geometric information sources, thus maintaining robust matching performance under various complex environments. 3. Bidirectional calibration of semantic and geometric information was implemented. The matching results were reverse-verified and confirmed through group geometric consistency, greatly improving the accuracy of the final matching results. After the initial round of matching, the system evaluates the overall geometric structure stability of the pre-matched point pairs. When strong evidence of geometric consistency appears, the system trusts this group geometric constraint and uses it to forcibly confirm or even correct individual matching point pairs that may have inconsistent semantic labels. This mechanism, which performs reverse calibration of local matching based on global geometry, effectively identifies and corrects individual matching errors that may be caused by semantic segmentation errors or scene understanding biases, ensuring that the final output matching results have extremely high reliability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The drawings are used to provide a further understanding of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 illustrates a flowchart of an embodiment of this application.

[0018] Figure 2 shows a schematic diagram of the structure in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The present invention will now be described in detail with reference to Figures 1-2.

[0021] Referring to Figure 1, this invention proposes a VSLAM feature point matching method based on semantic topological constraints, comprising the following steps: S1. Acquire real-time image frames, perform feature point extraction and semantic segmentation on the real-time image frames, and generate a current feature point set containing preliminary semantic labels and semantic recognition results; S2. Based on the preliminary semantic labels in the current feature point set, perform targeted indexing from a preset map point cloud to generate an initial candidate matching pool; S3. Combine the semantic recognition results in the current feature point set to generate a dynamic geometric verification threshold for each current feature point; S4. Apply the dynamic geometric verification threshold to perform forward filtering on the initial candidate matching pool, and perform initial matching based on appearance description similarity to generate a list of high-potential pre-matching point pairs; S5. Analyze the geometric structure consistency of the list of high-potential pre-matching point pairs to generate a geometric structure confidence score; S6. Based on the geometric structure confidence score, perform reverse calibration and confirmation on the list of high-potential pre-matching point pairs to generate the final matching result; S7. Fuse the final accurate matching result to calculate and update the current pose, and generate semantic label correction instructions based on the reverse calibration process.

[0022] In one embodiment of the present invention, step S1 includes the following steps: Specifically, the device to be located, equipped with a camera and a computing unit, is activated. The camera hardware of the device is invoked to capture a frame of the current scene. This frame is sent to the processing unit as a real-time image frame. This real-time image frame is a digital image file, such as an RGB color image with a resolution of 640x480 pixels. After receiving the real-time image frame, the system starts two independent but synchronous processing tasks in parallel. The first task is feature point extraction. The system applies the ORB feature extraction algorithm to scan the entire real-time image frame. This algorithm locates key points by quickly detecting corner points with drastic brightness changes in the image and calculates a unique binary descriptor for each key point. This descriptor summarizes the image texture information around the key point.

[0023] The second task is semantic segmentation. The system inputs the same real-time image frame into a pre-trained lightweight analysis model for recognizing object contours and regions. The lightweight analysis model analyzes the image content pixel by pixel and assigns a predefined category label to each pixel, ultimately generating a semantic map with the same dimensions as the original image. The value of each pixel in the semantic map represents its object category, such as wall, floor, or furniture. The outputs of the two tasks are then fused. For each keypoint obtained through the feature point extraction task, the system records its two-dimensional pixel coordinates in the image. The system uses these coordinates to query the semantic map generated by the semantic segmentation task and reads the category label of the pixel at the same location. This category label is then designated as the initial semantic label for that keypoint. By traversing all extracted keypoints and repeating this association operation, the system finally generates a structured set of current feature points.

[0024] Each data item in the current feature point set fully records the feature point's location coordinates, appearance texture descriptor, and its associated preliminary semantic label, providing geometric features with scene understanding information for the subsequent matching process.

[0025] The device to be located refers to a hardware entity that needs to determine its position and orientation in physical space, such as a mobile robot or augmented reality glasses equipped with a camera and processor. Its core function is to perceive the environment in real time and perform calculations. A real-time image frame represents a single static image captured by the camera at a specific moment in the device to be located. Its characteristic attributes are color and brightness information stored in the form of a pixel matrix. Feature point extraction is a technical operation that identifies salient and stable pixels from the real-time image frame. These feature points can be repeatedly detected even when the viewing angle and lighting change. Semantic segmentation is an image processing function that classifies each pixel in the real-time image frame into a preset category, such as classifying pixels as "sky," "building," or "road." The preliminary semantic label is a data label attached to each feature point. Its value is the object category obtained after semantic segmentation of the pixel location of that feature point. This label represents a preliminary understanding of the environment in which the feature point is located.

[0026] The current feature point set is a dynamically updated data list used to store complete information about all feature points successfully extracted and assigned preliminary semantic labels in the current real-time image frame. The data structure of the current feature point set is a collection of elements, each containing the two-dimensional coordinates of the feature point, a feature descriptor for matching, and an integer preliminary semantic label. The integer value of the label is based on a predefined mapping table; for example, 1 represents "wall," 2 represents "floor," and 3 represents "desktop." This mapping table is determined based on model training and category statistics on 10,000 publicly available images covering various indoor scenes.

[0027] For example, suppose the device to be located is in an office environment, and its camera captures a real-time image frame with a resolution of 640x480. On the left side of this real-time image frame is a white wall with a painting hanging on it, and on the right side is a wooden desk. Feature point extraction detects 500 feature points at locations such as the boundary between the wall and the painting frame, and the corners of the desk. Parallel semantic segmentation divides the real-time image frame into three main regions, assigning an initial semantic label "1" to pixels belonging to the wall, "4" to pixels belonging to the painting frame, and "3" to pixels belonging to the desk. For a feature point located at image coordinates (120, 250), this location is marked as "1" in the semantic segmentation result, and therefore this feature point is assigned an initial semantic label of "1". Another feature point located at coordinates (550, 400), whose location corresponds to semantic region "3", is assigned an initial semantic label of "3".

[0028] After iterating through all 500 feature points, a current feature point set containing 500 entries is finally generated. Each entry in the current feature point set contains the coordinates of the point, a 256-bit binary feature descriptor, and an integer preliminary semantic label representing "wall", "desktop", or "picture frame".

[0029] In one embodiment of the present invention, step S2 includes the following steps: Specifically, firstly, the system traverses the data of each feature point in the current feature point set. For any feature point in the set, the system reads its associated preliminary semantic label, such as the integer value "1" representing "wall". Then, the system uses this extracted preliminary semantic label "1" as a direct query instruction to perform a targeted indexing operation on the pre-built and stored map point cloud. The specific implementation of this targeted indexing operation is that the map point cloud data is already classified and organized according to its own semantic labels during storage, similar to a dictionary structure. Each semantic label is a key, and the corresponding value is a list of all map points with that label. When the query instruction is "1", the system can locate and extract all map points pre-marked as "wall" in the map without scanning the entire map point cloud.

[0030] This process is repeated for all feature points in the current feature point set, merging all retrieved map points that match the semantic category of the current frame. For example, if the feature points of the current frame involve both "wall" and "desktop," then all map points labeled "wall" and all map points labeled "desktop" will be retrieved. These retrieved map points are then aggregated into a temporary, significantly reduced-size data container, forming the initial candidate matching pool.

[0031] The pre-built map point cloud is a 3D environment model established before device positioning. It consists of tens of thousands of 3D spatial points. Its data characteristics are that each point not only contains its 3D coordinates (X, Y, Z) and feature descriptors for identification, but also stores its associated semantic label. The definition of this label is completely consistent with the initial semantic label category system. Targeted indexing is an efficient data retrieval operation. Its function is to filter out a subset of map points with the same semantic label from the vast pre-built map point cloud based on a given semantic label as the index key, avoiding the inefficient calculation of comparing all data one by one.

[0032] The initial candidate matching pool is a temporarily generated data set. Its data structure is a list, which contains all map points selected from the preset map point cloud based on the preliminary semantic labels of the feature points of the current frame. The feature attributes are that its size is much smaller than the complete preset map point cloud, and the semantic category of all points inside is related to the object category appearing in the current scene.

[0033] For example, a current feature point set containing 500 feature points is obtained, and the initial semantic labels of these points cover three categories: "1" (wall), "3" (desktop), and "4" (picture frame). Assume that the preset map point cloud contains a total of 1,000,000 map points, of which 200,000 points are labeled "1", 50,000 points are labeled "3", 5,000 points are labeled "4", and the remaining points are labeled with other categories.

[0034] The system performs targeted indexing. Instead of examining each of the 1,000,000 points individually, it directly extracts from the classified map data based on the labels "1", "3", and "4" appearing in the current frame: 200,000 points labeled "1", 50,000 points labeled "3", and 5,000 points labeled "4". The system aggregates these three groups, totaling 255,000 map points, to form an initial candidate matching pool. This initial candidate matching pool of 255,000 points will serve as the sole data source for the next stage of refined matching, and its size is significantly reduced compared to the original pre-set map point cloud of 1,000,000 points.

[0035] In one embodiment of the present invention, step S3 includes the following steps: Specifically, reading the semantic recognition result associated with the current feature point. This result includes not only the preliminary semantic label determined in the preceding steps, but also the confidence score output by the semantic segmentation analysis model. This confidence score is a floating-point value between 0 and 1, quantifying the degree of certainty of the analysis model in classifying the pixel containing the feature point as the current preliminary semantic label; a higher score indicates a more reliable classification. After obtaining this confidence score, the system applies a preset functional relationship to calculate the intermediate variable, namely the geometric tolerance parameter. This functional relationship is set to be positively correlated with the confidence score, i.e., a high confidence corresponds to a lenient geometric tolerance parameter, while a low confidence corresponds to a strict geometric tolerance parameter. This abstract geometric tolerance parameter is concretized into a quantifiable evaluation criterion to constitute a dynamic geometric verification threshold specifically for this matching.

[0036] The dynamic geometric verification threshold specifies the maximum permissible difference in local geometric attributes between candidate map points and current feature points in subsequent matching and filtering. For example, it sets a maximum distance threshold for the 3D position deviation in the camera's local coordinate system and a maximum angle threshold for the angle between surface normal vectors.

[0037] Establish a functional relationship between the confidence score and the geometric tolerance parameter, satisfying the following formula:

[0038] Here, T_g represents the generated geometric tolerance parameter, such as distance tolerance and angle tolerance; C_sem represents the confidence score in the semantic recognition result of the current feature point, ranging from 0 to 1; T_max and T_min represent the preset maximum and minimum values ​​of the geometric tolerance parameter, respectively. These values ​​are set based on empirical boundaries determined through statistical analysis of a large amount of real-world scene data, ensuring a balance between robustness and accuracy in matching. For example, based on the analysis of 200 sets of measured data in indoor environments, the distance tolerance T_max can be set to 0.5 meters and T_min to 0.05 meters; the angle tolerance T_max can be set to 15 degrees and T_min to 3 degrees. This ensures that when the confidence score C_sem is at its highest value of 1, the geometric tolerance parameter T_g takes its maximum value T_max, forming the most lenient verification threshold; when the confidence score C_sem is at its lowest value of 0, the geometric tolerance parameter T_g takes its minimum value T_min, forming the most stringent verification threshold.

[0039] The associated information refers to the additional data stored on each candidate map point in the initial candidate matching pool, mainly including the point's 3D coordinates, surface normal vectors, and geometric and appearance attributes such as feature descriptors. The semantic recognition result is a complete information packet output by the semantic segmentation network after classifying the pixel containing the current feature point. Its data structure includes preliminary semantic labels and corresponding confidence scores. The confidence score reflects the semantic segmentation network's grasp of the classification result. The geometric tolerance parameter is a dynamically calculated value or a set of values ​​used to define the leniency of the geometric similarity assessment. The dynamic geometric verification threshold is a specific set of quantitative rules for this round of matching, derived from the geometric tolerance parameter. Its characteristic attribute is that its value changes dynamically according to the semantic confidence of each current feature point.

[0040] For example, a dynamic geometric verification threshold is generated for the current feature point on the desk. The semantic segmentation analysis model gives the initial semantic label of this point as "3" (desktop) and outputs its confidence score of 0.95, indicating that the model determines this is a desktop. Substituting C_sem=0.95 into the formula, assuming the distance tolerance T_max is 0.5 meters and T_min is 0.05 meters, the calculated distance tolerance parameter is 0.4775 meters. Assuming the angle tolerance T_max is 15 degrees and T_min is 3 degrees, the calculated angle tolerance parameter is 14.4 degrees. The dynamic geometric verification threshold generated for this high-confidence desktop feature point is that the local coordinate deviation of the candidate map point must be less than 0.4775 meters, and the difference in the angle between the normal vectors must be less than 14.4 degrees. This threshold is relatively lenient.

[0041] Conversely, for feature points on the edge of a picture frame on the wall, the confidence score may be only 0.6, the calculated distance tolerance parameter is 0.32 meters, and the angle tolerance parameter is 10.2 degrees, forming a more stringent dynamic geometric verification threshold.

[0042] In one embodiment of the present invention, step S4 includes the following steps: Specifically, each candidate map point in the initial candidate matching pool is traversed; this process is called forward filtering. For any candidate point in the pool, its stored local geometric attributes are read, such as its three-dimensional coordinates and surface normal vectors in the map. The system also obtains the geometric attributes corresponding to the currently processed feature point. The system calculates the geometric difference value between these two, specifically, the three-dimensional Euclidean distance between the candidate point and the current feature point in the local coordinate space, and the angle between their surface normal vectors. Subsequently, the system compares the calculated distance difference value and angle difference value with the dynamic geometric verification threshold specific to the current feature point. Only when both of these difference values ​​are strictly less than the distance tolerance and angle tolerance set in the threshold are the candidate points considered geometrically reasonable and retained. All candidate points that do not meet this condition are directly eliminated. After completing this round of geometric verification for all candidate points in the initial candidate matching pool, the system obtains a smaller list of remaining candidate points after filtering.

[0043] The system performs a final match based on the similarity of appearance descriptions among the remaining candidate points that have passed the geometric verification. The system calculates the similarity score between the feature descriptor of each point in the remaining candidate point list and the feature descriptor of the current feature point. For ORB descriptors, this is typically done by calculating the Hamming distance; a smaller Hamming distance indicates higher similarity. The system sorts all remaining candidate points by similarity score and selects the candidate point with the highest similarity, i.e., the smallest Hamming distance. This candidate point with the most similar appearance is paired with the current feature point to form a pre-matched point pair. The system repeats the above process for each point in the current feature point set and aggregates all successfully generated pre-matched point pairs to form a unified list of high-potential pre-matched point pairs for consistency verification in subsequent steps.

[0044] Forward filtering is a one-way screening process that uses a dynamic geometric verification threshold as a standard to perform a one-time check and elimination of all points in the initial candidate matching pool to narrow down the candidate range. The initial matching round, after forward filtering, involves a small-scale matching attempt among geometrically reasonable candidate points, based on the similarity of appearance features. The final output of this step is a list of high-potential pre-matching point pairs, a dataset whose data structure is a list containing multiple point pairs. Each element in the list is a "pre-matching point pair," consisting of a current feature point from the current real-time image frame and candidate map points selected from a pre-set map point cloud that are highly similar in both geometry and appearance. "High potential" means that while these point pairs represent the best single-point matching choices, their correctness has not yet been definitively confirmed by the final analysis of group geometric relationships.

[0045] For example, a matching is performed on a desktop feature point with a confidence score of 0.95, where the dynamic geometric check threshold is a distance of less than 0.4775 meters and an angle of less than 14.4 degrees. The initial candidate matching pool contains 50,000 candidate points labeled "3" (desktop). The system begins forward filtering, calculating the geometric difference between each of these 50,000 candidate points and the current feature point. Assuming that 15 candidate points meet the threshold requirements—for example, a distance difference of 0.2 meters and an angle difference of 8 degrees, both less than the threshold—they are retained. Another candidate point, with a distance difference of 0.6 meters, exceeds the threshold and is discarded. After filtering, a remaining list containing 15 candidate points is obtained. Next, the system calculates the Hamming distance between the ORB feature descriptor of each of these 15 candidate points and the ORB descriptor of the current desktop feature point. The calculation results show that one candidate point has a Hamming distance of 12, the lowest among all 15 points. The system then combines the candidate point with the current desktop feature point with the Hamming distance of 12 to form a pre-matched point pair, and stores this point pair in the list of high-potential pre-matched point pairs.

[0046] In one embodiment of the present invention, step S5 includes the following steps: Specifically, firstly, the list of high-potential pre-matching point pairs is scanned to identify whether there exists a subset constituting a stable geometric configuration whose spatial arrangement remains consistent in the current viewpoint and the global map. To achieve this identification, the system employs a strategy similar to RANSAC (Random Sample Consensus). It randomly selects a small subset from the list, such as three pre-matching point pairs. For this subset, the system extracts the two-dimensional coordinates of the three current feature points in the current real-time image frame, and the coordinates of the three map points they each match in the three-dimensional map. The system checks whether the relative spatial relationships within these two sets of point pairs are consistent; for example, it can calculate the shape attributes of the triangle formed by the three current feature points and compare it with the triangle formed by projecting the three corresponding map points into the current viewpoint in three-dimensional space. If the shape, size, and orientation of the two are highly similar within a certain error range, then these three point pairs are considered to constitute a stable geometric configuration. The system repeats this process, trying different subsets and counting the number of point pairs supporting each stable configuration.

[0047] If the system finds one or more such configurations, and the number of point pairs (i.e., interior points) supporting this configuration exceeds a preset minimum threshold, such as 5 point pairs, then the system determines that the match has high geometric consistency. Conversely, if no stable geometric configuration can be found that is supported by a sufficient number of point pairs, and the spatial relationships between the point pairs exhibit a random and disordered state, then it is determined to have low geometric consistency. Based on multiple indicators such as the number of interior points, the compactness of the interior point distribution, and the average reprojection error of the identified stable geometric configurations, a comprehensive geometric confidence score is quantitatively calculated through a weighted summation. The higher this score, the more robust and reliable the geometric structure constituted by the pre-matched point pair list.

[0048] Geometric consistency is either a Boolean-type judgment or a continuously varying score, used to describe whether the group of point pairs in a high-potential pre-matching point pair list is spatially consistent. A stable geometric configuration refers to a set of (at least 3) pre-matching point pairs whose relative positional relationship of current feature points in the 2D image and the relative positional relationship of their matched map points in 3D space can be interpreted through a unified rigid body transformation (i.e., camera pose). The geometric confidence score is the final output of this step; it is a quantitative value that comprehensively evaluates the quality of the identified stable geometric configurations. Its data characteristic is that a higher score indicates a higher reliability of the pre-matching point pair list as a whole, and vice versa. This score is based on the premise that a geometric configuration with a large number of inliers and small inlier reprojection errors is more resistant to noise and false matches, and therefore should receive a higher confidence score.

[0049] For example, suppose the list of high-potential pre-matching point pairs contains 50 pairs. The system begins by randomly selecting 3 pairs for analysis. The first 3 selected pairs have current feature points that form approximately isosceles triangles on the image, while the map points they match are arranged in a nearly linear pattern in 3D space; their geometric relationships are inconsistent. After multiple random selections and checks, a stable geometric configuration supported by 25 pre-matching point pairs is found. These 25 pairs are called "interior points," and the spatial relationship between the current feature points and map points they represent can be perfectly explained by the same camera pose transformation, while the remaining 25 "outterior points" do not conform to this transformation. Because a strong geometric configuration containing 25 interior points has been found, the system determines that this match has high geometric consistency.

[0050] The system calculates a comprehensive geometric structure confidence score of 0.92 based on metrics such as the number of inliers (25) and the average reprojection error of these inliers (e.g., 0.8 pixels). This high score indicates that the overall geometric structure of this batch of matching results is very reliable, and even if the semantic labels of individual point pairs are questionable, their geometric correctness is extremely high.

[0051] In one embodiment of the present invention, step S6 includes the following steps: Specifically, the geometric structure confidence score is compared with a preset high confidence threshold. This high confidence threshold is set based on a large amount of experimental statistics and aims to distinguish matching situations where the geometric structure is almost impossible to err on the side of error. For example, the threshold can be set to 0.9. If the geometric structure confidence score, such as 0.92, exceeds this high confidence threshold of 0.9, the system initiates a reverse calibration process. In this process, the system determines that the reliability of the group geometric evidence provided by the stable geometric configuration has surpassed the indicative role of the preliminary semantic label of a single feature point. Therefore, the system will forcibly confirm all pre-matched point pairs (i.e., all interior points) constituting the most stable geometric configuration as the final accurate matching result. The key to this decision is that even if the preliminary semantic label of the current feature point of a point pair in this stable configuration is inconsistent with the semantic label of the map point (for example, the current feature point is identified as "wall", but the matching map point is marked as "column" in the database), as long as it belongs to this strong geometric configuration. If the confidence score of the geometric structure, for example, 0.65, fails to exceed the preset high confidence threshold of 0.9, it indicates that although a certain geometric structure exists, its stability is insufficient to serve as decisive evidence to refute the semantic information.

[0052] In this scenario, the system will not perform reverse calibration but will instead revert to a more conservative confirmation strategy. It will examine each pair of points in the list of high-potential pre-matching points one by one and directly accept the best match selected in step S4 based on semantic confidence adjustment and appearance description similarity. In other words, it trusts the judgment that a single point pair won in the initial round of matching and directly adopts all pre-matching point pairs in the list as the final exact match result without making any changes based on population geometry. Through both paths, the system will ultimately output a confirmed, final exact match result.

[0053] Reverse calibration is a decision correction mechanism that, when geometric evidence is strong enough, uses the judgment result of group geometric consistency to overwrite or correct the preliminary matching conclusion based on the semantic information of a single feature point. The high-confidence threshold is a pre-set value that serves as the criterion for initiating reverse calibration. Its setting is based on ensuring that semantic information correction is only triggered when the geometric structure is highly consistent and the error probability is extremely low. This threshold is typically optimized through receiver operating characteristic (ROC) curve analysis to balance the risks of correct and incorrect corrections. The final accurate matching result is the core data output of this method. It is a list of point pairs confirmed after dual verification of semantic and geometric information, representing the most reliable correspondence between feature points and map points in the current frame. Its data structure is the same as the high-potential pre-matching point pair list, but its accuracy is guaranteed at a higher level.

[0054] For example, the system calculates a geometric confidence score of 0.92. The preset high confidence threshold is 0.9. Since 0.92 is greater than 0.9, the system initiates reverse calibration. Among the 25 interior point pairs constituting the stable geometric configuration, the system finds one pair with inconsistent semantic labels: the initial semantic label of a current feature point is "1" (wall), but the label of the map point it matches is recorded in the database as "5" (door frame). Despite the semantic discrepancy, because this point pair is part of a strong geometric configuration, the system ignores this semantic difference and forcibly confirms all 25 interior point pairs, including the semantically inconsistent pair, as the final exact match. The other 25 exterior points are discarded. If the calculated geometric confidence score had been only 0.7, below the high confidence threshold of 0.9, the system would not have performed reverse calibration and would have directly adopted all 50 point pairs from the initially generated high-potential pre-matching point pair list as the final exact match without modification, even if some of them might contain geometrically inconsistent matches.

[0055] In one embodiment of the present invention, step S7 includes the following steps: Specifically, after the final accurate matching result is generated, this step utilizes this high-quality matching information to complete two core tasks: updating the device pose and optimizing the map. The system inputs the final matching result into the pose solving algorithm module. This module uses the confirmed 2D to 3D point correspondence, i.e., the image coordinates of each current feature point and the 3D spatial coordinates of its matching map point, to calculate the current pose and position of the device to be located in the map coordinate system. A commonly used algorithm is the PnP solver, which can solve for the camera's rotation matrix and translation vector by minimizing the reprojection error of map points in the current camera view. These two quantities together constitute the device's current pose. After the system completes the pose calculation, it checks whether a forced confirmation event based on strong geometric evidence occurred during the reverse calibration process in step S6. If it occurred, it means that the system identified some matching point pairs whose initial semantic labels did not match the geometric structure but were forcibly confirmed.

[0056] The system records information about these point pairs, especially those whose initial semantic labels are inconsistent and have been "corrected" by geometric configuration. For example, it records the ID of a point on the map, its original semantic label being "pillar," and the feature point that matches it in the current frame being highly confidently identified as "wall." Based on these records, the system automatically generates a semantic label correction instruction. This instruction is a structured data packet pointing to the entry for that specific map point in the map database and contains suggested semantic information for updating, namely, suggesting that its original "pillar" label be changed to the "wall" label observed in the current frame and supported by strong geometric structure. This instruction is sent to the map maintenance module for future offline or online updates and optimizations of the pre-built map point cloud, thereby achieving long-term self-improvement of the map.

[0057] Fusion refers to integrating newly confirmed matching information into the existing positioning and mapping system process to drive subsequent calculations and updates. The current pose is the six-degree-of-freedom state of the device to be located in the global map coordinate system, including three dimensions of position coordinates (X, Y, Z) and three dimensions of rotational attitude (roll, pitch, yaw), describing the device's position and orientation in space. Strong geometric evidence refers to stable geometric configurations with extremely high geometric structure confidence scores identified in steps S5 and S6; the reliability of this evidence is sufficient to overturn the initial semantic judgment of a single feature point. The semantic label correction instruction is the output data generated by this step for map optimization, containing three parts: the unique identifier (ID) of the map point to be corrected, the original semantic label of the point, and the suggested update to the new semantic label. The function of this instruction is to provide correction suggestions validated in real-world scenarios for the long-term maintenance of the map.

[0058] For example, the system uses the final precise matching result containing 25 point pairs to calculate the device's current pose using the PnP algorithm: position coordinates (3.45, 2.10, 1.55), and pose quaternions (0.99, 0.01, 0.12, 0.05). The system records the semantically inconsistent point pair that was forcibly confirmed during the reverse calibration process: map point ID "MP7852", its original semantic label is "5" (door frame), while the initial semantic label of the matching current feature point is "1" (wall). Based on this record, the system generates a semantic label correction instruction, the content of which is: {target point ID: "MP7852", original label: 5, suggested label: 1}.

[0059] This instruction is stored in the pending queue. When the system is idle or during the next global map optimization, the map maintenance module will read this instruction and evaluate whether to permanently update the semantic label of map point "MP7852" from "door frame" to "wall".

[0060] Referring to Figure 2, this invention also proposes a VSLAM feature point matching system based on semantic topological constraints, comprising the following modules: a feature extraction and semantic segmentation module, used to acquire real-time image frames, perform feature point extraction and semantic segmentation on the real-time image frames, and generate a current feature point set containing preliminary semantic labels and semantic recognition results; a candidate matching pool generation module, based on the preliminary semantic labels in the current feature point set, performs targeted indexing from a preset map point cloud to generate an initial candidate matching pool; a dynamic geometric threshold generation module, combining the semantic recognition results in the current feature point set, generates a dynamic geometric verification threshold for each current feature point; and an initial round matching module; The matching and filtering module applies a dynamic geometric verification threshold to perform forward filtering on the initial candidate matching pool and performs initial matching based on appearance description similarity to generate a list of high-potential pre-matching point pairs. The geometric consistency analysis module analyzes the geometric structure consistency of the high-potential pre-matching point pair list and generates a geometric structure confidence score. The reverse calibration and confirmation module performs reverse calibration and confirmation on the high-potential pre-matching point pair list based on the geometric structure confidence score to generate the final matching result. The pose update and semantic correction module integrates the final accurate matching result to calculate and update the current pose, and generates semantic label correction instructions based on the reverse calibration process.

[0061] Each of the modules can be implemented in whole or in part through software, hardware, or a combination thereof. It supports hardware embedded in or independent of the processor in the computer device, and also supports software stored in the memory of the computer device, so that the processor can call and execute the operations corresponding to each of the above modules.

[0062] It should be noted that the human information (including but not limited to human device information and personal information) and data (including but not limited to data used for analysis, data stored and data displayed) involved in this invention are all information and data authorized by the human body or fully authorized by all parties. The collection, use and processing of related data require relevant legal standards.

[0063] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A VSLAM feature point matching method based on semantic topological constraints, characterized in that, Includes the following steps: S1. Acquire real-time image frames, perform feature point extraction and semantic segmentation on the real-time image frames, and generate a current feature point set containing preliminary semantic labels and semantic recognition results; S2. Based on the preliminary semantic labels in the current feature point set, perform targeted indexing from the preset map point cloud to generate an initial candidate matching pool; S3. Combine the semantic recognition results in the current feature point set to generate a dynamic geometric verification threshold for each current feature point; S4. Apply dynamic geometric verification threshold to positively filter the initial candidate matching pool, and perform initial matching based on appearance description similarity to generate a list of high-potential pre-matching pairs. S5. Analyze the geometric structure consistency of the list of high-potential pre-matching point pairs and generate a geometric structure confidence score. S6. Based on the geometric structure confidence score, perform reverse calibration and confirmation on the list of high-potential pre-matching point pairs to generate the final matching results; S7. The final accurate matching result is fused to calculate and update the current pose, and semantic label correction instructions are generated based on the reverse calibration process.

2. The VSLAM feature point matching method based on semantic topological constraints according to claim 1, characterized in that, Generating the current feature point set includes the following steps: performing feature point extraction in parallel on real-time image frames to obtain feature point coordinates and appearance descriptions, and performing semantic segmentation to obtain a semantic map containing the category classification of each pixel; for each feature point, querying the semantic map using its feature point coordinates, using the queried category classification as the initial semantic label, and using the classification certainty level output by the semantic segmentation network as the semantic recognition result, which together with the feature point coordinates and appearance description constitutes one piece of data in the current feature point set.

3. The VSLAM feature point matching method based on semantic topological constraints according to claim 1, characterized in that, The initial candidate matching pool is generated by the following steps: for any feature point in the current feature point set, extract its preliminary semantic label; using the extracted preliminary semantic label as a query index, select a subset of map points with the same semantic label from the preset map point cloud that has been classified and organized according to the semantic label; merge all the selected map point subsets to form the initial candidate matching pool.

4. The VSLAM feature point matching method based on semantic topological constraints according to claim 1, characterized in that, The process of generating a dynamic geometric verification threshold for each current feature point includes the following steps: obtaining a confidence score from the semantic recognition results associated with the current feature point; applying a preset function that is positively correlated with the confidence score to map the confidence score to a geometric tolerance parameter, wherein a higher confidence score corresponds to a more lenient geometric tolerance parameter; and setting specific quantitative evaluation criteria based on the geometric tolerance parameter to form a dynamic geometric verification threshold.

5. The VSLAM feature point matching method based on semantic topological constraints according to claim 1, characterized in that, Generating a list of high-potential pre-matching point pairs includes the following steps: traversing the initial candidate matching pool and calculating the geometric attribute difference value between each candidate point in the pool and the current feature point; comparing the geometric attribute difference value with a dynamic geometric verification threshold, retaining candidate points with difference values ​​below the threshold to form a list of remaining candidate points; sorting the remaining candidate points according to the similarity of appearance description, selecting the candidate point with the highest similarity to form a pre-matching point pair with the current feature point, and aggregating all generated pre-matching point pairs into a list of high-potential pre-matching point pairs.

6. The VSLAM feature point matching method based on semantic topological constraints according to claim 1, characterized in that, Generating a geometric structure confidence score includes the following steps: identifying whether a stable geometric configuration consisting of multiple point pairs exists in a list of high-potential pre-matched point pairs; wherein the relative spatial relationship of the current feature points of these point pairs in a real-time image frame is consistent with the relative spatial relationship of the matched map points in three-dimensional space; and quantifying the geometric structure confidence score based on the quality of the identified stable geometric configuration.

7. The VSLAM feature point matching method based on semantic topological constraints according to claim 1, characterized in that, The final matching result is generated by comparing the geometric structure confidence score with a preset high confidence threshold. If the geometric structure confidence score exceeds the high confidence threshold, all point pairs that constitute a stable geometric configuration are confirmed as the final accurate matching result, even if there are point pairs with inconsistent initial semantic labels.

8. The VSLAM feature point matching method based on semantic topological constraints according to claim 1, characterized in that, The semantic label correction instruction is generated based on the reverse calibration process, including the following steps: during the reverse calibration process, point pair information that is forcibly confirmed due to belonging to a stable geometric configuration and whose initial semantic labels are inconsistent is recorded; based on the recorded point pair information, a semantic label correction instruction is generated that includes map point identifiers, original semantic labels, and new semantic labels suggested by the semantic recognition results of the current frame.

9. A VSLAM feature point matching system based on semantic topological constraints, characterized in that, It includes the following modules: a feature extraction and semantic segmentation module, which is used to acquire real-time image frames, perform feature point extraction and semantic segmentation on the real-time image frames, and generate a current feature point set containing preliminary semantic labels and semantic recognition results; The candidate matching pool generation module generates an initial candidate matching pool by performing targeted indexing from a preset map point cloud based on the preliminary semantic labels in the current feature point set; the dynamic geometric threshold generation module generates a dynamic geometric verification threshold for each current feature point by combining the semantic recognition results in the current feature point set. The initial matching and filtering module applies a dynamic geometric verification threshold to perform positive filtering on the initial candidate matching pool and performs initial matching based on appearance description similarity to generate a list of high-potential pre-matching point pairs. The geometric consistency analysis module is used to analyze the geometric consistency of a list of high-potential pre-matched point pairs and generate a geometric confidence score. The reverse calibration and verification module performs reverse calibration and verification on the list of high-potential pre-matching point pairs based on the geometric confidence score, and generates the final matching result. The pose update and semantic correction module integrates the final accurate matching results to calculate and update the current pose, and generates semantic label correction instructions based on the reverse calibration process.

Images