Global positioning method and system based on multi-source information fusion and graph theory

Abstract

The invention discloses a global positioning method and system based on multi-source information fusion and a graph theory, and the method comprises the steps: fusing a laser radar point cloud, an image sequence and IMU data in a prior map construction stage, segmenting an image, extracting a bounding rectangle, and generating an object-level map; in the online mapping stage, three-dimensional reconstruction of a dynamic object is realized through a monocular depth estimation network and pose information; in the global data association stage, a soft association mechanism is innovatively proposed, an affinity matrix is constructed, and an optimal matching relation is solved through graph optimization; and finally, global positioning is realized through pose transformation calculation. According to the method, the data processing flow of the multi-modal sensor is uniformed in a breakthrough mode, the matching robustness in a cross-modal scene is remarkably improved by adopting an object-level map expression and subgraph division strategy and combining two-way K-nearest neighbor matching and shape similarity evaluation, and compared with a traditional geometric method, the method has the advantages that the registration precision is improved, and the matching robustness is improved. The problem of association ambiguity caused by depth estimation difference in a dynamic environment is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous navigation and high-precision positioning of mobile robots, and in particular to a global positioning method and system based on multi-source information fusion and graph theory. Background Art

[0002] In recent years, with the rapid development of science and technology and people's ever-increasing pursuit of a higher quality of life, intelligent and digital devices have been widely used in various fields. The maturity of key technologies such as artificial intelligence, 5G communications, and high-performance computing has driven the in-depth deployment of various intelligent mobile devices in industrial manufacturing, public services, and domestic life, such as warehouse logistics robots, agricultural inspection robots, medical assistance robots, and household sweeping robots. As a typical representative of intelligent mobile devices, mobile robots integrate multiple perception and positioning sensors. They rely on key technologies such as multi-source information fusion, precise positioning, autonomous decision-making, and path control to support stable operation in complex environments. Accurate pose estimation is the foundation for path planning and dynamic control. Complex scenarios place higher demands on their positioning systems: 1) High precision: precise positioning is required to meet the needs of robots for sophisticated operations in complex environments; 2) Real-time performance: robots must quickly respond to external changes and adjust their operating strategies in real time; 3) Stability: they must be robust to geometric changes, lighting interference, and dynamic objects to achieve all-weather, highly reliable operation.

[0003] From the perspective of positioning tasks, the robot positioning process can be divided into two stages: global positioning and pose tracking. Global positioning involves inferring the robot's initial pose in the global coordinate system based on a given environmental map. It is typically used for system startup, closed-loop testing, or relocalization after positioning loss. Its implementation methods mainly include relying on external sensors to obtain absolute position information or matching sensory data with a priori maps for relocalization. The Global Navigation Satellite System (GNSS) receives signals from multiple navigation satellites to calculate the receiver's three-dimensional position and time information. It can achieve high-precision, all-weather absolute positioning in unobstructed environments and is widely used in outdoor scenarios. However, it is susceptible to environmental obstructions such as urban canyons and tunnels, which degrades positioning performance. Ultra-wideband (UWB) technology relies on the deployment of multiple base stations within the operating area. It achieves centimeter-level high-precision positioning by measuring the propagation delay of radio pulses. It has excellent anti-interference capabilities and strong real-time performance, making it suitable for precise positioning in structured and semi-enclosed environments. However, its effective coverage is limited, system deployment requires advance planning, and the initial cost is high. Visual marker localization employs artificial visual markers placed in the environment, combining cameras with computer vision algorithms to identify the markers in real time and calculate their relative positions. This approach offers advantages such as flexible deployment, low cost, and ease of implementation, making it suitable for indoor environments and standardized venues. However, its performance is susceptible to lighting variations, occlusions, and camera field of view limitations, resulting in limited positioning stability. Another typical global localization approach involves relocalizing the current position by matching global features with a pre-built environmental map. This approach is often used for loop closure detection, system initialization, or relocalization during long-term operation. This approach, which does not rely on external hardware infrastructure, primarily extracts global descriptions of sensory data, such as the current image or point cloud, and performs similarity searches against feature representations in a priori maps to identify possible pose candidates. This approach performs well in environments with distinct static structures and rich sensory features. However, matching accuracy can be affected in environments with sparse features, drastically changing lighting, or highly dynamic scenes, potentially leading to relocalization failure or accuracy degradation.

[0004] In order to make up for the limitations of various methods, a global positioning method and system that fuses multi-source information is needed. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a global positioning method and system based on multi-source information fusion and graph theory, which integrates lidar, vision and inertial information and uses graph theory optimization to achieve global positioning.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] The global positioning method based on multi-source information fusion and graph theory provided by the present invention includes the following steps:

[0008] Step 1: Prior map construction phase: Using the LiDAR point cloud, image sequence, and IMU information, combined with the FastSAM image segmentation algorithm, the objects in the image are segmented and the minimum bounding rectangle of the segmented area is extracted. The segmented area is reconstructed in 3D using the LiDAR point cloud to generate a prior object map.

[0009] Step 2, online mapping phase: Using image sequences and IMU information, combined with the FastSAM image segmentation algorithm, objects in the image are segmented, and a monocular depth estimation network is used to generate a 3D point cloud to generate an online object map;

[0010] Step 3, global data association phase: Identify the prior object map and the online object map and establish object-level matching relationships to complete global data association;

[0011] Step 4, global positioning stage: Based on the results of global data association, solve the pose transformation between the prior object map and the online object map to complete global positioning.

[0012] Furthermore, the construction of the prior object map in step 1 further includes the following steps:

[0013] Step 11) Use YOLOv7 to remove dynamic objects in the image;

[0014] Step 12) extracting the minimum bounding rectangle of the segmented area and calculating the corresponding CLIP semantic vector;

[0015] Step 13) Back-project the segmented area into three-dimensional space and perform three-dimensional reconstruction based on the LiDAR point cloud;

[0016] Step 14) Introducing an object tracking and association mechanism, matching historical objects by voxel grid overlap, and initializing unmatched objects as new objects;

[0017] Step 15) performing normalized averaging on the CLIP vectors of multiple associated segmented objects to form a stable semantic description;

[0018] Step 16) introduces an object merging strategy. When two 3D voxels or their 2D projections on the image plane have a high degree of overlap, they are considered to be the same object and merged.

[0019] Furthermore, the construction of the online object map in step 2 further includes the following steps:

[0020] Step 21) Use YOLOv7 to remove dynamic objects in the image;

[0021] Step 22) Use FastSAM to perform image segmentation and calculate the CLIP semantic vector;

[0022] Step 23) Generate a 3D point cloud through a monocular depth estimation network and perform 3D reconstruction based on the camera pose information;

[0023] Step 24) Using the same object tracking and association mechanism as the prior object mapping, historical objects are matched based on voxel grid overlap;

[0024] Step 25) performing normalized averaging on the CLIP vectors of multiple associated segmented objects to form a stable semantic description;

[0025] Step 26) introduces an object merging strategy to avoid redundant segmentation due to perspective differences.

[0026] Furthermore, the global data association process in step 3 is performed according to the following steps:

[0027] Step 31) constructing a set of matching pairs between the prior object map and the online object map based on the geometric features, semantic information and gravity prior of the objects;

[0028] Step 32) Screen candidate matching pairs through a soft association mechanism, using geometric distance and semantic similarity for dual constraints;

[0029] Step 33) constructing an affinity matrix to quantify the consistency relationship of candidate matching pairs, where the matrix elements integrate geometric consistency, semantic similarity and shape similarity;

[0030] Step 34) Calculate the optimal matching pair based on the affinity matrix to complete the global data association.

[0031] Furthermore, the global data association stage further includes the following steps:

[0032] Step 311) dividing the prior object map and the online object map into a plurality of object subgraphs, each subgraph containing a set of objects within a certain range;

[0033] Step 312) constructing a set of matching pairs between subgraphs, and performing soft association screening using geometric distance and semantic similarity;

[0034] Step 313) constructing an affinity matrix, wherein the matrix elements incorporate geometric consistency, semantic similarity, and shape similarity;

[0035] Step 314) Based on the gravity prior, the geometric consistency of the matching pairs is constrained to ensure that the pose transformation of the matching pairs conforms to the direction of the gravity vector.

[0036] Furthermore, the soft association mechanism includes the following steps:

[0037] Step 321) Generate initial matching pairs using full-pair full matching;

[0038] Step 322) Using the mutual K-nearest neighbor matching strategy to screen candidate matching pairs;

[0039] Step 323) Calculate the matching distance, including geometric distance and semantic similarity;

[0040] Step 324) retains the matching pairs of the source object and the target object that are mutually K-nearest neighbors, and constructs a mutual K-nearest neighbor candidate matching set.

[0041] Furthermore, the construction of the affinity matrix includes the following steps:

[0042] Step 331) defining the matrix elements as the harmonic mean of geometric consistency, semantic similarity and shape similarity;

[0043] Step 332) Geometric consistency is based on the distance deviation between centroids, semantic similarity is based on CLIP vector cosine similarity, and shape similarity is based on bounding box volume and covariance properties;

[0044] Step 333) Use the gravity prior to constrain the geometric consistency of the matching pair to ensure that the pose transformation of the matching pair conforms to the direction of the gravity vector.

[0045] The global positioning system based on multi-source information fusion and graph theory provided by the present invention includes an object-level mapping module, a global data association module, and a global positioning module;

[0046] The object-level mapping module is used to construct a priori object maps and online object maps;

[0047] The global data association module is used to establish a matching relationship between the prior object map and the online object map;

[0048] The global positioning module is used to solve the posture transformation according to the matching results and complete the global positioning;

[0049] The object-level mapping module includes a priori mapping submodule and an online mapping submodule;

[0050] The prior mapping submodule uses lidar point clouds, image sequences, and IMU information in combination with the FastSAM image segmentation algorithm to generate a priori object maps;

[0051] The online mapping submodule uses image sequences and IMU information, combined with the FastSAM image segmentation algorithm and the monocular depth estimation network to generate an online object map;

[0052] The global data association module includes a soft association submodule, an affinity matrix construction submodule, and an optimization solution submodule;

[0053] The soft association submodule is used to screen candidate matching pairs through geometric and semantic dual constraints;

[0054] The affinity matrix construction submodule is used to quantify the consistency relationship of candidate matching pairs;

[0055] The optimization solution submodule is used to solve the optimal matching pair based on the affinity matrix to complete data association.

[0056] Furthermore, the optimization solution submodule is performed according to the following steps:

[0057] a) Construct an affinity matrix to quantify the consistency relationship of candidate matching pairs;

[0058] b) Solve the binary vector to determine the matching relationship between the source set and the target set;

[0059] c) Based on the matching results, solve the pose transformation and complete global positioning.

[0060] The global positioning system based on multi-source information fusion and graph theory provided by the present invention includes a memory, a processor and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the above method is implemented.

[0061] The beneficial effects of the present invention are:

[0062] The present invention discloses a global positioning method based on multi-source information fusion and graph theory, which includes four stages: the prior map construction stage fuses lidar point cloud, image sequence and IMU data, uses FastSAM to segment the image and extract the circumscribed rectangle, and combines three-dimensional reconstruction to generate an object-level map containing point cloud, centroid, shape and semantic information; the online mapping stage realizes three-dimensional reconstruction of dynamic objects through a monocular depth estimation network and VIO pose information; the global data association stage innovatively proposes a soft association mechanism, screens candidate matching pairs based on the dual constraints of Wasserstein geometric distance and CLIP semantic similarity, constructs an affinity matrix that integrates geometric consistency, semantic similarity and gravity prior, and solves the optimal matching relationship through graph optimization; finally, global positioning is achieved through pose transformation calculation.

[0063] This method, a groundbreaking approach, unifies the multimodal sensor data processing pipeline. It employs object-level map representation and subgraph partitioning strategies, combined with bidirectional K-nearest neighbor matching and shape similarity assessment, significantly improving matching robustness in cross-modal scenarios. Experiments demonstrate that this method reduces the average localization error on the Kimera dataset to 0.359 meters, the angular error to 2.782 degrees, and the recall rate to 70%. This improves registration accuracy by 65% ​​compared to traditional geometric methods, effectively resolving association ambiguity caused by depth estimation discrepancies in dynamic environments.

[0064] This method addresses the current global positioning problem, where sensor modality inconsistencies and significant differences in perspective between the robot's sensors and the prior map construction lead to inconsistent object representation, global registration failure, and unreliable pose verification. It provides a perspective-invariant, cross-modal, robust object-level map-based positioning solution. Based on a unified data representation, it uses geometric features, semantic information, and gravity priors to perform global data association, and then estimates and verifies pose based on the association results, ultimately outputting a global positioning result. Leveraging the complementary characteristics of different sensors significantly enhances the system's accuracy, stability, and adaptability in complex dynamic environments, and has become a core development trend in high-precision, full-scenario intelligent positioning systems.

[0065] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration.

[0067] Figure 1 This is a flow chart of the global positioning algorithm based on multi-source information fusion.

[0068] Figure 2 Constructing processes for prior object maps.

[0069] Figure 3 Build processes for online maps.

[0070] Figure 4 This is the global data association flow chart.

[0071] Figure 5 The process of building a map for the sparkal1 sequence.

[0072] Figure 6 The process of building a graph for the sparkal2 sequence.

[0073] Figure 7 Build renderings for object-level maps.

[0074] Figure 8 Heatmap of the registration error between the prior object map and the submap of the online object map.

[0075] Figure 9 This is the global positioning effect of this method. DETAILED DESCRIPTION

[0076] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0077] like Figure 1 As shown, Figure 1 This is a flow chart of a global positioning algorithm based on multi-source information fusion. This embodiment provides a global positioning method based on multi-source information fusion and graph theory, which includes the following steps:

[0078] Step 1, the prior map construction phase, is to achieve object-level mapping. The specific process is as follows:

[0079] 11) In the prior map construction stage, the lidar point cloud, image sequence and IMU (inertial measurement unit) information are combined with the FastSAM image segmentation algorithm to segment the objects in the image and extract the minimum bounding rectangle of the segmented area;

[0080] 12) Use the LiDAR point cloud to perform 3D reconstruction of the segmented area and generate a priori object map, where each object contains point cloud information, centroid coordinates, shape descriptor and semantic embedding;

[0081] Step 2: In the online mapping phase, the image sequence and IMU information are combined with the FastSAM image segmentation algorithm to segment the objects in the image, and a monocular depth estimation network is used to generate a 3D point cloud to generate an online object map.

[0082] In this embodiment, steps 1 and 2 describe the use of a LiDAR point cloud / monocular depth estimation network, image sequences, and an IMU (inertial measurement unit) to stitch the segmented regions into a 3D map, which is the object map. The monocular depth estimation network takes an image as input and outputs a depth map of the same resolution, where each pixel value represents the depth information corresponding to that pixel. The monocular depth estimation network and the LiDAR point cloud serve the same purpose here, providing depth information for each pixel. The LiDAR point cloud, using LiDAR and camera extrinsics, converts the point cloud to the image plane, providing depth information for each pixel.

[0083] Step 3, global data association phase: Identify the prior object map and the online object map and establish object-level matching relationships to complete global data association. The specific steps are as follows:

[0084] 31) Building a set of matching pairs between the prior object map and the online object map based on the geometric features, semantic information and gravity prior of the objects;

[0085] 32) Candidate matching pairs are screened through a soft association mechanism, using geometric distance (Wasserstein distance) and semantic similarity (CLIP vector cosine similarity) for dual constraints;

[0086] 33) Construct an affinity matrix to quantify the consistency relationship of candidate matching pairs, where the matrix elements integrate geometric consistency, semantic similarity and shape similarity;

[0087] 34) Solve the optimal matching pair based on the affinity matrix and complete global data association;

[0088] Step 4, global positioning stage: Based on the results of global data association, solve the pose transformation between the prior object map and the online object map to complete global positioning.

[0089] In the global localization task provided by this embodiment, object-level maps are constructed in both the prior map construction and online localization stages. FastSAM is uniformly used for image segmentation, and 3D restoration is achieved by combining segmentation masks, depth information, and odometry data.

[0090] During the prior map construction phase, the system uses LiDAR to obtain high-precision depth information and perform fine 3D reconstruction of objects segmented in the image, forming a priori object map covering a large environment as a reference for subsequent global positioning.

[0091] During the online localization phase, the system acquires images and completes object segmentation in real time. It then uses monocular depth estimation combined with odometry information from the current frame to perform local 3D reconstruction of the object in the current viewing angle, generating an online object map with a smaller spatial range.

[0092] The prior object map in this embodiment is constructed based on a combination of multiple sensors. The process is as follows: Figure 2As shown, the input data includes lidar point clouds, image sequences, and IMU information, with pose information provided by LiDAR. First, YOLOv7 is used to remove dynamic objects such as pedestrians from the image. FastSAM is then used to segment the image. The minimum bounding rectangle of the segmented region is extracted, and the corresponding CLIP semantic vector is calculated to describe the semantic category and context of the object. The segmented 2D object regions in the image are then back-projected into 3D space. The lidar point cloud provides accurate spatial structure information for each segmentation mask. Each object is represented as a point cloud and serves as a mapping unit. During the inter-frame mapping process, an object tracking and association mechanism is introduced. Objects in the current frame are matched against historical objects using global nearest neighbor matching based on voxel grid overlap. Objects that fail to match are initialized as new objects. Furthermore, the CLIP vectors of multiple associated segmented objects are normalized and averaged to form a stable semantic description. To avoid the problem of a single object being misidentified as multiple segmentations due to viewpoint differences, the system incorporates an object merging strategy: when two 3D voxels or their 2D projections on the image plane have a high degree of overlap, they are considered the same object and merged. Finally, the system outputs an object-level 3D map represented by a point cloud. Each object in the map contains point cloud information, centroid coordinates, shape descriptors, and semantic embeddings.

[0093] The online object map construction process in this embodiment is as follows Figure 3 As shown in Figure 2, its input data includes image sequences and IMU information, with VIO providing camera pose information. The image processing pipeline remains consistent with the prior stage: YOLOv7 is used to remove dynamic objects, FastSAM performs image segmentation, and CLIP semantic vectors are calculated for the retained object regions. Unlike prior mapping, 3D information recovery in the online localization stage uses a monocular depth estimation network, Metric3D-v2, to perform pixel-level depth prediction on the image. Combined with the current camera pose, the 2D segmented regions are projected into 3D space to form a dense point cloud representation. Objects are also represented as point clouds and participate in the subsequent mapping process. This module uses the same object tracking and association mechanism as the prior object mapping, achieving object-level matching and updating based on voxel grid overlap, and fusing CLIP semantic vectors. The object merging strategy remains the same to suppress redundant segmentations under viewpoint changes. The final output is an object-level 3D map, whose structure and semantic representation are consistent with the prior map, providing the foundation for subsequent global data association.

[0094] In this embodiment, during the global data association phase, the goal is to find the correct matching pairs in the two object-level maps. Since the pixel depth information in the prior map comes from lidar and image fusion, while the online map relies on monocular depth estimation, the point clouds of the generated objects have modal differences, making it difficult to achieve matching based solely on the geometric features of the objects. This embodiment adopts a global data association method based on graph theory, combining the geometric features of the objects, semantic information, and gravity priors. First, a soft association mechanism is used to eliminate outliers in the full-pair full-match set. Then, an affinity matrix is ​​constructed to solve the optimal matching pair and complete robust data association.

[0095] The global data association process is as follows Figure 4 As shown in the figure, to achieve efficient association and registration between object-level maps, the system divides the map into a series of spatially overlapping object subgraphs. Each subgraph contains a set of objects within a certain radius, with the center position as the reference, and the geometric center of each object is uniformly represented in the coordinate system of the subgraph.

[0096] To control the computational overhead of subgraphs, the system sets a maximum number of objects per subgraph and removes objects that are too far from the subgraph center when the number exceeds a threshold. As the robot's pose continues to change, if the current position deviates from the current subgraph center by more than a set threshold, a new subgraph is automatically created to form a fully covered subgraph sequence.

[0097] The core task of global data association is to identify and establish accurate object-level matching relationships between the prior object subgraph and the online object subgraph. This process is subject to high uncertainty and challenges due to the potential for outliers, shape variations, and semantic inconsistencies in object observations. Robust matching mechanisms are required to improve the accuracy and stability of registration.

[0098] This embodiment improves the accuracy of object matching in the absence of prior matching information. This embodiment proposes a soft association mechanism. This mechanism aims to filter out more reliable candidate matching pairs from the initial all-pairs all-match set through dual geometric and semantic constraints, reducing inefficient computation and improving the accuracy of subsequent graph matching results.

[0099] Specifically, we assume that the i-th subgraph of the prior map and the j-th subgraph of the online map contain m and n point cloud segmented objects, respectively. These are referred to as the source set and target set, respectively. In the initial stage, a hypothetical matching set is constructed using an all-to-all approach, generating a total of m × n initial matching pairs. Subsequently, soft association filtering is performed based on the covariance characteristics and semantic information of the objects to eliminate obviously unreasonable matches.

[0100] Specifically, a similarity metric based on geometric information and semantics is designed, which is the "distance", that is, D(p i ,q j) The initial matching pairs are matched using the mutual K-nearest neighbor matching strategy to generate the initial candidate matching set. In this strategy, the matching pairs from each object in the source set to the first K nearest neighbor objects in the target set and the matching pairs from each object in the target set to the first K nearest neighbor objects in the source set are calculated. For any two objects p in the prior subgraph i and the online subgraph j i and q j Define the matching distance between them as:

[0101]

[0102] Among them, D(p i ,q j ) represents the similarity measure based on geometric information and semantics; W 2 (p i ,q j ) represents the Wasserstein distance between two objects in terms of geometric distribution, sim(·) is the cosine similarity between CLIP semantic vectors, and CLIP(·) is the semantic vector representation of the object. At the geometric level, the Wasserstein distance is used to measure the difference between the covariance matrices of two objects:

[0103]

[0104] in, and are the covariance attributes of the two object geometric features, and Tr(·) represents the trace operation of the matrix.

[0105] At the semantic level, the CLIP model is used to extract the semantic vector of the object image region, and the semantic consistency is evaluated by the standard cosine similarity:

[0106]

[0107] On the basis of the above two-way matching, the matching pairs of the source object and the target object that are each other's K nearest neighbors are retained, thereby constructing a mutual K nearest neighbor candidate matching set. This two-way screening mechanism not only improves the recall rate of the true correspondence, but also significantly reduces the false matching, and enhances the accuracy and robustness of the matching results. The final matching pair set A constructed by soft association is s Will be used to construct the affinity matrix.

[0108] The strategy for constructing the affinity matrix in this embodiment is as follows: after completing the soft association, according to the constructed matching pair set A s To further calculate the affinity matrix M, which quantifies the consistency relationship between candidate object matching pairs. In the original form of the affinity matrix, the off-diagonal elements of the matrix only depend on the geometric distance between the object centroids.

[0109] However, in the multimodal object-level map registration involved in this embodiment, since the prior map is built based on lidar, while the online map relies on monocular depth estimation, the point clouds generated by the two have significant differences in density, structural integrity, and distribution characteristics, resulting in greater noise in the object center of mass position and a limited number of valid inlier matches, which increases the ambiguity and uncertainty of data association. To address these issues, this embodiment directly integrates geometric and semantic information into the construction of the affinity matrix M. The matrix elements are defined as follows:

[0110]

[0111] Among them, a p =(p i ,p j ), a q =(q i ,q j ) is from the matching pair set A s The elements in the source set and the target set represent two matching pairs. HM(·) represents the harmonic mean operation used to fuse multiple scoring indicators. a (a p ,a q ) represents the geometric consistency score between two matching pairs, s o (·) consists of semantic similarity and shape similarity, which is calculated as follows:

[0112] s o (a p )=HM(s semantic (a p ),s shape (a p ))

[0113] Among them, the semantic similarity measure s semantic (a p ): obtained by calculating the cosine similarity between the CLIP semantic vectors of two objects:

[0114] s semantic (a p )=sim(CLIP(p i ),CLIP(p j ))

[0115] in:

[0116] Among them, the cosine similarity of CLIP vector pairs in images is usually higher than 0.7, which makes semantic similarity not play a significant role in data association in the following formula:

[0117] s semantic (ap )=sim(CLIP(p i ),CLIP(p j ))

[0118] Using Hyperparameters and Rescale the cosine similarity so that it is less than The score is set to 0, and the score is greater than The score is set to 1.0, between and The scores between are linearly scaled to range from 0 to 1, and finally only the matching relationships with cosine similarity higher than the threshold are retained to eliminate semantically irrelevant object pairs, thereby further improving the matching quality; Indicates the lower limit of semantic cosine degree; Represents the upper limit of semantic cosine degree; shape similarity measure s shape (a p ) is calculated based on the geometric properties of the object, where the shape of the object is represented by a four-dimensional vector f(p) = [f1(p), f2(p), f3(p), f4(p)]; where f1(p) is the volume of the bounding box created by the point cloud of object p, and f2(p), f3(p), and f4(p) represent the linearity, flatness, and scattering properties respectively calculated by covariance. The scoring function takes the following form:

[0119] s shape (a p )=HM(ratio(f(p i ),f(p j )))

[0120] where ratio(·) performs an element-by-element comparison of the scale of each shape dimension. When the difference between the two objects in their shape attributes is small, the score is close to 1; when there is a significant difference, the score is close to 0.

[0121] In this embodiment, the direction of the gravity vector is clear through the onboard IMU, and the s a (·,·), and utilize this additional information in the data association step to guide the solver to choose associations that are consistent with the direction of the gravity vector. Specifically, this gravity vector prior is represented by decoupled calculations in the xy plane and the z axis:

[0122]

[0123] Where σ is the geometric matching tolerance coefficient, which controls the consistency score s a The sensitivity of (·,·) to d(·,·), d xy (a p,a q ) represents the distance deviation between the centroids of two matching pairs on the xy plane, d z (a p ,a q ) represents their difference in the z-axis direction, which can be expressed as:

[0124] d xy (a p ,a q )=|||c xy (p i )-c xy (q i )||-||c xy (p j )-c xy (q j )|||

[0125] d z (a p ,a q )=|||c z (p i )-c z (q i )||-||c z (p j )-c z (q j )|||

[0126] Where, when d(·,·) is less than the threshold ò, this set of distance deviations is considered valid, and ò is the geometric distance threshold.

[0127] Substitute the constructed affinity matrix M into the following optimization problem:

[0128]

[0129] stu i u j =0if M ij =0

[0130] Among them, M ij represents the value of the element in row i and column j in affinity matrix M, u i Represents the i-th element of vector u, u j The jth element of vector u is represented by m, and the length of vector u is m. The overall logic is as follows: First, a soft association is performed to obtain the matching set and As, which has a total of m matching pairs. Then, an affinity matrix is ​​constructed based on these m matching pairs, resulting in an m*m affinity matrix. The size of u, which is then solved after constructing the optimization problem, is also m*1.

[0131] The resulting binary vector u represents the matching relationship between the source and target sets, which in turn determines the matching objects between the prior object subgraph and the online object subgraph, completing the global data association operation. Based on the data association results, the pose is solved to complete global positioning.

[0132] This example analyzes the results of the mapping experiment: In this experiment, segmentation is performed based on image information, and pixel depth information from lidar or monocular depth estimation, as well as odometry data, is combined to convert the segmented objects into three dimensions and construct an object-level map.

[0133] exist Figure 5 The left side of the figure shows multiple segmented objects extracted using the FastSAM algorithm, including vehicles, tires, and manhole covers. Through image segmentation and fusion of depth information, these objects are successfully converted into a 3D point cloud, and their relative positions are shown in the figure. The right side shows the relative spatial positions of the segmented objects. The precise positions of these objects in the robot's local coordinate system provide important information for subsequent object matching and sub-image registration. This process constitutes the construction of an online object map.

[0134] exist Figure 6 In the figure, the process of constructing a priori object maps is shown. The objects in the figure include cars, bushes, etc. The object mask is used to restore the 3D position of the target through the odometry information and lidar point cloud.

[0135] The above mapping process shows that compared to the monocular camera input of the SparkAl2 sequence, the SparkAl2 sequence, combined with the additional depth information provided by the LiDAR, enables more accurate 3D information recovery of segmented objects, especially in complex scenes (such as those containing buildings and vehicles). The relative positional relationships of these segmented objects in the map provide a solid foundation for subsequent sub-image matching, ensuring that matching objects can be accurately found and effective coordinate system transformations can be completed during the map registration process.

[0136] Figure 7 is constructed for object-level maps, where Figure 7 (a) shows the online object map constructed by the sparkal1 sequence moving from the lower right corner to the upper left corner of the image, successfully capturing various segmented objects along the way. There are a total of 213 semantic objects, and the map is divided into 10 sub-graphs; Figure 7 (b) shows the Sparkal2 sequence, also starting from the lower right corner and moving towards the upper left corner, constructing a priori object maps. The different colored blocks in the figure represent the segmented object information of the recovered 3D structure. There are a total of 191 semantic objects, and the map is divided into 12 sub-graphs.

[0137] Analysis of registration experiment results: Four sets of comparative experiments were conducted: using only geometric information, adding soft associations, adding semantic and gravity information, and using soft associations with semantic and gravity information. The corresponding inter-subimage registration errors were calculated, including translation error and rotation error. Translation error was measured as the Euclidean distance between the estimated translation vector and the true translation, while rotation error was calculated as the angular difference between the rotation matrices.

[0138] Figure 8 Heatmap of the registration error between the prior object map and the submap of the online object map. Figure 8 (a) in the figure indicates that only geometric information is used for matching in the global association stage, and the registration result of geometric information is used; Figure 8 (b) in the figure uses the soft association mechanism in the global association stage and adds the registration result of the soft association mechanism; Figure 8 (c) introduces the matching method of semantic and gravity information, and adds semantic and gravity priors to the registration results; Figure 8 Figure (d) shows the method proposed in this example, which uses soft associations along with global associations based on semantic and gravity information. In the figure, the horizontal axis represents the numbers of the 12 subimages in the sparkal2 sequence, i.e., the subimages of the prior object map, and the vertical axis represents the numbers of the 10 subimages in the sparkal1 sequence, i.e., the subimages of the online object map. Each grid represents the error between the estimated result and the ground truth after spatial registration of the subimages in the two object-level maps at the corresponding horizontal and vertical coordinates. The left image represents the rotation error, and the right image represents the translation error. Lighter colors in the figure indicate smaller errors, i.e., higher registration accuracy. The red frame indicates the correct registration pair after comprehensive consideration of translation and rotation errors. It can be observed that some areas, such as the coordinate position (4, 7), between the 8th subimage of the prior object map and the 5th subimage of the online object map, show a distinct lighter color in all four method combinations, indicating that the registration results for this pair of subimages are highly consistent with the ground truth.

[0139] To validate the effectiveness of our method, we evaluated four registration algorithms on the Sparkal (road), Apis (building), and Jackal (mixed road and building) sequences from kimera-multi-data. The results are listed in Table 1. Compared with geometric methods, soft association, and the semantics plus gravity method, the method in this embodiment achieved the lowest angular error (2.782°) and translation error (0.359m), and improved the recall rate to 70.00%. The soft association mechanism helps eliminate obvious mismatches, while the semantic and gravity priors help locate correct correspondences across modalities and viewpoint differences. The combination of these two further improves overall performance.

[0140] Table 1 Registration experimental results

[0141]

[0142] In this embodiment, the experimental parameters are set as follows:

[0143] Table 2 Method parameter settings

[0144]

[0145] Figure 9 Figure 2 shows the global positioning effect of this method, including (a) sparkal sequence, (b) apis sequence, and (c) jackal sequence; green represents the prior object map, and red represents the constructed online object subgraph. After completing the global data association, the pose transformation can be solved by matching the objects in the online subgraph with the objects in the prior map. The above figure shows the specific position of the platform in the prior map. After completing the global positioning, the corresponding positions of the online subgraph and the prior object map roughly coincide. Figure 9 The three figures in the figure are the test results in the sparkal sequence, apis sequence and jackal sequence respectively.

[0146] The above embodiments are merely preferred embodiments for the purpose of fully illustrating the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are within the scope of protection of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A global positioning method based on multi-source information fusion and graph theory, characterized by: The following steps are involved: Step 1: Prior map construction phase: Using the LiDAR point cloud, image sequence, and IMU information, combined with the FastSAM image segmentation algorithm, the objects in the image are segmented and the minimum bounding rectangle of the segmented area is extracted. The segmented area is reconstructed in 3D using the LiDAR point cloud to generate a prior object map. Step 2, online mapping phase: Using image sequences and IMU information, combined with the FastSAM image segmentation algorithm, objects in the image are segmented, and a monocular depth estimation network is used to generate a 3D point cloud to generate an online object map; Step 3, global data association phase: Identify the prior object map and the online object map and establish object-level matching relationships to complete global data association; Step 4, global positioning stage: Based on the results of global data association, solve the pose transformation between the prior object map and the online object map to complete global positioning.

2. The global positioning method based on multi-source information fusion and graph theory according to claim 1, characterized in that: The construction of the prior object map in step 1 further includes the following steps: Step 11) Use YOLOv7 to remove dynamic objects in the image; Step 12) extracting the minimum bounding rectangle of the segmented area and calculating the corresponding CLIP semantic vector; Step 13) Back-project the segmented area into three-dimensional space and perform three-dimensional reconstruction based on the LiDAR point cloud; Step 14) Introducing an object tracking and association mechanism, matching historical objects by voxel grid overlap, and initializing unmatched objects as new objects; Step 15) performing normalized averaging on the CLIP vectors of multiple associated segmented objects to form a stable semantic description; Step 16) introduces an object merging strategy. When two 3D voxels or their 2D projections on the image plane have a high degree of overlap, they are considered to be the same object and merged.

3. The global positioning method based on multi-source information fusion and graph theory according to claim 1, characterized in that: The construction of the online object map in step 2 further includes the following steps: Step 21) Use YOLOv7 to remove dynamic objects in the image; Step 22) Use FastSAM to perform image segmentation and calculate the CLIP semantic vector; Step 23) Generate a 3D point cloud through a monocular depth estimation network and perform 3D reconstruction based on the camera pose information; Step 24) Using the same object tracking and association mechanism as the prior object mapping, historical objects are matched based on voxel grid overlap; Step 25) performing normalized averaging on the CLIP vectors of multiple associated segmented objects to form a stable semantic description; Step 26) introduces an object merging strategy to avoid redundant segmentation due to perspective differences.

4. The global positioning method based on multi-source information fusion and graph theory according to claim 1, characterized in that: The global data association process in step 3 is performed according to the following steps: Step 31) constructing a set of matching pairs between the prior object map and the online object map based on the geometric features, semantic information and gravity prior of the objects; Step 32) Screen candidate matching pairs through a soft association mechanism, using geometric distance and semantic similarity for dual constraints; Step 33) constructing an affinity matrix to quantify the consistency relationship of candidate matching pairs, where the matrix elements integrate geometric consistency, semantic similarity and shape similarity; Step 34) Calculate the optimal matching pair based on the affinity matrix to complete the global data association.

5. The global positioning method based on multi-source information fusion and graph theory according to claim 4, characterized in that: The global data association stage further comprises the following steps: Step 311) dividing the prior object map and the online object map into a plurality of object subgraphs, each subgraph containing a set of objects within a certain range; Step 312) constructing a set of matching pairs between subgraphs, and performing soft association screening using geometric distance and semantic similarity; Step 313) constructing an affinity matrix, wherein the matrix elements incorporate geometric consistency, semantic similarity, and shape similarity; Step 314) Based on the gravity prior, the geometric consistency of the matching pairs is constrained to ensure that the pose transformation of the matching pairs conforms to the direction of the gravity vector.

6. The global positioning method based on multi-source information fusion and graph theory according to claim 4, characterized in that: The soft association mechanism includes the following steps: Step 321) Generate initial matching pairs using full-pair full matching; Step 322) Using the mutual K-nearest neighbor matching strategy to screen candidate matching pairs; Step 323) Calculate the matching distance, including geometric distance and semantic similarity; Step 324) retains the matching pairs of the source object and the target object that are mutually K-nearest neighbors, and constructs a mutual K-nearest neighbor candidate matching set.

7. The global positioning method based on multi-source information fusion and graph theory according to claim 4, characterized in that: The construction of the affinity matrix comprises the following steps: Step 331) defining the matrix elements as the harmonic mean of geometric consistency, semantic similarity and shape similarity; Step 332) Geometric consistency is based on the distance deviation between centroids, semantic similarity is based on CLIP vector cosine similarity, and shape similarity is based on bounding box volume and covariance properties; Step 333) Use the gravity prior to constrain the geometric consistency of the matching pair to ensure that the pose transformation of the matching pair conforms to the direction of the gravity vector.

8. A global positioning system based on multi-source information fusion and graph theory, characterized by: Includes object-level mapping module, global data association module, and global positioning module; The object-level mapping module is used to construct a priori object maps and online object maps; The global data association module is used to establish a matching relationship between the prior object map and the online object map; The global positioning module is used to solve the posture transformation according to the matching results and complete the global positioning; The object-level mapping module includes a priori mapping submodule and an online mapping submodule; The prior mapping submodule uses lidar point clouds, image sequences, and IMU information in combination with the FastSAM image segmentation algorithm to generate a priori object maps; The online mapping submodule uses image sequences and IMU information, combined with the FastSAM image segmentation algorithm and the monocular depth estimation network to generate an online object map; The global data association module includes a soft association submodule, an affinity matrix construction submodule, and an optimization solution submodule; The soft association submodule is used to screen candidate matching pairs through geometric and semantic dual constraints; The affinity matrix construction submodule is used to quantify the consistency relationship of candidate matching pairs; The optimization solution submodule is used to solve the optimal matching pair based on the affinity matrix to complete data association.

9. The global positioning system based on multi-source information fusion and graph theory according to claim 8, characterized in that: The optimization solution submodule is performed according to the following steps: a) Construct an affinity matrix to quantify the consistency relationship of candidate matching pairs; b) Solve the binary vector to determine the matching relationship between the source set and the target set; c) Based on the matching results, solve the pose transformation and complete global positioning.

10. A global positioning system based on multi-source information fusion and graph theory, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images