Multi-robot collaborative semantic SLAM and dynamic exploration method

Abstract

The invention belongs to the technical field of multi-robot collaborative perception and autonomous navigation, and particularly relates to a multi-robot collaborative semantic SLAM and dynamic exploration method, which comprises the following steps: acquiring environmental data; the environment data are input into an improved YOLOv8 model, semantic information of an object is obtained, and the improved YOLOv8 model is obtained by improving the YOLOv8 model; constructing a three-dimensional ellipsoid model based on the semantic information of the object; matching and aligning multi-view data based on object semantic information and the three-dimensional ellipsoid model, and performing global pose joint optimization in combination with a nonlinear optimization algorithm to obtain a high-precision map; based on a high-precision map, dynamic exploration is carried out in combination with multi-robot exploration tasks. According to the method, a collaborative SLAM framework of geometric and semantic information is fused, a layered exploration strategy is designed, and the overall exploration efficiency is remarkably improved while the mapping precision is improved by combining the semantic enhanced perception capability and a task allocation mechanism.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-robot collaborative perception and autonomous navigation, and in particular relates to a multi-robot collaborative semantic SLAM and dynamic exploration method. Background Art

[0002] Traditional simultaneous localization and mapping (SLAM) technology relies on geometric sensors (such as lidar and visual cameras) to build maps and locate robots in unknown environments. However, its core flaw is the lack of semantic understanding ability. It can only perceive the geometric position of obstacles and cannot distinguish object categories (such as doors, equipment areas) or scene functional attributes, making it difficult for robots to perform tasks that require semantic interaction (such as target grasping and avoiding dangerous areas).

[0003] Existing semantic SLAM technologies, by integrating deep learning models (such as YOLO and Mask R-CNN), empower the system with object recognition capabilities. However, their application is limited to single-robot scenarios and struggles to adapt to large-scale dynamic environments (such as disaster relief and industrial inspections). Single-robot systems suffer from low mapping efficiency and limited coverage. Traditional multi-robot collaborative solutions rely on high-bandwidth communication to transmit raw data, which can easily cause network congestion. They also lack a unified semantic-geometric fusion optimization mechanism, leading to accumulated map stitching errors and redundant semantic information.

[0004] Among the current multi-robot collaboration methods, centralized semantic SLAM (such as global server optimization) suffers from insufficient real-time performance due to communication delays, while distributed solutions (such as particle filtering) have difficulty handling dynamic interference and semantic consistency. Therefore, a multi-robot collaboration framework based on lightweight communication, semantic constraint optimization, and distributed fault-tolerant mechanisms is urgently needed to solve the problems of efficient mapping and semantic mapping. Figure 1 Key technical challenges include consistency and multi-machine dynamic collaboration. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes a multi-robot collaborative semantic SLAM and dynamic exploration method, a collaborative SLAM framework that integrates geometric and semantic information, and designs a hierarchical exploration strategy. Combining semantically enhanced perception capabilities with a task allocation mechanism, it significantly improves the overall exploration efficiency while improving mapping accuracy.

[0006] To achieve the above objectives, the present invention proposes a multi-robot collaborative semantic SLAM and dynamic exploration method, comprising:

[0007] Obtain environmental data;

[0008] Inputting the environmental data into an improved YOLOv8 model to obtain object semantic information, wherein the improved YOLOv8 model is obtained by improving the YOLOv8 model;

[0009] constructing a three-dimensional ellipsoid model based on the semantic information of the object;

[0010] Based on the semantic information of the object and the three-dimensional ellipsoid model, the multi-view data is matched and aligned, and the global pose is jointly optimized in combination with a nonlinear optimization algorithm to obtain a high-precision map;

[0011] Based on the high-precision map, dynamic exploration is carried out by combining multiple robot exploration tasks.

[0012] Optionally, the YOLOv8 model is improved by replacing the C3 module with a cross-stage feature compression C2f module, introducing a bidirectional feature pyramid PANet to enhance the cross-scale feature fusion module, and adopting a decoupled design for the head;

[0013] The cross-stage feature compression C2f module is used to reduce computational redundancy through parallel convolution and pooling;

[0014] The bidirectional feature pyramid strengthens the cross-scale feature fusion module to improve the accuracy of small target detection;

[0015] The head is used to optimize bounding box positioning and category prediction.

[0016] Optionally, constructing a three-dimensional ellipsoid model based on the object semantic information includes:

[0017] constructing a dual quadratic surface model based on the semantic information of the object;

[0018] The center and radius of the sphere in the dual quadratic surface model are initialized to obtain the three-dimensional ellipsoid model.

[0019] Optionally, aligning multi-source data based on the object semantic information and the three-dimensional ellipsoid model, and performing global pose joint optimization in combination with a nonlinear optimization algorithm to obtain a high-precision map includes:

[0020] Matching and aligning multi-view data based on the object semantic information and the three-dimensional ellipsoid model to obtain object map points and semantic map points;

[0021] Utilizing the semantic map points to optimize the postures of multiple robots and obtain the posture relationships between the multiple robots;

[0022] Performing global pose optimization on the pose relationships among the multiple robots using a graph optimization method to obtain global pose estimation data;

[0023] The high-precision map is obtained based on the global pose estimation data and the object map points.

[0024] Optionally, aligning the multi-view data based on the object semantic information and the three-dimensional ellipsoid model to obtain the object and semantic map points includes:

[0025] Based on the object semantic information, a preset number of detection frames are set, and the object semantic information is projected to obtain a corresponding number of projection frames;

[0026] Based on the three-dimensional ellipsoid model, calculating the intersection-over-union ratio of the detection frame and the projection frame, and using the intersection-over-union ratio as a correlation score;

[0027] Obtaining the association relationship based on the association score and the Hungarian method;

[0028] The association relationship is optimized using the multi-view data to obtain the object and semantic map point.

[0029] Optionally, performing multi-robot pose optimization using the semantic map points to obtain pose relationships between the multiple robots includes:

[0030] Based on the semantic map points, obtaining a key frame set containing landmark points;

[0031] Processing the keyframe set using a bag-of-words model to obtain a similarity score;

[0032] According to the similarity scores, the posture relationships between the multiple robots are obtained.

[0033] Optionally, building a bag-of-words model includes:

[0034] Based on the key frame set, obtaining a local feature descriptor;

[0035] Clustering the local feature descriptors using a K-Means clustering algorithm to obtain visual words;

[0036] Based on the visual words, obtaining a fork tree dictionary;

[0037] Obtaining a bag-of-words vector based on the appearance frequency of the visual words in the tree dictionary;

[0038] Based on the bag-of-words vector, the bag-of-words model is obtained.

[0039] Optionally, a graph optimization method is used to perform global pose optimization on the pose relationships among the multiple robots, and obtaining global pose estimation data further includes:

[0040] The global posture optimization of the posture relationship between the multiple robots is performed in combination with the objective function, wherein the objective function is:

[0041] ;

[0042] in, is the internal edge of robot A and robot B The posture error, is the cross-robot constraint, i.e. the alignment error of the two robots’ poses, is the second norm of the error, is the overall optimization objective function of the multi-robot pose estimation problem. Its value represents the total error between the current pose estimate and all observation data. n and m are index variables used to identify the observation constraints between different robots in the common area.

[0043] Optionally, based on the high-precision map, performing dynamic exploration in conjunction with a multi-robot exploration task includes:

[0044] Generate a global Voronoi map based on the real-time position of the robot in the high-precision map;

[0045] Based on the global Voronoi diagram, when the robot arrives at a new location and detects that there are no existing nodes within a preset range, a topological node mechanism is introduced, where the topological nodes in the topological node mechanism are used as key path hubs connecting known areas and unknown areas;

[0046] When exploring task allocation, the distance between each robot and the topological node is calculated based on the global Voronoi diagram, and the most efficient task assignment based on location is achieved according to the distance.

[0047] Compared with the prior art, the present invention has the following advantages and technical effects:

[0048] The present invention first proposes a target detection method based on YOLOv8, combines the dual quadratic surface model to model semantic objects, and generates structured semantic map points by fusing image and depth information. A geometric-semantic joint feature matching strategy is designed, and semantic consistency constraints are introduced to effectively improve the robustness and relocation performance of the system in complex indoor scenes. Secondly, a multi-robot collaborative SLAM system architecture is constructed, which performs coarse matching and ICP fine registration across robots based on semantic feature points, and integrates global pose information through graph optimization to achieve collaborative construction of semantic maps among multiple robots. Finally, based on the multi-robot semantic SLAM framework, a "global planning-local adjustment" hierarchical exploration mechanism is designed. Among them, the global layer realizes the real-time division of multi-robot task areas through dynamic Voronoi diagrams, and the local layer dynamically adjusts the path strategy with the help of the topological map network, effectively balancing the load of each robot and improving the efficiency and coverage of collaborative exploration. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0050] Figure 1 This is a flow chart of a multi-robot collaborative semantic SLAM and dynamic exploration method according to an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of a quadratic surface according to an embodiment of the present invention;

[0052] Figure 3 1 is a comparison diagram of semantic map point extraction according to an embodiment of the present invention, wherein (a) is the original image, (b) is a schematic diagram of OA-SLAM extraction, and (c) is a schematic diagram of extraction according to the present method;

[0053] Figure 4 (a) shows the trajectory diagram of the ORB-SLAM method;

[0054] Figure 4 (b) shows the trajectory diagram of the OA-SLAM method;

[0055] Figure 4 (c) is the trajectory diagram of this method;

[0056] Figure 5 2 is a schematic diagram of a multi-robot SLAM framework according to an embodiment of the present invention;

[0057] Figure 6 2 is a schematic diagram of relative pose estimation according to an embodiment of the present invention;

[0058] Figure 7 is a flow chart of relative pose estimation according to an embodiment of the present invention;

[0059] Figure 8 2 is a schematic diagram of multi-robot global pose estimation according to an embodiment of the present invention;

[0060] Figure 9 (a) shows the trajectory of robot 1 in the trajectory fusion result of the self-built laboratory dataset;

[0061] Figure 9 (b) shows the trajectory of robot 2 in the trajectory fusion result of the self-built laboratory dataset;

[0062] Figure 9 (c) shows the global trajectory map of the trajectory fusion results under the self-built laboratory dataset;

[0063] Figure 10 is a Voronoi partition diagram of an embodiment of the present invention;

[0064] Figure 11 (a) is a schematic diagram of the topology node-based allocation scheme without adding topology nodes;

[0065] Figure 11 (b) is a schematic diagram of adding topological nodes to the allocation scheme based on topological nodes;

[0066] Figure 12 is a schematic diagram of multi-robot exploration according to an embodiment of the present invention;

[0067] Figure 13 This is a Gazebo simulation map according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0069] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0070] This embodiment proposes a multi-robot collaborative semantic SLAM and dynamic exploration method, such as Figure 1 As shown, the specific steps include:

[0071] Obtain environmental data;

[0072] Inputting environmental data into an improved YOLOv8 model to obtain semantic information of the object, wherein the improved YOLOv8 model is obtained by improving the YOLOv8 model;

[0073] Construct a three-dimensional ellipsoid model based on the object's semantic information;

[0074] Based on the semantic information of the object and the 3D ellipsoid model, the multi-view data is aligned and matched, and the global pose is jointly optimized with the nonlinear optimization algorithm to obtain a high-precision map.

[0075] Based on high-precision maps, dynamic exploration is carried out by combining multiple robot exploration tasks.

[0076] Specifically, this embodiment first proposes a target detection method based on YOLOv8, combined with a dual quadratic surface model to model semantic objects, and generates structured semantic map points by fusing image and depth information. Furthermore, a geometric-semantic joint feature matching strategy is designed, and semantic consistency constraints are introduced to effectively improve the robustness and relocalization performance of the system in complex indoor scenes. Secondly, a multi-robot collaborative SLAM system architecture is constructed, which performs coarse matching and ICP fine registration across robots based on semantic feature points. Global pose information is fused through graph optimization to achieve collaborative construction of semantic maps among multiple robots. Finally, based on the multi-robot semantic SLAM framework, a "global planning-local adjustment" hierarchical exploration mechanism is designed. The global layer realizes real-time partitioning of multi-robot task areas through dynamic Voronoi diagrams, while the local layer dynamically adjusts path strategies with the help of a topological graph network, effectively balancing the load of each robot and improving collaborative exploration efficiency and coverage.

[0077] Further improvements to the YOLOv8 model include: replacing the C3 module with a cross-stage feature compression C2f module, introducing a bidirectional feature pyramid PANet to enhance the cross-scale feature fusion module, and adopting a decoupled design for the head;

[0078] Cross-stage feature compression C2f module, which is used to reduce computational redundancy through parallel convolution and pooling;

[0079] A bidirectional feature pyramid strengthens the cross-scale feature fusion module to improve the accuracy of small target detection;

[0080] The head is used to optimize bounding box localization and category prediction.

[0081] Specifically, they first proposed a YOLOv8-based object detection method, combined with a dual quadratic surface model to model semantic objects, and generated structured semantic map points by fusing image and depth information. Furthermore, they designed a joint geometric-semantic feature matching strategy and introduced semantic consistency constraints, effectively improving the system's robustness and relocalization performance in complex indoor scenes. Secondly, they constructed a multi-robot collaborative SLAM system architecture, performing coarse matching and ICP fine registration across robots based on semantic feature points. Global pose information was fused through graph optimization to enable collaborative construction of semantic maps among multiple robots. Finally, based on the multi-robot semantic SLAM framework, they designed a hierarchical exploration mechanism based on "global planning and local adjustment." The global layer uses a dynamic Voronoi diagram to achieve real-time partitioning of multi-robot task areas, while the local layer dynamically adjusts path strategies using a topological graph network, effectively balancing the loads of each robot and improving collaborative exploration efficiency and coverage.

[0082] Furthermore, based on the semantic information of the object, constructing a three-dimensional ellipsoid model includes:

[0083] Based on the semantic information of the object, a dual quadratic surface model is constructed;

[0084] Initialize the center and radius of the sphere in the dual quadratic surface model to obtain a three-dimensional ellipsoid model.

[0085] Specifically, in SLAM research, closed quadratic surfaces (such as spheres, ellipsoids, etc.) can be used to represent objects in space. Generally, a quadratic surface is defined by all points on the surface, and its general expression is:

[0086] ;

[0087] Among them, the coefficient are constant coefficients, and x, y, and z are variables. Depending on the values ​​of these coefficients, quadratic surfaces can represent a variety of geometric shapes, including ellipsoids, hyperboloids, paraboloids, etc., and the coefficients of the quadratic terms are not all zero. Write equation (1) in matrix multiplication form:

[0088] ;

[0089] in, is a point on the surface in space , T is the transpose operation, A 4x4 symmetric matrix:

[0090] ;

[0091] Symmetric matrix There are 10 independent parameters, minus the parameter represented by the scale, for a total of 9 degrees of freedom.

[0092] Traditional quadratic surfaces describe points on the surface, but dual quadratic surfaces use tangent planes as basic elements. This representation has significant advantages in SLAM:

[0093] (1) Projection invariance: The dual quadratic surface can maintain a closed-form expression under camera projection, which facilitates the calculation of its projected ellipse on the image plane.

[0094] (2) Parameter decoupling: by decomposing the dual quadratic surface Matrix, the position, attitude and size parameters of the ellipsoid can be directly extracted.

[0095] Figure 2 In , L is four detection edges, is the tangent plane, and X is the intersection of the tangent plane and the ellipsoid, i.e., the tangent point. Any tangent plane , the cut point is , there must be ,therefore ,because On the plane On, so there is , we can get the following formula:

[0096] ;

[0097] Now prove that the tangent plane Must The tangent plane of , then Now we just need to prove On a quadratic surface, due to:

[0098] ;

[0099] therefore, On a quadratic surface. By theorem: the dual quadratic surface of a nondegenerate quadratic surface is still a quadratic surface, and ,get:

[0100] ;

[0101] Let the camera's projection matrix be , for the projection ellipse of the quadratic surface on the camera plane, we can use To express, where K is the camera intrinsic parameter matrix, R is the camera rotation matrix, is the matrix of the dual quadratic curve, which represents the dual form of the quadratic full plane projected on the 2D plane.

[0102] use To represent an ellipsoidal quadratic surface with no translation or rotation at the origin:

[0103] ;

[0104] in, Represents the length of the three axes of the ellipsoid, the transformation matrix ,in, is the rotation matrix, To project a translation vector for an object, you can use:

[0105] , to represent the ellipsoid after coordinate transformation, which is expressed as a vector with 9 degrees of freedom:

[0106] ;

[0107] in, The three parameters represent the lengths of the three semi-axes of the ellipsoid. The three parameters represent the center position of the ellipsoid. Three parameters represent its posture.

[0108] OA-SLAM proposes using a standard sphere to represent objects during initialization, and then optimizing the ellipsoid in subsequent images to improve the efficiency of object reconstruction. This embodiment also adopts the same approach. First, after obtaining the object detection frame, the object is tracked. If the object is tracked in multiple subsequent frames, a rough estimate of the object is created.

[0109] The center of the sphere is obtained by triangulating the center of the multi-frame detection frame. First, the three-dimensional center coordinates of the sphere are defined as , the pixel coordinates of the center of the detection box , through the camera projection matrix:

[0110] ;

[0111] in, is the camera projection matrix, is the camera intrinsic parameter matrix, is the camera rotation matrix, is the projection matrix The row vector of the first row, is the projection matrix The row vector of the second row, is the projection matrix The row vector of the third row, is the camera translation vector.

[0112] The relationship between the pixel coordinates of the sphere center and the three-dimensional center coordinates can be obtained as follows:

[0113] ;

[0114] Will Substituting the above formula into the simplified form, we can get:

[0115] ;

[0116] in, is the horizontal pixel coordinate of the center of the object detection frame, The vertical pixel coordinate of the center of the object detection frame.

[0117] The equation obtained by multi-frame observation can be obtained:

[0118] ;

[0119] in, For the The horizontal pixel coordinate of the center of the object detection frame in the frame, For the The vertical pixel coordinate of the center of the object detection frame in the frame, For the The horizontal pixel coordinate of the center of the object detection frame in the frame, For the The horizontal pixel coordinate of the center of the object detection frame in the frame, For the Frame camera projection matrix The third row vector of For the Frame camera projection matrix The first row vector of For the Frame camera projection matrix The second row vector of For the Frame camera projection matrix The third row vector of For the Frame camera projection matrix The first row vector of For the Frame camera projection matrix The second row vector of .

[0120] The least squares solution can be obtained using singular value decomposition , and normalize it to get the center coordinates of the sphere:

[0121] ;

[0122] in, To obtain the center coordinates of the sphere after normalization, is the unnormalized solution obtained using singular value decomposition Axis component, is the unnormalized solution obtained using singular value decomposition Axis component, is the unnormalized solution obtained using singular value decomposition Axis component, is the homogeneous scaling factor of the solution vector.

[0123] The radius of the sphere is determined by the average size of the multi-frame detection box back-projected into the world coordinate system:

[0124] ;

[0125] Among them, radius represents the average size, The detection frame width of the frame object is , the height is , 、 are the camera internal parameters, For the The depth coordinate of the frame object center in the camera coordinate system, is the number of observation frames.

[0126] Furthermore, based on the semantic information of the object and the matching of the 3D ellipsoid model, multi-source data is aligned, and the global pose is jointly optimized in combination with a nonlinear optimization algorithm to obtain a high-precision map, including:

[0127] Based on the object semantic information and 3D ellipsoid model matching, multi-view data is aligned to obtain objects and semantic map points;

[0128] Use semantic map points to optimize the posture of multiple robots and obtain the posture relationship between multiple robots;

[0129] The graph optimization method is used to perform global pose optimization on the pose relationship between multiple robots to obtain global pose estimation data;

[0130] Obtain high-precision maps based on global pose estimation data.

[0131] Furthermore, based on the object semantic information and the 3D ellipsoid model, the multi-view data is aligned to obtain the object and semantic map points, including:

[0132] Based on the object semantic information, a preset number of detection frames are set, and the object semantic information is projected to obtain a corresponding number of projection frames;

[0133] Based on the 3D ellipsoid model, the intersection-over-union (IoU) of the detection frame and the projection frame is calculated and used as the association score.

[0134] According to the correlation score, the Hungarian method is combined to obtain the correlation relationship;

[0135] Utilize multi-view data to optimize association relationships and obtain objects and semantic map points.

[0136] Specifically, this embodiment relates to a data association and optimization method in a semantic SLAM system, which aims to address the map redundancy and optimization bias problems caused by data association failure in the prior art. In a semantic SLAM system, if the observation data is not accurately associated with the map elements, each detected object will be treated as a new instance, resulting in the accumulation of redundant objects in the map, the failure of back-end optimization constraints, and ultimately affecting positioning and mapping accuracy. To this end, this embodiment proposes a dual association mechanism based on detection boxes and semantic feature points, combined with multi-view geometric verification, to improve the robustness and accuracy of data association.

[0137] This embodiment first adopts a detection box-based object association strategy: when the camera motion between adjacent frames is stable, short-term association is achieved through two-dimensional bounding box tracking. To address the tracking failure problem caused by perspective change, occlusion or motion blur, a three-dimensional ellipsoid model is introduced. The object is projected onto the image plane based on the camera pose, and the intersection over union (IoU) between the detection box and the projected box is calculated as the matching score. The score matrix is ​​constructed and the optimal matching is achieved through the Hungarian algorithm.

[0138] Assume that the kth frame has N detection boxes and M objects are maintained in the current system. Based on the ellipsoid projection model, the intersection-over-union ratio of the projection area and the detection box is calculated as the score:

[0139] ;

[0140] in, is the di-th object detection frame, is the projection frame of the pj-th object, For the Detection box in the frame With detection box degree of matching.

[0141] Construct the score matrix of the kth frame :

[0142] ;

[0143] If the IoU between the projection frame and the detection frame is lower than the threshold, or the projection deviation of the ellipsoid center exceeds the limit, it is judged as an invalid association and the current frame does not participate in the object update.

[0144] To further improve association accuracy, this embodiment proposes an object association method based on semantic feature points: map feature points located within the object detection frame or ellipsoid are bound to the corresponding object, and the association relationship is optimized by combining multi-view observation data. To address the problem of misassociated outliers due to perspective error, detection frame offset, or inaccurate depth estimation, a three-dimensional geometric consistency verification mechanism is adopted: if a feature point is located outside the ellipsoid or exceeds the limit of distance from the geometric center of the object, it is judged as an outlier and removed. At the same time, through multi-view observation redundancy analysis, outliers that have not been consistently observed across multiple frames are removed, ensuring the geometric consistency and purity of the semantic point cloud.

[0145] Compared to traditional Quadric SLAM methods, which suffer from reduced accuracy due to insufficient ellipsoid modeling capabilities, this implementation significantly improves complex object modeling accuracy and relocalization reliability by integrating detection box projection matching, semantic feature point binding, and multi-view verification mechanisms. Experiments show that this method significantly shortens exploration time in multi-robot collaborative scenarios compared to single-robot systems, and reduces path redundancy and the number of nodes generated compared to traditional RRT algorithms, validating its efficiency and robustness in dynamic environments.

[0146] This example conducts a semantic mapping and trajectory accuracy comparison experiment on the "fr2_desk" sequence of the TUM RGB-D dataset. Figure 3 As shown in (a)-(c), the semantic map point extraction effects of the traditional OA-SLAM and the method of this embodiment are compared: OA-SLAM adopts the semantic feature extraction method based on the target detection frame, which is easy to introduce background areas ( Figure 3 (b) in the figure), resulting in inconsistent binding between semantic feature points and objects, affecting subsequent association and pose estimation; the method in this embodiment accurately segments the object contour through semantic segmentation mask ( Figure 3 In (c) above, the geometric consistency and expression accuracy of the extracted semantic feature points are significantly improved, reducing background noise interference.

[0147] In the trajectory reconstruction comparison experiment, as shown in Figure 4 (a), Figure 4 (b) and Figure 4 (c), the trajectory generated by ORB-SLAM (pure geometric feature method) is smooth and highly close to the real trajectory; after the method of this embodiment integrates semantic information, the trajectory continuity and stability are close to ORB-SLAM; while OA-SLAM relies on ellipsoid modeling, and the geometric expression errors accumulate in scenes with insufficient observation or occlusion, resulting in obvious drift in the trajectory in the later stage.

[0148] Experimental results show that after introducing semantic information, the trajectory accuracy of the method in this embodiment is close to that of the pure geometric method (ORB-SLAM), and is significantly better than the traditional semantic SLAM method (such as OA-SLAM), verifying the effectiveness and robustness of the semantic-geometric fusion strategy.

[0149] Furthermore, semantic map points are used to optimize the multi-robot poses and obtain the pose relationships between the multiple robots, including:

[0150] Based on the semantic map points, obtain a set of key frames containing landmark points;

[0151] Use the bag-of-words model to process the keyframe set and obtain the similarity score;

[0152] According to the similarity score, the pose relationship between multiple robots is obtained.

[0153] Specifically, such as Figure 5 As shown in Figure 1, this embodiment uses a centralized architecture to implement a multi-robot collaborative SLAM system. In this architecture, each robot performs local semantic SLAM and uploads keyframes, semantic map points, and relevant object features to a central server. The central server performs global optimization, generating a consistent global map through graph optimization and data fusion.

[0154] The present embodiment relates to the field of multi-robot visual SLAM technology, and in particular to a method for relative pose estimation and map fusion between robots.

[0155] In multi-robot visual SLAM systems, effectively calculating the relative poses between robots to achieve coordinate system transformation and fusion is a key challenge. Existing methods based on physical robot encounter detection rely on robots "meeting" to trigger fusion, but their applicability and effectiveness are limited by the rarity of encounter events in large-scale scenarios.

[0156] To this end, this embodiment provides an improved method. This method recognizes that when robots independently run SLAM, their trajectories often pass through some of the same areas due to exploring the same scene and potential closed-loop detection, forming spatial overlapping observations. By identifying these overlapping areas in their respective maps and matching and associating their feature point information, the coordinate transformation relationship between the robots can be restored, thereby completing the local map splicing and global map splicing. Figure 1 Consistent construction.

[0157] Specifically, if Figure 6 As shown in Figure 1, robots A and B observe the same landmark point P at different times. Using this common landmark point, the pose transformation matrix of robot B relative to robot A can be calculated. However, this type of matching based on visual landmark points has the risk of mismatching.

[0158] In order to reduce this risk, this embodiment introduces Figure 7The decision process is shown in Figure 1. During this process, robots A and B first independently run their own SLAM systems to generate keyframe sets containing landmark points. Subsequently, using a mechanism similar to loop closure detection, the landmark points in the two keyframe sets are traversed and preliminarily matched. Feature descriptors are used to filter and eliminate most obvious mismatches. A bag-of-words model is typically used to find similar locations.

[0159] Furthermore, constructing the dictionary vector includes:

[0160] Based on the key frame set, obtain the local feature descriptor;

[0161] Use K-Means clustering algorithm to cluster local feature descriptors to obtain visual words;

[0162] Based on visual words, obtain a tree dictionary;

[0163] Get the bag-of-words vector based on the frequency of visual words in the tree dictionary;

[0164] Based on the bag-of-words vector, obtain the bag-of-words model.

[0165] Specifically, the dictionary is the core of the bag-of-words model and consists of a set of visual words (i.e., cluster centers of feature descriptors). Essentially, it abstracts and summarizes image features, dividing the high-dimensional feature space into discrete categories to simplify image representation and comparison. The dictionary size (number of visual words) is adjustable: a larger dictionary size results in a more refined representation but increases computational complexity; a smaller dictionary size results in a coarser representation but improves efficiency.

[0166] By mapping image features to visual words, the SLAM system can quickly calculate image similarity for key frame screening, loop detection, scene recognition, etc. Construction steps:

[0167] 1. Feature extraction: Extract local feature descriptors from a large number of images. These feature descriptors are usually high-dimensional vectors that can capture key information in the image.

[0168] 2. Cluster analysis: Use the K-Means clustering algorithm to cluster all feature descriptors and obtain a set of cluster centers, i.e., visual words, as the first layer of K-Means.

[0169] 3. Dictionary generation: Repeat step 2 for each type of descriptor until the number of layers is met, and store the results in a K-ary tree, where the leaf nodes of the K-ary tree are each word.

[0170] 4. Bag-of-Words Vector Generation: For each image, its feature descriptors are mapped to visual words in the dictionary. The frequency of each visual word is counted to generate a bag-of-words vector. A bag-of-words vector is a fixed-length vector with a dimension equal to the size of the dictionary, which can concisely represent the characteristics of the image.

[0171] By building a dictionary, SLAM systems can process image data more efficiently, improving the system's real-time performance and robustness. While computationally intensive, dictionary construction can typically be completed offline. The online phase only requires feature mapping and similarity calculations, significantly reducing computational complexity.

[0172] After constructing the bag of words, each image can be represented as a fixed-length word vector, i.e., a semantic feature histogram. To measure the similarity between two images, the Euclidean distance is often used to compare their bag-of-words vectors.

[0173] For images , the total number of feature points of a word is ,word The number of feature points in the current image is , then its frequency of occurrence in the image (Term Frequency, TF) is:

[0174] ;

[0175] in, is the word frequency of word w in the image;

[0176] However, directly using word frequency can be biased. Certain common semantic words (such as "ground" and "wall") appear frequently in most images, making them less useful for distinguishing images. Therefore, we introduce the inverse document frequency (IDF) to weight word frequency.

[0177] For a certain image , No. The weight of each word is calculated as follows:

[0178] ;

[0179] in, is the total number of features contained in the dictionary, For words The number of features included. ;

[0180] in, For words The TF-IDF weight of For words The word frequency, For words The inverse document frequency of .

[0181] Represent each image as a vector containing all dictionary words , by comparing the vectors of two images, we can get their similarity scores, where, To indicate the first TF-IDF weight of visual words.

[0182] ;

[0183] in, is the similarity score of the two images, The first picture contains the vector of all dictionary words, The vector containing all the dictionary words for the second image.

[0184] The score is used to determine whether there is a similar position. When the score is greater than the threshold, there is a similar position, and the two frames of images are matched and the relative posture is transformed.

[0185] Furthermore, the graph optimization method is used to perform global pose optimization on the pose relationship between multiple robots, and the global pose estimation data is obtained including:

[0186] In order to further improve the consistency of the overall map and the accuracy of global pose estimation, this embodiment adopts a graph optimization method in the back-end of multi-robot SLAM, which is similar to the graph optimization framework commonly used in single-robot systems. Each node in the graph represents the pose of a keyframe, and the edge represents the observation constraint between adjacent keyframes. After detecting the common area, an edge (i.e., cross-robot constraint) is added between the keyframes that are successfully matched between robot A and robot B, indicating the common view relationship between the two robots observing the same scene at this location. Figure 8 As shown, if robot A and robot B successfully establish a posture matching relationship in the common area, the corresponding transformation matrix is ​​recorded as . Indicates that robot A is and robot B The transfer matrix at , which constitutes a new constraint edge in the graph.

[0187] In graph optimization, the error term is used to measure the deviation between the actual observation and the current estimate. The overall optimization objective function can be expressed as:

[0188] ;

[0189] in, is the internal edge of robot A and robot B The posture error, is the cross-robot constraint, i.e. the alignment error of the two robots’ poses, is the second norm of the error, is the overall optimization objective function for the multi-robot pose estimation problem. Its value represents the total error between the current pose estimate and all observations. n and m are index variables used to identify the observation constraints between different robots in the common area. The first and second terms are the dichotomous norms of the errors in the internal maps of robots A and B, respectively, while the third term is the dichotomous norm of the errors in observations between robots in the common area. Using the Gauss-Newton method for iterative optimization can reduce the inter-robot error. This is because the introduction of the global error between robots provides more information for optimization in the global map, theoretically leading to more accurate global pose estimation results.

[0190] To verify the effectiveness of the proposed multi-robot semantic SLAM system in real-world scenarios, this example conducted an experimental evaluation on the public TUM RGB-D dataset, splitting the dataset into two unrelated parts. The focus was on examining the system's performance in fusion of different robot trajectories and global pose optimization.

[0191] Figures 9(a), 9(b), and 9(c) show the fusion results of two robot trajectories using a laboratory-built dataset. Figures 9(a) and 9(b) represent the original trajectories of Robot 1 and Robot 2, respectively. Due to different starting positions, the two robots construct maps in their respective local coordinate systems, resulting in significant offsets. Figure 9(c) shows the alignment of the two trajectories in the global coordinate system after similar position detection and semantic ICP fine registration. The red lines represent the established cross-robot constraint connections, the blue dots represent the starting position of Robot 1, and the orange squares represent the starting position of Robot 2. The fused global trajectories are precisely aligned in space, verifying the adaptability and consistency of this embodiment in real-world environments.

[0192] Furthermore, based on high-precision maps, dynamic exploration is carried out by combining multiple robots to perform exploration tasks, including:

[0193] Generate a global Voronoi diagram based on the real-time position of the robot in the high-precision map;

[0194] Based on the global Voronoi diagram, when the robot arrives at a new location and detects that there are no existing nodes within the preset range, a topological node mechanism is introduced. The topological nodes in the topological node mechanism are used to connect the key path hubs of the known area and the unknown area.

[0195] When exploring task allocation, the distance between each robot and the topological node is calculated based on the global Voronoi diagram, and the most efficient task assignment based on location is achieved according to the distance.

[0196] Specifically, in multi-robot autonomous exploration tasks, a reasonable task allocation strategy is key to improving exploration efficiency. Traditional methods can lead to uneven distribution of task areas, excessive path intersections between robots, and even insufficient exploration of some areas. To address this, this embodiment proposes a hierarchical task allocation method that combines global Voronoi diagram segmentation with local topological node guidance to achieve reasonable task division and dynamic adjustment, thereby improving overall exploration efficiency.

[0197] Step 1: The system core first dynamically generates a global Voronoi map based on the real-time positions of all robots, using them as seed points. This map automatically divides the entire unknown environment into multiple Voronoi cells, each containing all the spatial points closest to its corresponding robot.

[0198] During initial task allocation, each robot is prioritized for exploration within its own Voronoi cell. This spatial proximity-based allocation effectively ensures an initial uniform distribution of robots in the environment, avoiding resource waste and concentrated work.

[0199] To accommodate the dynamic nature of exploration (robot movement and environmental updates), the system periodically updates the Voronoi diagram. Crucially, once a robot completes its current unit's mission, the system automatically removes its geographical restrictions, enabling it to collaboratively explore the remaining area across units, thereby dynamically optimizing overall resource utilization and exploration efficiency.

[0200] Step 2: However, the Voronoi diagram may suffer from unreasonable partitioning (e.g., path intersections and exploration blind spots) in complex obstacle environments. To address this key challenge, this embodiment innovatively introduces a topological node guidance mechanism based on the Voronoi partitioning framework.

[0201] In Figures 11(a) and 11(b), the robot autonomously generates key topological nodes during its movement: when it arrives at a new location and detects that there are no existing nodes within its preset range, it creates a new node at that location and connects it to the topological network. These nodes become key path hubs connecting known and unknown areas.

[0202] The system breaks down the large area to be explored into frontier sub-areas that match the field of view (FOV) of the robot's camera. (A larger FOV results in larger sub-areas, while a smaller FOV results in finer divisions.) Each topological node connects multiple such frontier sub-areas, forming a dynamic task network.

[0203] Topological nodes are managed in two states: "Unexplored" (connected sub-regions to be explored) and "Completed" (all connected sub-regions explored). When assigning tasks, the system directly uses the Voronoi diagram framework to calculate the distance between each robot and the node, prioritizing "unexplored" nodes to the nearest idle robot, achieving optimal location-based task assignment.

[0204] Within their respective assigned Voronoi cells, the robots use the Dijkstra algorithm to calculate the shortest path to the frontier sub-area they are connected to based on the topological node network, forming an efficient local exploration route and effectively avoiding path conflicts and blind spots.

[0205] Step 3: The powerful collaborative ability of this method comes from the real-time linkage monitoring and response of the Voronoi unit state and the topological node state.

[0206] The system continuously tracks the status of topological nodes: once all frontier sub-areas connected to a node have been explored, it is marked as "completed." More importantly, if all topological nodes within a Voronoi cell are detected to be in the "completed" state, the cell is deemed mission-complete, and its robots are immediately released, allowing them to actively request assistance from adjacent, unfinished cells.

[0207] Especially in the regional boundary areas, when multiple robots are in the collaborative range, the system comprehensively considers the real-time distance between each robot and the "uncompletely explored" nodes at the boundary, the current task load and completion progress, and dynamically fine-tunes the Voronoi partition boundaries to achieve seamless task takeover and cross-regional collaboration, greatly improving collaborative efficiency.

[0208] To ensure the real-time and simplicity of the topological network, new nodes are continuously generated and connected as the robot explores. At the same time, strict minimum spacing constraints are used to avoid the generation of redundant nodes, and historical nodes are recorded in different states.

[0209] exist Figure 12 In the paper, two robots divide the entire map into different areas. Each area contains two topological nodes, and the topological nodes are connected to the unknown areas closest to them according to the Dijkstra tree. The nodes are assigned to the two robots according to the distance, and the robots explore in their respective local areas.

[0210] In order to verify the effectiveness and superiority of the proposed multi-robot collaborative exploration algorithm based on Voronoi diagram partitioning and dynamic task allocation of topological nodes, this embodiment designed and built a complete simulation verification platform, integrated ROS (Robot Operating System) and Gazebo simulator, and realized map construction and robot status monitoring through RViz visualization tool. Figure 10 As shown, the exploration scene is a The indoor map contains multiple rooms, obstacle walls and open areas, which can effectively simulate the typical exploration challenges faced by mobile robots in real environments. The experimental robot is Turtlebot3Burger, equipped with a Kinect v2 depth camera to obtain environmental information, such as Figure 13 shown.

[0211] In a single-robot exploration experiment, the robot starts from its initial position, rotates in place to scan the environment, collects point cloud information, extracts boundary detection frontier points, then extracts frontier blocks centered on its current position, constructs initial topological nodes, and establishes connections. The robot then proceeds to the target block based on the shortest path planning, dynamically generating new nodes and updating connections, maintaining minimum spacing constraints between nodes to avoid redundant generation. Ultimately, it achieves full coverage of the area and forms a complete topological network. Experimental results show that this embodiment generates eight topological nodes in 94 seconds, reducing path redundancy by over 30% compared to the traditional RRT algorithm (128 seconds, 39 nodes) and increasing node utilization by 79.5%, demonstrating its significant advantages in path coherence, coverage completeness, and computational efficiency.

[0212] In a single-robot scenario, this method generates 8 topological nodes in a total of 94 seconds, achieving full coverage of the environment. Compared with the traditional RRT algorithm (which takes 128 seconds and generates 39 nodes), it reduces redundant paths by more than 30%, increases node utilization by 79.5%, and shortens path length by 23%, verifying its significant advantages in path continuity, coverage integrity, and computational efficiency.

[0213] For multi-robot collaboration scenarios, this example deploys three TurtleBot3 robots in a 20m×20m simulation environment, achieving communication and task coordination through ROS. A central module generates a Voronoi diagram based on global frontier information, dynamically partitions the task area, and uniquely assigns frontier blocks to each robot. The partitions are updated in real time to minimize interference while maintaining node spacing constraints. Experiments show that the multi-robot collaborative system completes the entire map in 39 seconds, a 58.5% improvement over a single robot. It also achieves complete topological coverage of the entire map, avoiding overlap and collision risks, and provides an efficient and robust solution for collaborative operations in large-scale dynamic environments.

[0214] To further validate the superiority of the technical solution of this embodiment, a comparative experiment comparing the proposed exploration method based on Voronoi partitioning and dynamic topology scheduling with a two-layer RRT algorithm was conducted under identical map conditions and robot configurations. The experimental results show that the conventional RRT algorithm suffers from inefficient path planning due to its random sampling mechanism, particularly in areas with dense obstacles. Redundant paths are often generated, and overlapping paths among multiple robots lead to task conflicts and execution delays. Statistics from 10 repeated experiments, as shown in Table 1, show that the RRT algorithm completes map exploration in an average of 57 seconds, a 46.2% improvement over the 39 seconds of this embodiment. Furthermore, the number of topological nodes generated by the RRT algorithm increases by over 35%, and the average path length increases by 28.3%. This embodiment, through dynamic Voronoi diagram partitioning and topological node guidance, achieves balanced task area allocation and optimized path separation, achieving a stable 100% map coverage rate and reducing path redundancy by 40%, demonstrating its efficiency and robustness in complex scenarios.

[0215] Table 1: Statistics after 10 repeated tests

[0216] method Total time (s) Number of topological nodes Average path length (m) Map coverage (%) The method of this embodiment 39±2.3 #timg# #timg# 98.6 RRT method #timg# #timg# #timg# 95.3

[0217] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multi-robot collaborative semantic SLAM and dynamic exploration method, characterized in that: include: Obtain environmental data; Inputting the environmental data into an improved YOLOv8 model to obtain object semantic information, wherein the improved YOLOv8 model is obtained by improving the YOLOv8 model; constructing a three-dimensional ellipsoid model based on the semantic information of the object; Based on the semantic information of the object and the three-dimensional ellipsoid model, the multi-view data is matched and aligned, and the global pose is jointly optimized in combination with a nonlinear optimization algorithm to obtain a high-precision map; Based on the high-precision map, dynamic exploration is carried out by combining multiple robot exploration tasks.

2. A multi-robot collaborative semantic SLAM and dynamic exploration method according to claim 1, characterized in that, Improvements to the YOLOv8 model include replacing the C3 module with a cross-stage feature compression C2f module, introducing a bidirectional feature pyramid PANet to enhance the cross-scale feature fusion module, and adopting a decoupled design for the head. The cross-stage feature compression C2f module is used to reduce computational redundancy through parallel convolution and pooling; The bidirectional feature pyramid strengthens the cross-scale feature fusion module to improve the accuracy of small target detection; The head is used to optimize bounding box positioning and category prediction.

3. A multi-robot collaborative semantic SLAM and dynamic exploration method according to claim 1, characterized in that, Constructing a three-dimensional ellipsoid model based on the object semantic information includes: constructing a dual quadratic surface model based on the semantic information of the object; The sphere center and radius in the dual quadratic surface model are initialized to obtain the three-dimensional ellipsoid model.

4. A multi-robot collaborative semantic SLAM and dynamic exploration method according to claim 1, characterized in that, Based on the semantic information of the object and the 3D ellipsoid model, the multi-source data is matched and aligned, and the global pose is jointly optimized in combination with a nonlinear optimization algorithm to obtain a high-precision map, including: Matching and aligning multi-view data based on the object semantic information and the three-dimensional ellipsoid model to obtain object map points and semantic map points; Utilizing the semantic map points to optimize the postures of multiple robots and obtain the posture relationships between the multiple robots; Performing global pose optimization on the pose relationships among the multiple robots using a graph optimization method to obtain global pose estimation data; The high-precision map is obtained based on the global pose estimation data and the object map points.

5. A multi-robot collaborative semantic SLAM and dynamic exploration method according to claim 4, characterized in that: Acquiring object map points and semantic map points based on the object semantic information and the three-dimensional ellipsoid model matching and aligning the multi-view data includes: Based on the object semantic information, a preset number of detection frames are set, and the object semantic information is projected to obtain a corresponding number of projection frames; Based on the three-dimensional ellipsoid model, calculating the intersection-over-union ratio of the detection frame and the projection frame, and using the intersection-over-union ratio as a correlation score; According to the association score, the association relationship is obtained by combining the Hungarian method; The association relationship is optimized using the multi-view data to obtain the object map point and the semantic map point.

6. A multi-robot collaborative semantic SLAM and dynamic exploration method according to claim 4, characterized in that: Utilizing the semantic map points to optimize the multi-robot posture and obtain the posture relationship between the multiple robots includes: Based on the semantic map points, obtaining a key frame set containing landmark points; Processing the keyframe set using a bag-of-words model to obtain a similarity score; According to the similarity scores, the posture relationships between the multiple robots are obtained.

7. A multi-robot collaborative semantic SLAM and dynamic exploration method according to claim 6, characterized in that: Building a bag-of-words model involves: Based on the key frame set, obtaining a local feature descriptor; Clustering the local feature descriptors using a K-Means clustering algorithm to obtain visual words; Based on the visual words, obtaining a fork-tree dictionary; Obtaining a bag-of-words vector based on the appearance frequency of the visual words in the tree dictionary; Based on the bag-of-words vector, the bag-of-words model is obtained.

8. A multi-robot collaborative semantic SLAM and dynamic exploration method according to claim 4, characterized in that: Using a graph optimization method to perform global pose optimization on the pose relationships among the multiple robots, obtaining global pose estimation data further includes: The global posture optimization of the posture relationship between the multiple robots is performed in combination with the objective function, wherein the objective function is: ； in, is the internal edge of robot A and robot B The posture error, is the cross-robot constraint, i.e. the alignment error of the two robots’ poses, is the second norm of the error, is the overall optimization objective function of the multi-robot pose estimation problem. Its value represents the total error between the current pose estimate and all observation data. n and m are index variables used to identify the observation constraints between different robots in the common area.

9. A multi-robot collaborative semantic SLAM and dynamic exploration method according to claim 1, characterized in that: Based on the high-precision map, dynamic exploration is performed by combining multiple robots to perform exploration tasks, including: Generate a global Voronoi map based on the real-time position of the robot in the high-precision map; Based on the global Voronoi diagram, when the robot arrives at a new location and detects that there are no existing nodes within a preset range, a topological node mechanism is introduced, where the topological nodes in the topological node mechanism are used as key path hubs connecting known areas and unknown areas; When exploring task allocation, the distance between each robot and the topological node is calculated based on the global Voronoi diagram, and the most efficient task assignment based on location is achieved according to the distance.

Images