Multi-robot collaborative semantic slam and dynamic exploration method

Abstract

The application belongs to the technical field of multi-robot cooperative perception and autonomous navigation, and particularly relates to a multi-robot cooperative semantic SLAM and dynamic exploration method, which comprises the following steps: acquiring environment data; inputting the environment data into an improved YOLOv8 model to obtain object semantic information, wherein the improved YOLOv8 model is obtained by improving the YOLOv8 model; constructing a three-dimensional ellipsoid model based on the object semantic information; matching and aligning multi-view data based on the object semantic information and the three-dimensional ellipsoid model, and performing global pose joint optimization combined with a nonlinear optimization algorithm to obtain a high-precision map; and performing dynamic exploration based on the high-precision map and in combination with a multi-robot exploration task. The application fuses a cooperative SLAM framework of geometric and semantic information, and designs a layered exploration strategy, thereby combining a semantic-enhanced perception capability and a task allocation mechanism to improve the mapping precision and significantly improve the overall exploration efficiency.

Description

Technical Field

[0001] This invention belongs to the field of multi-robot collaborative perception and autonomous navigation technology, and particularly relates to a multi-robot collaborative semantic SLAM and dynamic exploration method. Background Technology

[0002] Traditional Simultaneous Localization and Mapping (SLAM) technology relies on geometric sensors (such as LiDAR and visual cameras) to build maps and locate robots in unknown environments. However, its core deficiency lies in the lack of semantic understanding capabilities. It can only perceive the geometric position of obstacles and cannot distinguish object categories (such as doors and equipment areas) or scene functional attributes, making it difficult for robots to perform tasks that require semantic interaction (such as target grasping and danger zone avoidance).

[0003] Existing semantic SLAM technologies endow systems with object recognition capabilities by fusing deep learning models (such as YOLO and Mask R-CNN), but their applications are mostly limited to single-robot scenarios and are difficult to adapt to large-scale dynamic environments (such as disaster relief and industrial inspection). Single-robot systems suffer from bottlenecks such as low mapping efficiency and limited coverage, while traditional multi-robot collaborative solutions rely on high-bandwidth communication to transmit raw data, which is prone to network congestion and lacks a unified semantic-geometric fusion optimization mechanism, resulting in the accumulation of map stitching errors and semantic information redundancy.

[0004] In current multi-robot collaborative methods, centralized semantic SLAM (such as global server optimization) suffers from insufficient real-time performance due to communication latency, while distributed schemes (such as particle filtering) struggle to handle dynamic interference and semantic consistency. Therefore, there is an urgent need for a multi-robot collaborative framework based on lightweight communication, semantic constraint optimization, and distributed fault-tolerant mechanisms to address efficient mapping and semantic consistency. Figure 1 Key technological challenges include consistency and dynamic multi-machine collaboration. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention proposes a multi-robot collaborative semantic SLAM and dynamic exploration method. It integrates a collaborative SLAM framework that combines geometric and semantic information, and designs a hierarchical exploration strategy. By combining semantically enhanced perception capabilities and a task allocation mechanism, it significantly improves overall exploration efficiency while enhancing mapping accuracy.

[0006] To achieve the above objectives, this invention proposes a multi-robot cooperative semantic SLAM and dynamic exploration method, comprising:

[0007] Acquire environmental data;

[0008] The environmental data is input into the improved YOLOv8 model to obtain object semantic information, wherein the improved YOLOv8 model is obtained by improving the YOLOv8 model;

[0009] Based on the semantic information of the object, a three-dimensional ellipsoid model is constructed;

[0010] Based on the semantic information of the object and the three-dimensional ellipsoid model, multi-view data is matched and aligned, and global pose joint optimization is performed by combining nonlinear optimization algorithm to obtain a high-precision map.

[0011] Based on the high-precision map, dynamic exploration is carried out in conjunction with multi-robot exploration tasks.

[0012] Optionally, improvements to the YOLOv8 model include: replacing the C3 module with the cross-stage feature compression C2f module, introducing a bidirectional feature pyramid PANet to enhance the cross-scale feature fusion module, and adopting a decoupled design for the head.

[0013] The cross-stage feature compression C2f module is used to reduce computational redundancy through parallel convolution and pooling;

[0014] The bidirectional feature pyramid enhanced cross-scale feature fusion module is used to improve the detection accuracy of small targets;

[0015] The header is used to optimize bounding box localization and category prediction.

[0016] Optionally, constructing a three-dimensional ellipsoid model based on the semantic information of the object includes:

[0017] Based on the semantic information of the object, a dual quadratic surface model is constructed;

[0018] The center and radius of the sphere in the dual quadric surface model are initialized to obtain the three-dimensional ellipsoid model.

[0019] Optionally, based on the semantic information of the object and the 3D ellipsoid model, multi-source data is matched and aligned, and a global pose joint optimization is performed using a nonlinear optimization algorithm to obtain a high-precision map, including:

[0020] Based on the object semantic information and the 3D ellipsoid model, match and align multi-view data to obtain object map points and semantic map points;

[0021] The semantic map points are used to optimize the pose of multiple robots and obtain the pose relationships between multiple robots.

[0022] A graph optimization method is used to perform global pose optimization on the pose relationships between the multiple robots to obtain global pose estimation data.

[0023] The high-precision map is obtained based on the global pose estimation data and object map points.

[0024] Optionally, based on the object semantic information and the 3D ellipsoid model, matching and aligning multi-view data to obtain object and semantic map points includes:

[0025] Based on the semantic information of the object, a preset number of detection boxes are set, and the semantic information of the object is projected to obtain the corresponding number of projection boxes.

[0026] Based on the three-dimensional ellipsoid model, the intersection-union ratio (IUU) of the detection box and the projection box is calculated, and the IUU is used as the association score.

[0027] The association relationship is obtained based on the association score and the Hungarian method.

[0028] The multi-view data is used to optimize the relationship and obtain the object and semantic map points.

[0029] Optionally, using the semantic map points to optimize the poses of multiple robots and obtain the pose relationships between them includes:

[0030] Based on the semantic map points, obtain a set of keyframes containing landmarks;

[0031] The keyframe set is processed using the bag-of-words model to obtain a similarity score;

[0032] Based on the similarity score, the pose relationship between the multiple robots is obtained.

[0033] Optionally, constructing a bag-of-words model includes:

[0034] Based on the set of keyframes, obtain local feature descriptors;

[0035] The local feature descriptors are clustered using the K-Means clustering algorithm to obtain visual words;

[0036] Based on the visual words, obtain a cross-tree dictionary;

[0037] Based on the frequency of occurrence of the visual words in the cross-tree dictionary, obtain the bag-of-words vector;

[0038] Based on the bag-of-words vectors, the bag-of-words model is obtained.

[0039] Optionally, using graph optimization to perform global pose optimization on the pose relationships between the multiple robots, and obtaining global pose estimation data further includes:

[0040] Global pose optimization is performed on the pose relationships among the multiple robots by combining an objective function, wherein the objective function is:

[0041] ;

[0042] in, The internal edges of robot A and robot B The pose error, It is a cross-robot constraint term, namely the pose alignment error between the two robots. It is the L2 norm of the error. The objective function for the overall optimization of the multi-robot pose estimation problem is denoted by , where represents the total error between the current pose estimation and all observation data. n and m are index variables used to identify the observation constraints between different robots in the common area.

[0043] Optionally, based on the high-precision map, dynamic exploration using a multi-robot exploration task includes:

[0044] A global Venn diagram is generated based on the robot's real-time position in the high-precision map.

[0045] Based on the global Venn diagram, when the robot arrives at a new location and detects no existing nodes within a preset range, a topology node mechanism is introduced. The topology nodes in the topology node mechanism are used to connect critical path hubs between known and unknown regions.

[0046] When exploring task allocation, the distance between each robot and the topological node is calculated based on the global Venograph, and the best task assignment based on the location is realized according to the distance.

[0047] Compared with the prior art, the present invention has the following advantages and technical effects:

[0048] This invention first proposes a YOLOv8-based object detection method, combining a dual quadratic surface model to model semantic objects, and generating structured semantic map points through the fusion of image and depth information. A geometry-semantic joint feature matching strategy is designed, introducing semantic consistency constraints to effectively improve the system's robustness and relocalization performance in complex indoor scenes. Secondly, a multi-robot collaborative SLAM system architecture is constructed, performing coarse matching and ICP fine registration across robots based on semantic feature points. Global pose information is fused through graph optimization to achieve collaborative construction of semantic maps among multiple robots. Finally, based on the multi-robot semantic SLAM framework, a hierarchical exploration mechanism of "global planning—local adjustment" is designed. The global layer uses a dynamic Voronoi diagram to achieve real-time division of the multi-robot task area, while the local layer dynamically adjusts the path strategy using a topological graph network, effectively balancing the load of each robot and improving collaborative exploration efficiency and coverage. Attached Figure Description

[0049] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 This is a flowchart of a multi-robot collaborative semantic SLAM and dynamic exploration method according to an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of a quadratic surface according to an embodiment of the present invention;

[0052] Figure 3 These are comparison images of semantic map point extraction according to embodiments of the present invention, wherein (a) is the original image, (b) is a schematic diagram of OA-SLAM extraction, and (c) is a schematic diagram of extraction using the present method.

[0053] Figure 4(a) shows the trajectory of the ORB-SLAM method;

[0054] Figure 4(b) shows the trajectory of the OA-SLAM method;

[0055] Figure 4(c) shows the trajectory of this method;

[0056] Figure 5 This is a schematic diagram of the multi-robot SLAM framework according to an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram of relative pose estimation according to an embodiment of the present invention;

[0058] Figure 7 This is a flowchart of the relative pose estimation process according to an embodiment of the present invention;

[0059] Figure 8 This is a schematic diagram of global pose estimation for multiple robots according to an embodiment of the present invention;

[0060] Figure 9(a) shows the trajectory of robot 1 in the trajectory fusion results under the self-built laboratory dataset;

[0061] Figure 9(b) shows the trajectory of robot 2 in the trajectory fusion results under the self-built laboratory dataset;

[0062] Figure 9(c) shows the global trajectory map in the trajectory fusion results under the self-built laboratory dataset;

[0063] Figure 10 This is a Voronoi partitioning diagram according to an embodiment of the present invention;

[0064] Figure 11(a) is a schematic diagram of the allocation scheme based on topology nodes without adding topology nodes;

[0065] Figure 11(b) is a schematic diagram of adding topology nodes in the topology node-based allocation scheme;

[0066] Figure 12 This is a schematic diagram of multi-robot exploration according to an embodiment of the present invention;

[0067] Figure 13 This is a Gazebo simulation map according to an embodiment of the present invention. Detailed Implementation

[0068] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0069] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0070] This embodiment proposes a multi-robot cooperative semantic SLAM and dynamic exploration method, such as... Figure 1 As shown, the specific steps include:

[0071] Acquire environmental data;

[0072] Environmental data is input into the improved YOLOv8 model to obtain semantic information about objects. The improved YOLOv8 model is obtained by improving the YOLOv8 model.

[0073] Construct a three-dimensional ellipsoid model based on the semantic information of the object;

[0074] High-precision maps are obtained by matching and aligning multi-view data based on object semantic information and 3D ellipsoid model, and combining global pose joint optimization with nonlinear optimization algorithm.

[0075] Dynamic exploration is carried out by combining multiple robots based on high-precision maps.

[0076] Specifically, this embodiment first proposes a YOLOv8-based object detection method, combining a dual quadratic surface model to model semantic objects, and generating structured semantic map points through the fusion of image and depth information. Furthermore, a geometry-semantic joint feature matching strategy is designed, introducing semantic consistency constraints to effectively improve the system's robustness and relocalization performance in complex indoor scenes. Secondly, a multi-robot collaborative SLAM system architecture is constructed, performing coarse matching and ICP fine registration across robots based on semantic feature points. Global pose information is fused through graph optimization to achieve collaborative construction of semantic maps among multiple robots. Finally, based on the multi-robot semantic SLAM framework, a hierarchical exploration mechanism of "global planning—local adjustment" is designed. The global layer uses a dynamic Voronoi diagram to achieve real-time division of the multi-robot task area, while the local layer dynamically adjusts the path strategy using a topological graph network, effectively balancing the load of each robot and improving collaborative exploration efficiency and coverage.

[0077] Further improvements to the YOLOv8 model include: replacing the C3 module with the cross-stage feature compression C2f module, introducing the bidirectional feature pyramid PANet to enhance the cross-scale feature fusion module, and adopting a decoupled design for the head.

[0078] The cross-stage feature compression C2f module is used to reduce computational redundancy through parallel convolution and pooling;

[0079] A bidirectional feature pyramid enhances the cross-scale feature fusion module to improve the detection accuracy of small targets;

[0080] The head is used to optimize bounding box localization and category prediction.

[0081] Specifically, this paper first proposes a YOLOv8-based object detection method, combining a dual quadratic surface model to model semantic objects, and generating structured semantic map points through the fusion of image and depth information. Furthermore, a geometry-semantic joint feature matching strategy is designed, introducing semantic consistency constraints to effectively improve the system's robustness and relocalization performance in complex indoor scenes. Secondly, a multi-robot collaborative SLAM system architecture is constructed, performing coarse matching and ICP fine registration across robots based on semantic feature points. Global pose information is fused through graph optimization to achieve collaborative construction of semantic maps among multiple robots. Finally, based on the multi-robot semantic SLAM framework, a hierarchical exploration mechanism of "global planning—local adjustment" is designed. The global layer uses a dynamic Voronoi diagram to achieve real-time division of the multi-robot task area, while the local layer dynamically adjusts the path strategy using a topological graph network, effectively balancing the load of each robot and improving collaborative exploration efficiency and coverage.

[0082] Furthermore, based on the semantic information of the object, the construction of a three-dimensional ellipsoid model includes:

[0083] Construct a dual quadratic surface model based on object semantic information;

[0084] Initialize the center and radius of the sphere in the dual quadric surface model to obtain the three-dimensional ellipsoid model.

[0085] Specifically, in SLAM research, objects in space can be represented using quadratic surfaces with closed forms (such as spheres, ellipsoids, etc.). Generally, a quadratic surface is defined by all points on the surface, and its general expression is as follows:

[0086] ;

[0087] Among them, coefficient The constant coefficients are x, y, and z, which are variables. Depending on the different values ​​of these coefficients, a quadratic surface can represent various geometric shapes, including ellipsoids, hyperboloids, parabolas, etc., and the coefficients of the quadratic terms are not all zero. Equation (1) can be written in matrix multiplication form as follows:

[0088] ;

[0089] in, A point in space on a curved surface T represents the transpose operation. It is a 4x4 symmetric matrix:

[0090] ;

[0091] Symmetric matrix There are 10 independent parameters. Subtracting the parameter represented by the scale, the actual degrees of freedom is 9.

[0092] Traditional quadric surfaces describe points on the surface, but dual quadric surfaces use tangent planes as their basic elements. This representation has significant advantages in SLAM:

[0093] (1) Projection invariance: The expression of the dual quadratic surface that can maintain its closed form under camera projection is convenient for calculating its projection ellipse on the image plane.

[0094] (2) Parametric decoupling: by decomposing the dual quadratic surface The matrix can directly extract the position, orientation, and size parameters of the ellipsoid.

[0095] Figure 2 In the diagram, L represents four detection edges. Let X be the tangent plane, and let X be the point of intersection between the tangent plane and the ellipsoid, i.e., the point of tangency. For any tangent plane The tangent point is , there must be ,therefore ,because In plane Above, therefore there is We can obtain the following formula:

[0096] ;

[0097] Now we will prove the tangent plane. Must be Let the tangent plane be the point of tangency. Then there is Now all that is needed is to prove On the quadratic surface, since:

[0098] ;

[0099] therefore, On a quadric surface. By the theorem: the dual quadric surface of a nondegenerate quadric surface. It is still a quadratic surface, and ,get:

[0100] ;

[0101] Let the projection matrix of the camera be... For the projected ellipse of the quadratic surface onto the camera plane, it can be used Let K be the camera intrinsic parameter matrix and R be the camera rotation matrix. The matrix is ​​the dual quadratic curve, which represents the dual form of the quadratic curve projected onto the 2D plane.

[0102] use To represent an elliptic quadratic surface that has no translation or rotation at the origin:

[0103] ;

[0104] in, Transformation matrix representing the lengths of the three axes of an elliptic body ,in, Let be a rotation matrix. To obtain the object's projection translation vector, you can use:

[0105] Let be the vector representing the ellipsoid after coordinate transformation, expressed as a 9-DOF vector:

[0106] ;

[0107] in, The three parameters represent the lengths of the three semi-axes of the ellipsoid. The three parameters represent the center position of the ellipsoid. Three parameters represent its attitude.

[0108] OA-SLAM proposes using a standard sphere to represent objects during initialization, and then optimizing it into an ellipsoid in subsequent images, which improves the efficiency of object reconstruction. This embodiment also adopts the same method. First, after obtaining the object detection box, the object is tracked. If the object is also tracked in subsequent frames, a coarse estimate of the object is created.

[0109] The center of the sphere is obtained by triangulating the centers of the detection boxes across multiple frames. First, the three-dimensional center coordinates of the sphere are defined as follows: Pixel coordinates of the center of the detection box via camera projection matrix:

[0110] ;

[0111] in, For camera projection matrix, For the camera intrinsic parameter matrix, For the camera rotation matrix, Projection matrix The row vector of the first row. Projection matrix The row vector of the second row, Projection matrix The row vector of the third row, Let be the camera translation vector.

[0112] The relationship between the pixel coordinates of the sphere's center and the coordinates of its three-dimensional center can be obtained as follows:

[0113] ;

[0114] Will Substituting into the above equation and simplifying, we get:

[0115] ;

[0116] in, The x-axis pixel coordinates of the center of the object detection box The vertical pixel coordinates of the center of the object detection box.

[0117] The equation obtained from multi-frame observations can be used to derive:

[0118] ;

[0119] in, For the first The horizontal pixel coordinates of the center of the object detection box in the frame. For the first The vertical pixel coordinates of the center of the object detection bounding box in the frame. For the first The horizontal pixel coordinates of the center of the object detection box in the frame. For the first The horizontal pixel coordinates of the center of the object detection box in the frame. For the first Frame camera projection matrix The third row vector, For the first Frame camera projection matrix The first row vector, For the first Frame camera projection matrix The second row vector, For the first Frame camera projection matrix The third row vector, For the first Frame camera projection matrix The first row vector, For the first Frame camera projection matrix The second row vector.

[0120] Singular value decomposition can be used to find the least squares solution. After normalizing, the coordinates of the center of the sphere are obtained:

[0121] ;

[0122] in, To obtain the center coordinates of the sphere after normalization, For the non-normalized solution obtained using singular value decomposition Axial components, For the non-normalized solution obtained using singular value decomposition Axial components, For the non-normalized solution obtained using singular value decomposition Axial components, is the homogeneous scaling factor of the solution vector.

[0123] The radius of the sphere is determined by the average size obtained by back-projecting the detection boxes from multiple frames onto the world coordinate system.

[0124] ;

[0125] Where radius represents the average size, the first... The detection box width of the frame object is The height is , , These are camera-intra-camera parameters. For the first The depth coordinates of the object's center in the camera coordinate system. This represents the number of observation frames.

[0126] Furthermore, based on object semantic information and 3D ellipsoidal model matching and alignment of multi-source data, and combined with nonlinear optimization algorithms for global pose joint optimization, a high-precision map is obtained, including:

[0127] Based on object semantic information and 3D ellipsoid model matching and alignment of multi-view data, we obtain object and semantic map points;

[0128] Semantic map points are used to optimize the poses of multiple robots and obtain the pose relationships between them.

[0129] A graph optimization method is used to perform global pose optimization on the pose relationships between multiple robots to obtain global pose estimation data.

[0130] A high-precision map is obtained based on global pose estimation data.

[0131] Furthermore, based on object semantic information and 3D ellipsoid model matching and alignment of multi-view data, object and semantic map points are obtained, including:

[0132] Based on the semantic information of objects, a preset number of detection boxes are set, and the semantic information of objects is projected to obtain the corresponding number of projection boxes.

[0133] Based on a 3D ellipsoidal model, the intersection-union ratio (CIU) of the detection box and the projected box is calculated, and the CIU is used as the association score.

[0134] Based on the association score, the association relationship is obtained using the Hungarian method.

[0135] Optimize the relationships using multi-view data to obtain object and semantic map points.

[0136] Specifically, this embodiment relates to a data association and optimization method in a semantic SLAM system, aiming to solve the problems of map redundancy and optimization deviation caused by data association failure in existing technologies. In a semantic SLAM system, if the observation data and map elements are not accurately associated, each detected object will be treated as a new instance, leading to the accumulation of redundant objects in the map, the failure of backend optimization constraints, and ultimately affecting the accuracy of localization and mapping. To address this, this embodiment proposes a dual association mechanism based on detection boxes and semantic feature points, combined with multi-view geometric verification, to improve the robustness and accuracy of data association.

[0137] This embodiment first adopts an object association strategy based on detection boxes: when the camera motion is stable between adjacent frames, short-term association is achieved through two-dimensional bounding box tracking; to address the tracking failure problem caused by viewpoint changes, occlusion, or motion blur, a three-dimensional ellipsoid model is introduced, and the object is projected onto the image plane in combination with the camera pose. The intersection-union ratio (IoU) between the detection box and the projection box is calculated as the matching score, a score matrix is ​​constructed, and the optimal matching is achieved through the Hungarian algorithm.

[0138] Suppose there are N detection boxes in the k-th frame, and the current system maintains M objects. Based on the ellipsoidal projection model, calculate the intersection-union ratio (IUU) of the projected region and the detection boxes, which is used as the score:

[0139] ;

[0140] in, Let be the detection box for the i-th object. For the pj-th object's projection frame, For the first In-frame detection box With detection box The degree of matching.

[0141] Construct the score matrix for the k-th frame. :

[0142] ;

[0143] If the IoU between the projection box and the detection box is lower than the threshold, or the projection deviation of the ellipsoid center exceeds the limit, it is determined to be an invalid association, and the current frame will not participate in the update of the object.

[0144] To further improve the association accuracy, this embodiment proposes an object association method based on semantic feature points: map feature points located within the object detection box or inside the ellipsoid are bound to the corresponding objects, and the association relationship is optimized by combining multi-view observation data. To address the problem of erroneous association of outliers due to viewpoint errors, detection box offsets, or inaccurate depth estimation, a three-dimensional geometric consistency verification mechanism is adopted: if a feature point is located outside the ellipsoid or its distance from the geometric center of the object exceeds the limit, it is determined to be an outlier and removed; simultaneously, through multi-view observation redundancy analysis, abnormal points that have not been consistently observed across multiple frames are removed, ensuring the geometric consistency and purity of the semantic point cloud.

[0145] Compared to the accuracy degradation caused by insufficient ellipsoid modeling capabilities in traditional QuadricSLAM methods, this embodiment significantly improves the modeling accuracy and relocation reliability of complex objects by integrating detection box projection matching, semantic feature point binding, and multi-view verification mechanisms. Experiments show that the method significantly reduces exploration time compared to single-robot systems in multi-robot collaborative scenarios, and reduces path redundancy and node generation compared to traditional RRT algorithms, verifying its efficiency and robustness in dynamic environments.

[0146] This embodiment conducts a comparative experiment on semantic mapping and trajectory accuracy on the "fr2_desk" sequence of the TUM RGB-D dataset. Figure 3 As shown in (a)-(c), the semantic map point extraction performance of the traditional OA-SLAM and the method of this embodiment is compared: OA-SLAM adopts a semantic feature extraction method based on target detection boxes, which easily introduces background regions ( Figure 3 (b) in the example leads to inconsistent binding between semantic feature points and objects, affecting subsequent association and pose estimation; the method in this embodiment accurately segments the object contour through a semantic segmentation mask. Figure 3 In (c) of the model, the geometric consistency and expression accuracy of the extracted semantic feature points are significantly improved, and the interference of background noise is reduced.

[0147] In the trajectory reconstruction comparison experiment, as shown in Figures 4(a), 4(b) and 4(c), the trajectory generated by ORB-SLAM (pure geometric feature method) is smooth and highly similar to the real trajectory; after the method of this embodiment integrates semantic information, the trajectory continuity and stability are close to ORB-SLAM; while OA-SLAM relies on ellipsoid modeling, and the geometric expression error accumulates in the scene of insufficient observation or occlusion, resulting in obvious drift in the later stage of the trajectory.

[0148] Experimental results show that, after introducing semantic information, the trajectory accuracy of the method in this embodiment is close to that of the pure geometric method (ORB-SLAM) and significantly better than that of traditional semantic SLAM methods (such as OA-SLAM), verifying the effectiveness and robustness of the semantic-geometric fusion strategy.

[0149] Furthermore, semantic map points are used for multi-robot pose optimization to obtain the pose relationships between multiple robots, including:

[0150] Based on semantic map points, obtain a set of keyframes containing landmarks;

[0151] The bag-of-words model is used to process the keyframe set to obtain a similarity score;

[0152] Based on similarity scores, the pose relationships between multiple robots are obtained.

[0153] Specifically, such as Figure 5 As shown, this embodiment employs a centralized architecture to implement a multi-robot collaborative SLAM system. In this architecture, each robot performs local semantic SLAM and uploads keyframes, semantic map points, and relevant object features to a central server. The central server performs global optimization, generating a consistent global map through graph optimization and data fusion.

[0154] This embodiment relates to the field of multi-robot visual SLAM technology, and in particular to a method for relative pose estimation and map fusion between robots.

[0155] In multi-robot vision SLAM systems, effectively calculating the relative poses between robots to achieve coordinate system transformation and fusion is a key challenge. Existing methods based on robot physical encounter detection rely on robots "meeting" to trigger fusion, but their applicability and effectiveness are limited by the sparseness of encounter events in large-scale scenes.

[0156] Therefore, this embodiment provides an improved method. This method recognizes that when robots independently operate SLAM, their trajectories often pass through some of the same areas due to exploring the same scene and detecting potential loop closures, resulting in spatially overlapping observations. By identifying these overlapping areas in their respective maps and matching and associating their feature point information, the coordinate transformation relationship between robots can be recovered, thereby completing local map stitching and global map stitching. Figure 1 Consistent construction.

[0157] Specifically, such as Figure 6 As shown, robots A and B observe the same landmark P at different times. Using this common landmark, the pose transformation matrix of robot B relative to robot A can be calculated. However, this type of matching based on visual landmarks carries the risk of mismatches.

[0158] To mitigate this risk, this embodiment introduces the following: Figure 7The determination process is shown below. In this process, robots A and B first independently run their own SLAM systems, generating keyframe sets containing landmarks. Then, using a mechanism similar to loop closure detection, the landmarks in the two keyframe sets are traversed and initially matched. Feature descriptors are used for filtering, thus eliminating most obvious mismatches. A bag-of-words model is typically used to find similar locations.

[0159] Furthermore, constructing the dictionary vector includes:

[0160] Based on the keyframe set, local feature descriptors are obtained;

[0161] The K-Means clustering algorithm is used to cluster local feature descriptors to obtain visual words;

[0162] Based on visual words, obtain a cross-tree dictionary;

[0163] Obtain bag-of-words vectors based on the frequency of visual words in the cross-tree dictionary;

[0164] Obtain the bag-of-words model based on bag-of-words vectors.

[0165] Specifically, the dictionary is the core of the bag-of-words model, consisting of a set of visual words (i.e., cluster centers of feature descriptors). Essentially, it abstracts and generalizes image features, dividing the high-dimensional feature space into discrete categories, simplifying image representation and comparison. The dictionary size (number of visual words) is adjustable: a larger dictionary indicates more refined representation but is more computationally complex; a smaller dictionary indicates coarser representation but is more efficient.

[0166] By mapping image features to visual words, SLAM systems can quickly calculate image similarity, which can be used for keyframe selection, loop closure detection, scene recognition, and more. Construction steps:

[0167] 1. Feature Extraction: Extracting local feature descriptors from a large number of images. These feature descriptors are usually high-dimensional vectors that can capture key information in the image.

[0168] 2. Clustering analysis: The K-Means clustering algorithm is used to cluster all feature descriptors to obtain a set of cluster centers, i.e. visual words, which serve as the first layer of K-Means.

[0169] 3. Dictionary generation: Repeat step 2 for each type of descriptor until the required number of levels is met, and store the results in a K-ary tree, where the leaf nodes of the K-ary tree are each word.

[0170] 4. Bag-of-Words Vector Generation: For each image, its feature descriptors are mapped to visual words in the dictionary. The frequency of each visual word is counted to generate a bag-of-words vector. A bag-of-words vector is a fixed-length vector with dimensions equal to the size of the dictionary, capable of concisely representing the image's features.

[0171] By constructing a dictionary, SLAM systems can process image data more efficiently, improving the system's real-time performance and robustness. Although dictionary construction is computationally intensive, it can usually be done offline. The online phase only requires feature mapping and similarity calculation, greatly reducing computational complexity.

[0172] After constructing the bag-of-words structure, each image can be represented as a fixed-length word vector, i.e., a semantic feature histogram. To measure the similarity between two images, Euclidean distance is often used to compare their bag-of-words vectors.

[0173] For images The total number of feature points of the word is ,word The number of feature points in the current image is Then its frequency (Term Frequency, TF) in the image is:

[0174] ;

[0175] in, The word frequency of word w in this image;

[0176] However, directly using word frequency has certain biases. Some common semantic words (such as "ground" and "wall") appear frequently in most images, and their ability to distinguish images is limited. Therefore, inverse document frequency (IDF) is introduced to weight word frequencies.

[0177] For a certain image , No. The weights of each word are calculated as follows:

[0178] ;

[0179] in, This represents the total number of features contained in the dictionary. For words Number of features included. Weighted ;

[0180] in, For words TF-IDF weights, For words word frequency, For words Inverse document frequency.

[0181] Represent each image as a vector containing all the dictionary words. By comparing the vectors of two images, their similarity score can be obtained, where... To indicate the first in the dictionary TF-IDF weights of visual words.

[0182] ;

[0183] in, The similarity score between the two images is calculated. The first image contains a vector of all the dictionary words. The second image contains vectors of all the dictionary words.

[0184] The score determines whether there are similar positions. If the score is greater than the threshold, there are similar positions. Then, the two frames are matched and their relative poses are transformed.

[0185] Furthermore, a graph optimization method is used to perform global pose optimization on the pose relationships between multiple robots, and the global pose estimation data obtained includes:

[0186] To further improve the consistency of the overall map and the accuracy of global pose estimation, this embodiment employs a graph optimization method in the backend of multi-robot SLAM, similar to the graph optimization framework commonly used in single-robot systems. Each node in the graph represents the pose of a keyframe, and edges represent observation constraints between adjacent keyframes. After detecting a common region, an edge (i.e., a cross-robot constraint) is added between keyframes that are successfully matched between robot A and robot B, indicating a co-view relationship where the two robots observe the same scene at that location. Figure 8 As shown, if robot A and robot B successfully establish a pose matching relationship in a public area, the corresponding transformation matrix is ​​denoted as... . Indicates that robot A is in In the presence of robot B The transition matrix at point A forms a new constraint edge in the graph.

[0187] In graph optimization, an error term is used to measure the deviation between the actual observations and the current estimate. The overall optimization objective function can be expressed as:

[0188] ;

[0189] in, The internal edges of robot A and robot B The pose error, It is a cross-robot constraint term, namely the pose alignment error between the two robots. It is the L2 norm of the error. Let be the global optimization objective function for the multi-robot pose estimation problem. Its value represents the total error between the current pose estimation and all observation data. n and m are index variables used to identify the observation constraints between different robots in the common area. The first and second terms are the L2 errors of the internal graphs of robot A and robot B, respectively, and the third term is the L2 error of the observations between robots in the common area. Iterative optimization using the Gauss-Newton method can reduce the errors between robots. Because introducing global errors between robots provides more optimizable information for the global map, theoretically, it can lead to more accurate global pose estimation results.

[0190] To verify the effectiveness of the proposed multi-robot semantic SLAM system in real-world scenarios, this embodiment conducted experimental evaluations on the publicly available TUM RGB-D dataset. The dataset was split into two unrelated parts, and the performance of the system in fusion of different robot trajectories and global pose optimization was examined in detail.

[0191] Figures 9(a), 9(b), and 9(c) illustrate the fusion results of the two robot trajectories on a self-built laboratory dataset. Figures 9(a) and 9(b) represent the original trajectories of robot 1 and robot 2, respectively. Due to their different starting positions, the two robots construct maps in their respective local coordinate systems, resulting in significant offsets. Figure 9(c) shows the alignment results of the two trajectories in the global coordinate system after similarity position detection and semantic ICP fine registration. The red lines represent the established cross-robot constraint connections, the blue dots represent the starting positions of robot 1, and the orange squares represent the starting positions of robot 2. The fused global trajectories achieve precise spatial alignment, verifying the adaptability and consistency of this embodiment in a real-world environment.

[0192] Furthermore, based on high-precision maps, dynamic exploration through joint multi-robot exploration tasks includes:

[0193] A global Venn diagram is generated based on the robot's real-time position in a high-precision map;

[0194] Based on the global Venn diagram, when the robot arrives at a new location and detects no existing nodes within a preset range, a topology node mechanism is introduced. The topology nodes in the topology node mechanism are used as critical path hubs connecting known and unknown regions.

[0195] When exploring task allocation, the distance between each robot and the topological nodes is calculated based on the global Venn diagram, and the best task assignment based on the distance is realized.

[0196] Specifically, in multi-robot autonomous exploration tasks, a reasonable task allocation strategy is key to improving exploration efficiency. Traditional methods may lead to uneven distribution of task areas, excessive path intersections between robots, and even situations where some areas are not fully explored. To address this, this embodiment proposes a hierarchical task allocation method that combines global Voronoi diagram segmentation with local topological node guidance to achieve reasonable task division and dynamic adjustment, thereby improving overall exploration efficiency.

[0197] Step 1: The core of the system first uses the real-time positions of all robots as seed points to dynamically generate a global Vino map. This map automatically divides the entire unknown environment into multiple Vino cells, each containing all spatial points closest to its corresponding robot.

[0198] During initial task allocation, each robot is preferentially assigned to its respective Vino cell to perform exploration. This allocation based on spatial proximity effectively ensures the initial uniform distribution of robots in the environment, avoiding resource waste and concentrated operations.

[0199] To adapt to the dynamic nature of exploration (robot movement, environmental updates), the system periodically updates the Venn diagram. Crucially, once a robot completes its task within its current cell, the system automatically removes its geographical restrictions, enabling it to collaborate across cells to explore the remaining area, thereby dynamically optimizing overall resource utilization and exploration efficiency.

[0200] Step 2: However, in complex obstacle environments, the Vinio map may exhibit unreasonable partitioning (such as path intersections and blind spots). To address this key challenge, this embodiment innovatively introduces a topology node guidance mechanism on the Vinio partitioning framework.

[0201] In Figures 11(a) and 11(b), the robot autonomously generates key topology nodes during its movement: when it arrives at a new location and detects that there are no existing nodes within a preset range, it creates a new node at that location and connects to the topology network. These nodes become critical path hubs connecting known and unknown areas.

[0202] The system refines the large area to be explored into leading-edge sub-regions that match the robot's camera field of view (FOV) (a larger FOV results in a larger sub-region, and a smaller FOV results in a finer division). Each topology node connects multiple such leading-edge sub-regions, forming a dynamic task network.

[0203] The topology nodes employ a dual-state management system: "Not fully explored" (with connected sub-regions to be explored) and "Completed" (all connected sub-regions have been explored). During task allocation, the system directly uses the Veno diagram framework to calculate the distance between each robot and the node, prioritizing the allocation of "Not fully explored" nodes to the nearest idle robot, thus achieving optimal task assignment based on location.

[0204] Within their assigned Vino units, the robots utilize Dijkstra's algorithm to calculate the shortest paths to the leading edge sub-regions connected to them based on the topological node network, forming efficient local exploration routes that effectively avoid path conflicts and blind spots.

[0205] Step 3: The powerful collaborative capability of this method stems from the real-time linkage monitoring and response to the state of the Vino unit and the state of the topology nodes.

[0206] The system continuously tracks the status of topology nodes: once all leading sub-regions connected to a node have been explored, it is marked as "completed". More importantly, if all topology nodes within a Vino cell are detected to be in the "completed" state, the task of that cell is determined to be completed, its robot is immediately released, and it is allowed to actively request assistance from neighboring uncompleted cells.

[0207] Especially in the boundary areas, when multiple robots are within the collaborative range, the system comprehensively considers the real-time distance between each robot and the "not fully explored" node at the boundary, the current task load and completion progress, and dynamically fine-tunes the Vino partition boundary to achieve seamless task takeover and cross-regional collaboration, greatly improving collaborative efficiency.

[0208] To ensure the real-time performance and simplicity of the topology network, new nodes are continuously generated and connected as the robot explores, while strict minimum spacing constraints are used to avoid the generation of redundant nodes, and historical nodes are recorded in different states.

[0209] exist Figure 12 In the game, two robots divide the entire map into different regions. Each region contains two topological nodes, and the topological nodes are connected to the nearest unknown regions according to the Dijkstra tree. The nodes are assigned to the two robots according to the distance, and the robots explore their respective local regions.

[0210] To verify the effectiveness and superiority of the proposed multi-robot cooperative exploration algorithm based on Voronoi graph partitioning and dynamic task allocation of topological nodes, this embodiment designed and built a complete simulation verification platform. It integrates ROS (Robot Operating System) with the Gazebo simulator and uses the RViz visualization tool to achieve map construction and robot status monitoring. Figure 10 As shown, the exploration scene is one. The indoor map, containing multiple rooms, obstacle walls, and open areas, effectively simulates typical exploration challenges faced by mobile robots in real-world environments. The experimental robot, Turtlebot3Burger, is equipped with a Kinect v2 depth camera to acquire environmental information, such as... Figure 13 As shown.

[0211] In the single-robot exploration experiment, the robot starts from an initial position, scans the environment by rotating in place and collects point cloud information, extracts boundary detection front points, and then extracts the front block centered on the current position, constructing initial topology nodes and establishing connections. The robot plans to move to the target block according to the shortest path, dynamically generating new nodes and updating connections, maintaining the minimum spacing constraint between nodes to avoid redundant generation, and finally completing full area coverage to form a complete topology network. Experimental results show that this embodiment generates 8 topology nodes in 94 seconds, reducing path redundancy by more than 30% compared to the traditional RRT algorithm (128 seconds, 39 nodes), and improving node utilization by 79.5%, verifying its significant advantages in path continuity, coverage integrity, and computational efficiency.

[0212] In a single-robot scenario, this method generates 8 topology nodes in a total time of 94 seconds, achieving full environmental coverage. Compared with the traditional RRT algorithm (which takes 128 seconds and generates 39 nodes), it reduces redundant paths by more than 30%, improves node utilization by 79.5%, and shortens path length by 23%, verifying its significant advantages in path coherence, coverage integrity, and computational efficiency.

[0213] For multi-robot collaborative scenarios, this embodiment deploys three TurtleBot3 robots in a 20m×20m simulation environment, using ROS for communication and task coordination. The central module generates a Voronoi diagram based on global front information, dynamically divides the task area, uniquely assigns front blocks to each robot, and updates the partitions in real time to minimize interference while maintaining node spacing constraints. Experiments show that the multi-robot collaborative system takes 39 seconds, a 58.5% improvement in efficiency compared to a single-robot system. It fully covers the entire map with a consistent topology, avoiding duplicate coverage and collision risks, providing an efficient and robust solution for collaborative operations in large-scale dynamic environments.

[0214] To further verify the superiority of the technical solution in this embodiment, a comparative experiment was conducted between the proposed Voronoi diagram partitioning and dynamic topology scheduling method and the two-layer RRT algorithm under the same map conditions and robot configuration. Experimental results show that the traditional RRT algorithm suffers from low path planning efficiency due to its random sampling mechanism, especially prone to generating redundant paths in areas with dense obstacles, and the overlapping of multiple robot paths leads to task conflicts and execution delays. Statistics from 10 repeated experiments are shown in Table 1. The average time for the RRT algorithm to complete map exploration is 57 seconds, a 46.2% increase compared to this embodiment (39 seconds); simultaneously, the number of topology nodes generated by the RRT algorithm increases by more than 35%, and the average path length increases by 28.3%. This embodiment, through dynamic Voronoi diagram partitioning and topology node guidance mechanism, achieves balanced task area allocation and path separation optimization, with a stable map coverage of 100% and a 40% reduction in path redundancy, verifying its efficiency and robustness in complex scenarios.

[0215] Table 1: Statistical Table of 10 Repeated Trials

[0216] method Total time (s) Number of topology nodes Average path length (m) Map coverage (%) Method of this embodiment 39±2.3 98.6 RRT method 95.3

[0217] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-robot collaborative semantic SLAM and dynamic exploration method, characterized in that, include: Acquire environmental data; The environmental data is input into the improved YOLOv8 model to obtain object semantic information, wherein the improved YOLOv8 model is obtained by improving the YOLOv8 model; Based on the semantic information of the object, a three-dimensional ellipsoid model is constructed; Based on the semantic information of the object and the three-dimensional ellipsoid model, multi-view data is matched and aligned, and global pose joint optimization is performed by combining nonlinear optimization algorithm to obtain a high-precision map. Based on the semantic information of the objects and the matching and alignment of multi-source data using a 3D ellipsoid model, and combined with a nonlinear optimization algorithm for global pose joint optimization, a high-precision map is obtained, including: Based on the object semantic information and the 3D ellipsoid model, match and align multi-view data to obtain object map points and semantic map points; The semantic map points are used to optimize the pose of multiple robots and obtain the pose relationships between multiple robots. A graph optimization method is used to perform global pose optimization on the pose relationships between the multiple robots to obtain global pose estimation data. Based on the global pose estimation data and object map points, the high-precision map is obtained; Based on the object semantic information and the 3D ellipsoid model, matching and aligning multi-view data, obtaining object map points and semantic map points includes: Based on the semantic information of the object, a preset number of detection boxes are set, and the semantic information of the object is projected to obtain the corresponding number of projection boxes. Based on the three-dimensional ellipsoid model, the intersection-union ratio (IUU) of the detection box and the projection box is calculated, and the IUU is used as the association score. Based on the association score, the association relationship is obtained using the Hungarian method. The multi-view data is used to optimize the association relationship, and the object map points and semantic map points are obtained; Utilizing the semantic map points to optimize the poses of multiple robots and obtain the pose relationships between them includes: Based on the semantic map points, obtain a set of keyframes containing landmarks; The keyframe set is processed using the bag-of-words model to obtain a similarity score; Based on the similarity score, the pose relationships among the multiple robots are obtained; based on the high-precision map, dynamic exploration is carried out in conjunction with the multi-robot exploration task.

2. The multi-robot cooperative semantic SLAM and dynamic exploration method according to claim 1, characterized in that, The improvements to the YOLOv8 model include: replacing the C3 module with the cross-stage feature compression C2f module, introducing the bidirectional feature pyramid PANet to enhance the cross-scale feature fusion module, and adopting a decoupled design for the head. The cross-stage feature compression C2f module is used to reduce computational redundancy through parallel convolution and pooling; The bidirectional feature pyramid enhanced cross-scale feature fusion module is used to improve the detection accuracy of small targets; The header is used to optimize bounding box localization and category prediction.

3. The multi-robot cooperative semantic SLAM and dynamic exploration method according to claim 1, characterized in that, Based on the semantic information of the object, constructing a three-dimensional ellipsoid model includes: Based on the semantic information of the object, a dual quadratic surface model is constructed; The center and radius of the sphere in the dual quadric surface model are initialized to obtain the three-dimensional ellipsoid model.

4. The multi-robot cooperative semantic SLAM and dynamic exploration method according to claim 1, characterized in that, Building the bag-of-words model includes: Based on the set of keyframes, obtain local feature descriptors; The local feature descriptors are clustered using the K-Means clustering algorithm to obtain visual words; Based on the visual words, obtain a cross-tree dictionary; Based on the frequency of occurrence of the visual words in the cross-tree dictionary, obtain the bag-of-words vector; Based on the bag-of-words vectors, the bag-of-words model is obtained.

5. The multi-robot cooperative semantic SLAM and dynamic exploration method according to claim 1, characterized in that, The graph optimization method is used to perform global pose optimization on the pose relationships between the multiple robots. Obtaining global pose estimation data also includes: Global pose optimization is performed on the pose relationships among the multiple robots by combining an objective function, wherein the objective function is: ； in, The internal edges of robot A and robot B The pose error, It is a cross-robot constraint term, namely the pose alignment error between the two robots. It is the L2 norm of the error. The objective function for the overall optimization of the multi-robot pose estimation problem is denoted by , where represents the total error between the current pose estimation and all observation data. n and m are index variables used to identify the observation constraints between different robots in the common area.

6. The multi-robot cooperative semantic SLAM and dynamic exploration method according to claim 1, characterized in that, Based on the high-precision map, dynamic exploration using a multi-robot joint exploration task includes: A global Venn diagram is generated based on the robot's real-time position in the high-precision map. Based on the global Venn diagram, when the robot arrives at a new location and detects no existing nodes within a preset range, a topology node mechanism is introduced. The topology nodes in the topology node mechanism are used to connect critical path hubs between known and unknown regions. When exploring task allocation, the distance between each robot and the topological node is calculated based on the global Venograph, and the best task assignment based on the location is realized according to the distance.

Images