Geometry-semantic collaborative fusion mobile robot three-dimensional semantic map construction method

Abstract

The invention relates to the technical field of synchronous positioning and map construction, in particular to a geometric-semantic collaborative fusion mobile robot three-dimensional semantic map construction method, which is used for performing deep fusion on a geometric map and multi-view semantic information so as to solve the problem of difference of the two types of information in resolution, scale and dimension. The geometric map mainly provides structured information of the environment, and the semantic information describes object categories and features thereof in the environment. Direct fusion of the two types of information often causes spatial positioning errors and semantic information loss or redundancy problems. According to the method, a geometric map containing environmental structured information is constructed through an SLAM technology, and meanwhile, object classification, attribute labeling and semantic relationship analysis are performed on multi-view image data by utilizing the semantic information extraction capability of a large model. And finally, precise fusion of the geometric map and multi-view semantic information is realized, and meanwhile, the object recognition and semantic understanding capabilities of the robot in a complex scene are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of simultaneous localization and mapping, and particularly relates to a method for constructing a three-dimensional semantic map of a mobile robot with geometric-semantic collaborative fusion. Background Art

[0002] Simultaneous localization and mapping (SLAM) is an important technology for realizing autonomous localization and navigation of robots in unknown environments at present. It locates itself according to the position of the robot during movement and the map, and at the same time builds an incremental map based on its own localization, and is widely used in many application scenarios such as 3D modeling and driverless driving.

[0003] In the field of mobile robots, traditional SLAM mainly obtains three-dimensional point cloud data of the environment through devices such as lidar, stereo vision or depth sensors to construct a geometric map that can accurately reflect the spatial structure of the scene. Although it can assist the robot in path planning to a certain extent, the single geometric information makes it difficult for the robot to achieve a deep understanding of the environment and intelligent decision-making. There are obvious limitations especially in complex and unstructured scenarios. For example, in scenarios such as homes and offices, there are many items and their placement is irregular. The robot can only rely on the geometric map to identify the object category and understand the relationship between objects, thus restricting the further improvement of the robot's intelligence level. Thanks to the rapid development of computer vision technologies, especially technical means such as object detection and semantic segmentation, significant progress has been made in extracting object category information, regional features and semantic relationships from optical images. The combination of these image semantic information and geometric maps is undoubtedly more conducive to the mobile robot to correctly perceive, understand the environment and make actions. However, there are significant differences in resolution, scale and dimension between the discrete point cloud map obtained by devices such as lidar and the continuous images obtained by optical cameras, and geometric information and semantic information belong to heterogeneous and multimodal data. Directly fusing these two types of data not only leads to inconsistent semantic information and inaccurate spatial positions, but also causes problems of redundancy or missing of semantic labels. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for constructing a three-dimensional semantic map of a mobile robot with geometric-semantic collaborative fusion, aiming to change the phenomenon that traditional mobile robot map construction only relies on geometric information or has limitations in semantic understanding, and improve the object recognition and semantic understanding ability of the robot in complex scenarios.

[0005] To achieve the above purpose, the present invention provides a method for constructing a three-dimensional semantic map of a mobile robot with geometric-semantic collaborative fusion, including the following steps:

[0006] Step 1: Point cloud data and multi-view image acquisition;

[0007] Step 2: Geometric map and multi-view image preprocessing;

[0008] Step 3: Fusion of raster map and multi-view semantic information.

[0009] Optionally, the execution process of Step 1 includes the following steps:

[0010] Step 1.1: LiDAR data acquisition, with odometer-assisted positioning at the front end and optimization based on loop detection and factor graph of GTSAM at the back end;

[0011] Step 1.2: RGB-D camera multi-view image acquisition.

[0012] Optionally, the execution process of Step 2 includes the following steps:

[0013] Step 2.1: LiDAR data preprocessing;

[0014] Step 2.2: RGB-D multi-view image preprocessing;

[0015] Step 2.3: Object-oriented semantic relationship reasoning and scene construction.

[0016] Optionally, in the process of LiDAR data preprocessing, first remove the irrelevant points in the original point cloud scene, then downsample the point cloud, and after statistical filtering, convert the indoor scene point cloud to a raster map by using the method of coordinate system mapping.

[0017] Optionally, in the process of RGB-D multi-view image preprocessing, use an unsupervised 2D segmentation model to perform unsupervised segmentation on the objects in the image to generate a set of masks of candidate objects; and use a visual feature extractor to process the region of each object.

[0018] Optionally, in the process of object-oriented semantic relationship reasoning and scene construction, it is necessary to construct an object set O from the object data extracted from the image T , estimate the spatial relationship between objects to generate an edge set E T , and complete the construction of the scene graph.

[0019] Optionally, the execution process of Step 3 includes the following steps:

[0020] Step 3.1: Geometric consistency map fusion strategy;

[0021] Step 3.2: Semantic-geometric information fusion strategy;

[0022] Step 3.3: Data association and update.

[0023] Optionally, in the process of executing the geometric consistency map fusion strategy, the point cloud data of the lidar and the RGB-D camera are fused through spatial coordinate transformation.

[0024] Optionally, during the execution of the semantic-geometry information fusion strategy, an octree map is used to store the fused geometry map and semantic information.

[0025] Optionally, data association and update specifically refer to associating multi-view semantic information with a 2D grid map. During this process, given only the semantic information of the target, the robot can complete the navigation task.

[0026] The present invention provides a method for constructing a three-dimensional semantic map of a mobile robot with geometric-semantic collaborative fusion, which deeply fuses a geometric map with multi-view semantic information, thereby solving the problems of differences in resolution, scale, and dimension between the two types of information. The geometric map mainly provides the structural information of the environment (such as the position, shape, and spatial relationship of objects), while the semantic information describes the object categories and their characteristics in the environment. Directly fusing these two types of information often leads to problems such as spatial positioning errors, loss of semantic information, or redundancy; the present invention constructs a geometric map containing the structural information of the environment through SLAM technology, and at the same time utilizes the semantic information extraction ability of the large model to perform object classification, attribute annotation, and semantic relationship parsing on multi-view image data. Finally, the accurate fusion of the geometric map and multi-view semantic information is achieved, and at the same time, the object recognition and semantic understanding ability of the robot in complex scenarios is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0028] Figure 1 is a schematic flow chart of a method for constructing a three-dimensional semantic map of a mobile robot with geometric-semantic collaborative fusion of the present invention.

[0029] Figure 2 is a schematic flow chart of the basic process of constructing a combined map of the present invention.

[0030] Figure 3 is a schematic process diagram of preprocessing static point clouds in the present invention.

[0031] Figure 4 is a schematic comparison diagram of the conversion between a three-dimensional static point cloud map and a two-dimensional grid map in the present invention.

[0032] Figure 5 is a schematic flow chart of preprocessing RGB-D multi-view images in the present invention.

[0033] Figure 6It is a schematic diagram of the object - oriented semantic relationship reasoning and scene construction process in the present invention.

[0034] Figure 7 It is a schematic diagram of the effects before and after the fusion of the grid map and the RGB - D point cloud in the present invention.

[0035] Figure 8 It is a schematic diagram of the relationship between the semantic categories and the semantic map in the octree map of the present invention.

[0036] Figure 9 It is a schematic diagram of the experimental results of the indoor scene point cloud geometric grid map construction in a specific embodiment of the present invention.

[0037] Figure 10 It is a schematic diagram of the effect of generating a semantic map through a camera model after extracting semantic information from multiple views in a specific embodiment of the present invention.

[0038] Figure 11 It is a schematic diagram of the effect of fusing a two - dimensional geometric grid map and a three - dimensional semantic map in a specific embodiment of the present invention. Detailed implementation manners

[0039] The following details the embodiments of the present invention. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as limiting the present invention.

[0040] The present invention provides a method for constructing a three - dimensional semantic map of a mobile robot with geometric - semantic collaborative fusion, including the following steps:

[0041] Step 1: Point cloud data and multi - view image acquisition;

[0042] Step 2: Geometric map and multi - view image pre - processing;

[0043] Step 3: Grid map and multi - view semantic information fusion.

[0044] As Figure 1 shown is the flow schematic diagram of the method of the present invention. The following further explains in combination with the specific execution process:

[0045] Step 1: Point cloud data and multi - view image acquisition.

[0046] Step 1.1: LiDAR data acquisition. For the problem of weak GPS signals in indoor scenarios, auxiliary positioning means such as odometry are usually adopted to obtain observed environmental data using sensors during movement and complete the pose estimation of mobile robots. To improve the current situation where the front-end point cloud data is not fully processed, the point cloud registration accuracy is not high, the back-end loop detection and mapping effects are not good, and to solve problems such as trajectory drift and unprocessed dynamic obstacles, Figure 2 shows the basic process of collective map construction. The SLAM technology tightly coupled with 3D lidar and IMU is adopted to reduce the z-axis drift.

[0047] Front-end odometry part: Different map update strategies are selected by judging the number of key frames. When the number of key frames is less than 20, the point-to-plane feature map update method is adopted to update the local map by matching the plane features of the current frame and the previous frame; when the number of key frames exceeds 20, the Generalized ICP (GICP) algorithm is used for point cloud matching from plane to plane to improve the accuracy and handle more complex environments. This dynamic selection strategy based on the number of key frames provides accurate pose estimation and robust map update capabilities while ensuring computational efficiency.

[0048] Back-end map optimization part: To improve the accuracy and consistency of the global map. Loop detection is used to identify when the robot returns to a known area, and the map accuracy is improved by eliminating cumulative errors. The factor graph optimization method based on GTSAM (Georgia Tech Smoothing and Mapping) combines prior information and observation data to optimize the robot's trajectory and the environmental map, reducing the pose estimation error.

[0049] Step 1.2: RGB-D camera multi-view image acquisition. The multi-view image data collected by the RGB-D camera includes color images, depth maps, and confidence maps, which can achieve a comprehensive record and high-quality reconstruction of the scene. The processing includes decompression and parsing of the depth map, confidence map, and color image, adjusting the resolution to match the target resolution (1440×1920 pixels), while ensuring the integrity and consistency of the data. The camera internal parameters include focal length, principal point coordinates, and image size, etc., to ensure the alignment and accuracy of the color image and the depth map. In addition, the system combines the pose information of the multi-view data and converts it into a camera pose matrix for three-dimensional scene construction, providing accurate spatial position information. Through the joint processing of the depth map and the confidence map, a high-confidence depth map is also generated to better filter unreliable data points, thereby improving the overall quality and robustness of the model.

[0050] Step 2: Geometric map and multi-view image preprocessing.

[0051] Step 2.1: LiDAR data preprocessing. As shown in Figure 3 Figure 2 shows the preprocessing process of the original point cloud. Among them, (a) is the original point cloud scene (34,208,094 points), and the red ellipse frames in the figure mark the miscellaneous points and isolated points in the point cloud. These isolated points may come from sensor noise, dynamic objects or other errors. Removing these irrelevant points helps to reduce the impact on subsequent processing and modeling. (b) shows the scene after downsampling the point cloud (520,283 points), and the sampling ratio is 0.05. By downsampling the point cloud, the point cloud density is significantly reduced, thereby reducing the computational amount and improving the processing efficiency, while retaining the main structural information of the environment. (c) shows the scene after the point cloud is statistically filtered (489,855 points), and the filtering parameters are set as the number of neighboring points 100 and the standard deviation multiple 1. The noise points and outliers in the local area are removed, making the point cloud smoother and more accurate, and improving the accuracy of subsequent processing. The statistical filter can adaptively identify and remove outliers within the effective range, making the point cloud data more stable and reliable. (d) shows the point cloud after removing the ceiling and floor points. Only the determined obstacles in the map are retained, and the irrelevant points are removed, focusing on the point cloud of the ground or the target object, avoiding the interference of the ground and the ceiling on subsequent analysis and modeling, providing a prior map for robot path planning and semantic navigation, ensuring the quality and accuracy of the final point cloud, and reducing the unnecessary computational burden.

[0052] Through the preprocessing of the static point cloud, a point cloud of the indoor scene with only static features retained is obtained. It is necessary to convert it into a two-dimensional grid map for robot path planning and semantic navigation. The present invention adopts a coordinate system mapping method to project each point in the three-dimensional point cloud onto a two-dimensional plane and generate a corresponding grid map according to a predetermined resolution. Through this mapping method, efficient spatial representation and path planning can be achieved, providing clear environmental information for the robot.

[0053] Specifically, the point cloud data is usually represented by three-dimensional coordinates (x, y, z). When converting to a grid map, it is first necessary to project the point cloud data onto a two-dimensional plane. In most cases, assuming that the robot is on the ground, the part of the point cloud that does not belong to the ground can be ignored, and most of the remaining data is on the ground. To simplify the calculation process, each point in the point cloud only retains its two-dimensional coordinates x' and y' during conversion, ignoring the elevation information of the z-axis:

[0054]

[0055] After projecting the point cloud data onto a two-dimensional plane, the next step is to map it onto a grid map. Each cell of the grid map has a fixed size of d×d, which represents a spatial region in the environment. To map each point in the point cloud to a specific cell in the grid map, the projected coordinates (x′, y′) need to be converted into grid coordinates, (m, n), where m and n represent the row and column positions in the map. The conversion formula is as follows:

[0056]

[0057] where represents the floor operation, ensuring that each point in the point cloud is accurately mapped to the corresponding cell in the grid map.

[0058] Figure 4 shows the effect of converting a geometric map into a grid map. Each grid cell represents a small area in the map. When generating the grid map, the value of each grid cell is used to indicate whether the area is occupied. Assuming that all the point clouds mapped to the same grid cell represent obstacles, these grid cells are marked as occupied (usually represented by the value 1). Grid cells that are not mapped to any points represent free space and are marked as free (usually represented by the value 0).

[0059] Step 2.2: RGB-D multi-view image preprocessing. When processing the extraction of semantic information in images, especially when extracting the semantic information of objects from RGB-D images, it is usually necessary to identify the objects in the images through object detection and segmentation models, and assign labels, masks, and confidence scores to each object. According to Figure 5 shown, the system first receives a series of RGB images of scenes, and in order to further process, it must integrate the image data from multiple time steps and the relevant position information.

[0060] For each frame of input data I t , it contains the RGB image the depth image and the camera pose information θ t , and the data needs to be processed frame by frame. Each frame of data will update the existing object set O t-1 in an incremental manner, and instantiate new objects or update existing objects according to the current detection results. Specifically, for each frame of data can be expressed in the following form.

[0061] I = {I 1 , I 2 , …, I t}

[0062] The object set O tContains all the objects detected at the current time t, and its representation is:

[0063]

[0064] where o j represents the j-th object detected at t.

[0065] As the time step progresses, the objects in the scene may change. Therefore, the object set O t needs to be incrementally updated as each frame of the input image is updated. Incremental update means that if the objects in the current image match the previous objects, the existing objects are updated; otherwise, new objects are instantiated.

[0066] To extract the semantic information in the image, the present invention adopts a two-dimensional segmentation model without class restrictions, the Segment Anything Model (SAM), which can perform class-free segmentation of the objects in the image, thereby generating a set of masks for the candidate objects. For a given RGB image SAM will generate a set of object masks m t,i , representing the region of the i-th object in the image. The set of these masks can be expressed as:

[0067]

[0068] where Seg(·) represents the set of masks extracted from the RGB image by the SAM model, and M is the number of objects detected in the image.

[0069] To extract the semantic features from each object mask, a visual feature extractor (such as CLIP or DINO) is used to process the region of each object. The visual features of the object region can be extracted by the following formula:

[0070]

[0071] where f t,i is the semantic feature vector extracted from the corresponding region of the mask m in the RGB image t,i by the feature extractor. In this way, each object is not only labeled with class information, but also a corresponding semantic feature vector is generated for each object, which is used for further analysis and understanding.

[0072] In addition, to further describe the semantic relationships between the objects in the image, an edge set E t can be constructed, representing the relationship information between all the objects at the current time t:

[0073]

[0074] Among them, e k represents the k-th edge relationship feature, which describes the semantic association between two objects in the image. K is the number of edges detected in the image at the current moment. In this way, E t not only contains the semantic relationships between objects, but also provides a basis for global semantic reasoning and construction of the scene.

[0075] Step 2.3: Object-oriented semantic relationship reasoning and scene construction. Figure 6 In, after object recognition and semantic information extraction in the image, the next step is to perform object association and fusion, and finally construct a 3D scene semantic map for spatial relationship reasoning. An object set O is constructed from the object data extracted from the image T , and the spatial relationships between objects are estimated to generate an edge set E T . Both of these are needed to complete the construction of the entire scene graph.

[0076] For each newly detected object <p (t,i) , f (t,i) >>, it needs to be associated with the objects O in the existing map t-1,j = <p oj , f o,j >>. To calculate the similarity between each pair of objects, the geometric similarity and semantic similarity of the objects are evaluated.

[0077] The geometric similarity φ seg (i, j) measures the spatial overlap degree between the point cloud data of the objects. By calculating the proportion of the points in the point cloud p (t,i) that are the nearest neighbor points in the point cloud p (o,i) , the geometric similarity is obtained as:

[0078] φ geo (i, j) = nnratio(p (t,i) , p (o,i) )

[0079] Among them, the distance threshold is δ nn .

[0080] The semantic similarity φ sem (i, j) is represented by calculating the normalized cosine similarity [] between the visual descriptors of the objects. f t,i and f t,j are the semantic feature vectors of objects i and j respectively, then the semantic similarity is:

[0081]

[0082] These two similarity measures are combined to obtain the total similarity measure φ(i, j):

[0083] φ(i,j) = φ geo (i,j) + φ sem (i,j)

[0084] For each pair of objects, the newly detected objects in the current frame are matched with the existing objects through a greedy assignment strategy, and the match with the highest similarity score is selected. If no match with a similarity higher than the predetermined threshold δ sim is found, a new object is initialized. Over time, the object set O T will be continuously updated.

[0085] After the new object is matched with the existing object, the object fusion process is entered. If the detected object O t-1,j is successfully associated with an object in the map, the detection result is fused with the object in the existing map. Specifically, by updating the semantic feature f oj of the object, the formula is:

[0086]

[0087] where n oj represents the number of detections associated with the object O j so far. In this way, the new detection results are gradually fused into the existing object descriptions, and the semantic information of each object is gradually updated. At the same time, the point cloud data p (t,i) is merged with p (o,i) , and redundant points are removed through downsampling to keep the point cloud data in the scene concise and efficient.

[0088] After obtaining the set of 3D objects O T generated in the previous step, we need to estimate the spatial relationship between them, that is, the edge set E T , to complete the 3D scene graph. The potential connections between them are estimated based on the spatial overlap of the object nodes. We calculate the intersection over union (IoU) of the 3D bounding boxes between each pair of object nodes to obtain the similarity matrix S (i.e., a dense graph). Then, the similarity matrix S is processed by estimating the minimum spanning tree (MST) to select the most representative connecting edges. This step will determine the potential object - to - object edge set E T , that is, the connection relationship between each pair of objects with a high IoU value.

[0089] To further determine the semantic relationships, for each edge in the MST, the information of the object pair (including object description and 3D position) is input into the language model (LLM). The language model is prompted to describe the possible spatial relationships between the objects, such as "a is on b" or "b is inside a", and provide the reasoning behind it. The model will output a relationship label and its detailed explanation. The use of the language model in this step allows us to extend the above nominal edge types to other output relationships that the language model can interpret, such as "a backpack may be stored in a wardrobe" and "papers may be recycled in a trash can".

[0090] After processing the image sequence, a vision-language model (LVLM) is used to generate descriptions for each object. For each object O j , the corresponding images are cropped from the best 10 views and passed to the language model, using the prompt "Describe the central object in the image" to generate a set of preliminary descriptions for each detected object:

[0091]

[0092] Then, each set of preliminary descriptions is passed to another language model (LLM), and these preliminary descriptions are summarized into a more coherent and accurate final description c through prompt instructions j .

[0093] In this process, to remove noise and improve accuracy, the DBSCAN clustering method is first used to cluster the point cloud data and remove noise. Through DBSCAN clustering, meaningful objects can be separated according to density and random noise can be suppressed, which is important for further object recognition and fusion. The accuracy of the reconstruction is further improved through the camera model. The camera model can estimate the spatial relationships between objects by projecting the 3D coordinates of each object onto the image plane and combining the perspective changes. Through the calibration of the camera's internal and external parameters, the position of the object in space can be determined more accurately, and combined with the depth information for optimization, making the reconstruction of the object in 3D space more precise.

[0094] After the above steps, the generated 3D scene graph M T =(O T , E T ) contains the spatial positions, semantic information of the objects, and the spatial relationships between them. This method provides an accurate 3D scene graph representation by combining geometric information, semantic information, and the camera model, which is convenient for use in subsequent tasks.

[0095] Step 3: Grid map and multi-view semantic information fusion.

[0096] Step 3.1: Geometric Consistency Map Fusion Strategy. During the process of multi-view semantic information extraction, due to being in the same scene, the constructed semantic map has a corresponding spatial position relationship with the grid map mentioned above. To ensure the geometric consistency between them, precise geometric calibration and merging strategies must be adopted to achieve the alignment of the two.

[0097] First, it is necessary to ensure that the point cloud data obtained from the RGB-D camera and the lidar can be compared and fused in the same reference coordinate system. Since the lidar and the RGB-D camera are in different sensor coordinate systems respectively, spatial coordinate transformation is required.

[0098] The lidar point cloud P lidar ={p 1 ,p 2 ,…,p n} and the RGB-D point cloud Q rgbd ={q 1 ,q 2 ,…,q m} are respectively represented by points p l ={p l,x ,p l,y ,p l,z} T and q c ={q c,x ,q c,y ,q c,z}, then each point p T in the lidar point cloud needs to be transformed to the point cloud coordinate system of the RGB-D camera through the coordinate transformation matrix l : In the point cloud coordinate system of the RGB-D camera:

[0099]

[0100] Among them, is a 4×4 homogeneous transformation matrix, usually composed of a rotation matrix R and a displacement vector t. The matrix form is as follows:

[0101]

[0102] Here, R is a 3×3 rotation matrix, and t is a 3×1 displacement vector. Through this transformation matrix, each point p lidar in the lidar point cloud P l is transformed to the RGB-D coordinate system to obtain the corresponding point q c :

[0103]

[0104] Figure 7Shows the effects before (a) and after (b) the fusion of the grid map and the RGB-D point cloud. The two point clouds have geometric consistency and achieve spatial synchronization. Before the fusion ( Figure 7 (a)), the grid map and the RGB-D point cloud are not aligned in space, and the geometric information on the grid map and the data of the RGB-D point cloud are discrete in space. After the geometric consistency map fusion ( Figure 7 (b)), each grid point in the grid map is transformed into the coordinate system of the RGB-D point cloud through spatial projection, ensuring that each grid point in the grid map can find its corresponding position on the RGB-D point cloud, thus forming a consistent space representation. It provides prior knowledge for the subsequent semantic-geometric information fusion strategy.

[0105] Step 3.2: Semantic-geometric information fusion strategy. The semantic information extracted by the large model has a one-to-one correspondence with the semantic map constructed by RGB-D. Therefore, after the geometric consistency map fusion strategy, when the geometric map is mapped to the semantic map, the geometric map and the semantic map should also be consistent in spatial coordinates, which can ensure that the spatial position of each object is correctly aligned with its semantic category in the map. To effectively store this information, the fused geometric map and semantic information can be saved together into an octree map (Octomap).

[0106] As Figure 8 shown, in the octree map, each voxel represents a small block area in three-dimensional space. It not only contains spatial coordinate information but can also attach additional attributes, such as the semantic category of the object, the state of the object, etc. For each voxel V(x, y, z), the following content can be stored: spatial coordinates, the semantic label f of the object in the grid map fused with the RGBD point cloud map oj , and the node relationship E T between objects, which can be represented by the following relationship:

[0107]

[0108] Store the fused point cloud data, semantic labels, and spatial relationships into the octree structure. The octree divides the space recursively, dividing the three-dimensional space into cube units (voxels) of different sizes. Each voxel can store spatial information and additional attribute information, such as semantic labels and spatial relationships between objects. The size of the voxel can be dynamically adjusted according to actual needs and the accuracy of the sensor. Larger voxel sizes are used in larger spatial regions, and smaller voxel sizes are used in regions with higher detail requirements.

[0109] Step 3.3: Data Association and Update. Perform data association on the multi-view semantic information and the 2D grid map. At this time, the user does not need to give the robot a specific coordinate position, but only needs to give the semantic information of the target, and the robot can complete the navigation task.

[0110] In the 3D point cloud map, an object is a clustering of a series of points. Representing it in the grid map is an obstacle with a specific shape (it does not exist if it exceeds the point cloud filtering range). For robot navigation, it only needs to reach near the specific object, converting the traditional "A to B" navigation method to "A near B", that is, it does not need to know the coordinates of each point of the object's point cloud, but only needs the geometric center coordinates of the object category to represent the coordinates of this object. All the point clouds of each category are included in a 3D Bounding Box, and the geometric center of the 3D Bounding Box is used to represent the center of the object. Assume that the coordinates of any point in the object category are represented by P i (x, y, z), then the maximum and minimum values of the coordinates of this object in the x, y, and z directions are x max 、x min 、y max 、y min 、z max 、z min , then the geometric center coordinates of the 3D Bounding Box are:

[0111]

[0112] The geometric center point is represented by a six-dimensional matrix as:

[0113] [x, y, z, c, t, s]

[0114] Among them, x, y, and z are the coordinates of the center point, c is the color of the center point, t is the update time of the center point, and s is the semantic information of the center point.

[0115] In the grid map, in order to ensure that the obstacles and semantic information do not interfere with each other, there are four situations for the grid: idle and without semantic information, idle with semantic information, occupied and without semantic information, and occupied with semantic information, which are represented by the following matrix:

[0116] [x, y, o, s n , z n

[0117] Among them, x and y are the coordinates of the grid respectively, o represents the occupancy rate of the grid, s n represents the semantic information of the object on this grid, and z n ​Represents the height value corresponding to the semantic information. Since the number of objects on each grid is uncertain, if the quantity limit is not reached, then s n and z n are assigned a value of 0.

[0118] Furthermore, the present invention also illustrates through constructing experiments and comparing with other methods:

[0119] Figure 9 Shows the experimental results of constructing a geometric grid map of the indoor scene point cloud. The effect of using a LiDAR sensor to scan the scene and construct a geometric map ( Figure 9 (a)), after downsampling and filtering ( Figure 9 (b)), generates a two-dimensional grid map through coordinate transformation ( Figure 9 (c)).

[0120] Figure 10 Shows the effect of generating a semantic map through a camera model after extracting semantic information from multiple views. By projecting the two-dimensional semantic segmentation results of multiple views into a three-dimensional geometric map, a three-dimensional map containing semantic labels is formed. The generated semantic map not only retains the spatial structure information of the geometric map but also assigns a semantic category label to each point, thus realizing the organic combination of geometric information and semantic information. It can be observed in the experiment that different semantic categories (such as tables, walls, computers, chairs, etc.) are accurately segmented and presented in the three-dimensional map, with clear boundaries between categories and a high consistency between semantic annotations and geometric structures. This indicates that through the fusion of the camera model and multi-view semantic information, the detail richness and spatial accuracy of the semantic map can be effectively improved, providing a reliable environmental representation for robot navigation and task decision-making in complex scenarios. Table 1 below shows the accuracy verification of semantic information extraction for 3D semantic segmentation in the same scene, comparing the mAcc and F-mIOU of three-dimensional semantic segmentation using different segmentation models. The method of the present invention is significantly superior to other models in the zero-shot semantic segmentation task. In the two key indicators of mAcc and F-mIOU, the method of the present invention reaches 86.3% and 78.8% respectively, showing higher classification accuracy and segmentation boundary consistency compared with CLIP-Field, SAM3DPro, and SAGA. This advantage is attributed to the more accurate zero-shot detection and segmentation capabilities of Yolo-word and SAM, effectively solving problems such as resolution differences, spatial inaccuracies, and semantic information loss, resulting in good recognition effects for object categories and fine-grained features in complex indoor scenes. The experimental results verify the robustness and superiority of the method of the present invention in a dynamic multi-view environment, demonstrating the innovative value and application potential in the field of zero-shot segmentation.

[0121] Table 1 Comparison of mAcc and F-mIOU of different methods

[0122]

[0123] Figure 11 It shows the effect of fusing a two-dimensional geometric grid map with a three-dimensional semantic map. By projecting the semantic labels of the three-dimensional semantic map into the two-dimensional grid map, a two-dimensional semantic grid map with both geometric structure and semantic information is generated. The fused map not only retains the spatial distribution characteristics of the traditional two-dimensional geometric grid map but also further enriches its semantic information, enabling each grid cell to contain the corresponding semantic category label. In the experiment, it can be clearly seen that the semantic categories of different regions (such as computers, desks and chairs, door frames) are accurately presented in the two-dimensional grid map, and the classification boundaries are highly consistent with the geometric distribution. This fusion method effectively enhances the expression ability of the two-dimensional grid map at the semantic level and provides a more intuitive and information-rich environmental representation tool for robot path planning, object recognition, and task execution.

[0124] In summary, the method of the present invention has the following beneficial effects:

[0125] 1. Achieved precise fusion of the geometric map and multi-view semantic information: Successfully solved the mismatches in resolution, scale, and dimension between the 3D point cloud geometric information and multi-view semantic information, effectively avoiding problems such as spatial errors, duplicate or missing semantic labels that may occur during direct fusion.

[0126] 2. Improved the object recognition and semantic understanding ability of robots in complex scenarios: Overcame the challenges of difficult object category recognition and insufficient understanding of object relationships by mobile robots in complex environments, and significantly enhanced the navigation and intelligent decision-making ability of mobile robots in dynamic and unstructured scenarios.

[0127] The above-disclosed is only a preferred embodiment of the present invention. Of course, it cannot be used to limit the scope of the rights of the present invention. Those of ordinary skill in the art can understand the entire or part of the processes of implementing the above embodiments, and the equivalent changes made according to the claims of the present invention still fall within the scope covered by the invention.

Claims

1. A method for constructing a three-dimensional semantic map of a mobile robot by collaborative fusion of geometry and semantics, characterized in that: The following steps are involved: Step 1: Point cloud data and multi-view image acquisition; Step 2: Geometric map and multi-view image preprocessing; Step 3: Fusion of raster map and multi-view semantic information.

2. The method for constructing a three-dimensional semantic map of a mobile robot by using geometric-semantic collaborative fusion as claimed in claim 1, characterized in that: The execution process of step 1 includes the following steps: Step 1.1: LiDAR data collection, odometer-assisted positioning is used at the front end, and optimization is performed based on loop detection and GTSAM factor graph at the back end; Step 1.2: RGB-D camera multi-view image acquisition.

3. The method for constructing a three-dimensional semantic map of a mobile robot by using geometric-semantic collaborative fusion as claimed in claim 2, characterized in that: The execution process of step 2 includes the following steps: Step 2.1: LiDAR data preprocessing; Step 2.2: RGB-D multi-view image preprocessing; Step 2.3: Object-oriented semantic relationship reasoning and scenario construction.

4. The method for constructing a three-dimensional semantic map of a mobile robot by using geometric-semantic collaborative fusion as claimed in claim 3, characterized in that: The process of LiDAR data preprocessing is to first remove irrelevant points in the original point cloud scene, then downsample the point cloud, perform statistical filtering, and then use the coordinate system mapping method to convert the indoor scene point cloud into a raster map.

5. The method for constructing a three-dimensional semantic map of a mobile robot by using geometric-semantic collaborative fusion as claimed in claim 4, characterized in that: In the process of RGB-D multi-view image preprocessing, a two-dimensional segmentation model without category restriction is used to perform category-free segmentation on objects in the image to generate a mask set of candidate objects; And use visual feature extractor to process the region of each object.

6. The method for constructing a three-dimensional semantic map of a mobile robot by using geometric-semantic collaborative fusion as claimed in claim 5, characterized in that: The object-oriented semantic relationship reasoning and scene construction process requires constructing an object set O by extracting object data from the image. T , estimate the spatial relationship between objects to generate edge set E T , completing the construction of the scene graph.

7. The method for constructing a three-dimensional semantic map of a mobile robot by using geometric-semantic collaborative fusion as claimed in claim 6, characterized in that: The execution process of step 3 includes the following steps: Step 3.1: Geometric consistency map fusion strategy; Step 3.2: Semantic-geometric information fusion strategy; Step 3.3: Data association and update.

8. The method for constructing a three-dimensional semantic map of a mobile robot by using geometric-semantic collaborative fusion as claimed in claim 7, characterized in that: During the execution of the geometric consistency map fusion strategy, the point cloud data of the lidar and RGB-D camera are fused through spatial coordinate transformation.

9. The method for constructing a three-dimensional semantic map of a mobile robot by using geometric-semantic collaborative fusion as claimed in claim 8, characterized in that: During the execution of the semantic-geometric information fusion strategy, an octree map is used to save the fused geometric map and semantic information.

10. The method for constructing a three-dimensional semantic map of a mobile robot by using geometric-semantic collaborative fusion as claimed in claim 9, characterized in that: Data association and updating specifically involves data association between multi-view semantic information and two-dimensional grid maps. In this process, the robot can complete the navigation task with only the semantic information of the given target.

Images