A method for constructing outdoor instance-level semantic maps based on multi-sensor data fusion

Abstract

The present invention discloses a method for constructing an outdoor instance-level semantic map by fusion of multi-sensor data. The method comprises the following steps: calibrating external parameters of an RGB camera and a solid-state laser radar; performing panoramic segmentation on the RGB image to extract instance semantic information in the environment; providing real-time posture using a positioning algorithm based on the RGB camera and the solid-state laser radar; processing the received solid-state laser radar point cloud according to the real-time posture state and instance semantic information using a point cloud receiving strategy of cross-frame accumulation and latest frame selection to obtain a three-dimensional point cloud for semantic fusion; fusing the instance semantic information obtained by panoramic segmentation with the solid-state laser radar point cloud to obtain a point cloud cluster with semantic information; converting the point cloud cluster with semantic information to a global coordinate system according to the real-time posture to generate a global point cloud containing semantic information; implementing a global label hierarchical fusion strategy of environmental semantics and object instances on the global map, and then updating the global map to obtain an outdoor instance-level semantic map.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot autonomous environmental perception and multimodal data fusion, and in particular relates to a method for constructing an outdoor instance-level semantic map by fusion of multi-sensor data. Background Art

[0002] The semantic map of the outdoor scene provides semantic information for scene understanding, which can ensure the effectiveness of robot navigation, positioning and autonomous driving, and help the robot better complete advanced interactive tasks such as target search and change detection.

[0003] Radar point clouds and RGB images are two important sensor data types for constructing semantic maps of outdoor scenes. One existing approach to constructing semantic maps of outdoor scenes is to propose lidar-based semantic mapping. By using prior information in the scene structure, different levels of semantic information are extracted from the original point cloud. This method can not only construct an occupancy grid map for navigation, but also a hierarchical semantic map that meets the needs of customized disinfection tasks. However, the semantic map constructed by this algorithm is a two-dimensional semantic map that cannot contain complete environmental semantic information. Another approach believes that high-level situational awareness requires 3D representation and the inclusion of semantic information. It proposes a hierarchical 3D grid mapping framework, UFOMap, to construct a dense, high-resolution map containing semantic information. However, in the process of semantic extraction from outdoor RGB images, it is generally impossible to distinguish individual instances, and a large amount of environmental semantic information is ignored during instance segmentation. At the same time, because environmental semantics and object instances have different characteristics, it is impossible to construct an outdoor instance-level semantic map with global consistency in environmental semantics. Summary of the Invention

[0004] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provides a method for constructing an outdoor instance-level semantic map by fusion of multi-sensor data.

[0005] The technical solution of the present invention is: a method for constructing an outdoor instance-level semantic map by fusion of multi-sensor data, which is carried out in the following steps:

[0006] Step 1. Calibrate the external parameters of the RGB camera and solid-state lidar;

[0007] Step 2. Perform panoptic segmentation on the RGB image to extract instance semantic information in the environment;

[0008] Step 3. Use a positioning algorithm based on an RGB camera and solid-state lidar to provide real-time pose;

[0009] Step 4. Process the received solid-state lidar point cloud based on the real-time pose and instance semantic information to obtain a 3D point cloud for semantic fusion.

[0010] Step 5. Combining the calibration parameters of the RGB camera and solid-state LiDAR, the 3D point cloud is projected onto a 2D image plane. Based on the pixel positions of the projected 3D point cloud points within the 2D image plane, the corresponding instance semantic information is determined. This instance semantic information obtained from panoptic segmentation is then fused with the solid-state LiDAR point cloud to obtain a point cloud cluster with semantic information.

[0011] Step 6. Based on the real-time pose, transform the point cloud cluster with semantic information into the global coordinate system to generate a global point cloud with semantic information.

[0012] Step 7. Associate the single-frame information of the global point cloud containing semantic information with the global map data, determine the global labels of the environmental semantic instances and object instances in the global point cloud containing semantic information, implement hierarchical fusion of global labels on the global map, and then update the global map to obtain an outdoor instance-level semantic map.

[0013] The preferred technical solution is that step 1 is to collect a section of sensor data of the RGB camera and the solid-state laser radar as calibration data, and use a pixel-level automatic calibration method under target-free conditions to calibrate the external parameters of the RGB camera and the solid-state laser radar.

[0014] The preferred technical solution is that the step 2 is to use a panoramic segmentation module based on Mask2Former to

[0015] Semantic information is extracted from the RGB image and converted into a custom data format, which is an image semantic information topic for ROS system communication.

[0016] The preferred technical solution is that step 4 is carried out in sequence according to the following steps:

[0017] Step 4.1 point cloud frame reception;

[0018] Step 4.2: Update the latest point cloud frame;

[0019] Step 4.3 receives the point cloud and monitors the pose of the current frame. If there is no pose, return to step 4.1. If there is a pose, transfer the current frame to the global system and accumulate the point cloud to obtain the accumulated point cloud.

[0020] Step 4.4: Receive the panoramic segmentation results and the accumulated point cloud, and monitor the current frame pose. If there is no pose, use the latest frame point cloud as the 3D point cloud for semantic fusion. If there is a pose, pre-process the accumulated point cloud and transfer it to the current frame as the 3D point cloud for semantic fusion.

[0021] The preferred technical solution is that the global labels of the environment semantic instances and object instances in the global point cloud containing semantic information are determined in step 7 respectively, by following the steps:

[0022] Step 7.1: For a single frame of global point cloud containing semantic information, determine whether it is an object instance. If not, generate a global label using the semantic label. If yes, proceed to step 7.2.

[0023] Step 7.2 determines whether a global label exists. If not, a new global label is generated using the semantic label. If yes, the new global label is matched with the global label.

[0024] Compared with the prior art, the present invention first calibrates the external parameters of the RGB camera and solid-state laser radar, so that the solid-state laser radar point cloud can be subsequently projected onto the RGB image plane to achieve the fusion of geometric and semantic information; second, a positioning algorithm based on the RGB camera and solid-state laser radar is used to provide the algorithm with real-time pose; third, a point cloud receiving strategy of cross-frame accumulation and latest frame selection is proposed. This strategy retains the latest frame point cloud on the basis of cross-frame accumulation of point cloud, and chooses to use the accumulated point cloud or the latest frame point cloud according to the point cloud accumulation situation. It can retain robustness on the basis of collecting as much point cloud information as possible, and avoid the failure of the global mapping algorithm caused by point cloud accumulation failure; fourth, the global labels of the environment semantic instances and object instances in the global point cloud containing semantic information are determined respectively, and the hierarchical fusion of global labels is realized on the global map, which ensures the global consistency of the environment semantics and the individual specificity of the instance objects, and uses fixed colors for the unified environment semantic labels to improve the visualization effect of the global instance-level semantic map. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is an overall flow chart of an embodiment of the present invention.

[0026] Figure 2 This is a flowchart of step 4 of an embodiment of the present invention.

[0027] Figure 3 This is a flowchart of determining the global label in step 7 of an embodiment of the present invention.

[0028] Figure 4 This is a comparison chart of the geometric-semantic fusion results of the embodiment of the present invention and the prior art.

[0029] Figure 5 1 is an ablation experiment result diagram of the hierarchical fusion of environmental semantics and object instances and a comparative example according to an embodiment of the present invention.

[0030] Figure 6 It is an outdoor instance-level semantic map constructed by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention provides a method for constructing an outdoor instance-level semantic map based on multi-sensor data fusion, such as Figure 1Proceed as follows:

[0032] Step 1. Calibrate the external parameters of the RGB camera and solid-state lidar;

[0033] The specific method is to build a data collection platform with an RGB camera and a solid-state lidar, collect a section of sensor data from the RGB camera and the solid-state lidar as calibration data, and use a pixel-level automatic calibration method under target-free conditions to calibrate the external parameters of the RGB camera and the solid-state lidar and the internal parameters of the RGB camera, so that the solid-state lidar point cloud can be projected onto the RGB image plane, and the corresponding RGB image information and processed semantic information can be found to achieve the fusion of geometric and semantic information, providing a basis for subsequent semantic fusion and global mapping.

[0034] The pixel-level automatic calibration method under targetless conditions can be carried out according to the method described in the existing literature (Yuan C, Liu X, Hong X, et al. Pixel-Level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments[J]. IEEE Robotics and Automation Letters, 2021, 6(4): 7517-7524).

[0035] Step 2. Use the built data collection platform with RGB cameras and solid-state lidar to collect RGB images, perform panoptic segmentation on the RGB images, and extract instance semantic information in the environment;

[0036] The specific method is to convert the RGB image into the target image format that can be operated by OpenCV.

[0037] The panoramic segmentation module of Mask2Former extracts semantic information from RGB images and converts this information into a custom data format. This custom data format is an image semantic information topic that can be used for ROS system communication, as shown in Table 1. This data is stored in the ROS bag format to facilitate subsequent processing and complete the transfer of image semantic information between systems.

[0038] Table 1 Customized image semantic information topics

[0039]

[0040] Step 3. Use the SLAM positioning algorithm based on RGB camera and solid-state lidar to provide real-time pose and send the real-time pose using the ROS topic;

[0041] Step 4. Process the received solid-state lidar point cloud based on the real-time pose and instance semantic information to obtain a 3D point cloud for semantic fusion.

[0042] Because panoptic segmentation processing takes a long time and is significantly outpaced by the frequency of point cloud publications, single-frame point clouds are relatively sparse. Furthermore, while maintaining robustness of point cloud accumulation and ensuring algorithm integrity, this paper proposes a point cloud reception strategy that combines cross-frame accumulation with latest frame selection. This strategy, while utilizing cross-frame point cloud accumulation, retains the latest frame point cloud and, depending on the point cloud accumulation, selects either the accumulated point cloud or the latest frame point cloud. This maintains robustness while collecting as much point cloud information as possible, avoiding failures in the global mapping algorithm caused by point cloud accumulation failures.

[0043] The specific method is to follow the Figure 2 Follow the steps shown:

[0044] Step 4.1 point cloud frame reception;

[0045] Step 4.2: Update the latest point cloud frame;

[0046] Step 4.3: Current frame pose monitoring. If there is no pose, return to step 4.1. If there is a pose, transfer the current frame to

[0047] Global system, perform point cloud accumulation to obtain accumulated point cloud;

[0048] Step 4.4: Receive the panoptic segmentation results and the accumulated point cloud, and monitor the pose of the current frame. If there is no pose, use the point cloud of the latest frame as the 3D point cloud for semantic fusion. If there is a pose, pre-process the accumulated point cloud and transfer it to the current frame as the 3D point cloud for semantic fusion.

[0049] Step 5. Combining the calibration parameters of the RGB camera and solid-state LiDAR, the 3D point cloud is projected onto a 2D image plane. Based on the pixel positions of the projected 3D point cloud points within the 2D image plane, the corresponding instance semantic information is determined. This instance semantic information obtained from panoptic segmentation is then fused with the solid-state LiDAR point cloud to obtain a point cloud cluster with semantic information.

[0050] Step 6. Based on the real-time pose, transform the point cloud cluster with semantic information into the global coordinate system to generate a global point cloud with semantic information.

[0051] Step 7. In order to unify the single-frame instance and the global instance, the present invention uses a data association strategy based on frame count to associate the single-frame information of the global point cloud containing semantic information with the global map data, and respectively determine the global labels of the environment semantic instance and object instance in the global point cloud containing semantic information. The specific method is as follows: Figure 3 As shown:

[0052] Step 7.1: For a single frame of global point cloud containing semantic information, determine whether it is an object instance. If not, generate a global label using the semantic label. If yes, proceed to step 7.2.

[0053] Step 7.2 determines whether a global label exists. If not, a new global label is generated using the semantic label. If yes, the new global label is matched with the global label.

[0054] By implementing hierarchical fusion of global labels on the global map, and then updating the global map, we obtain an outdoor instance-level semantic map. This ensures the global consistency of environmental semantics and the individual specificity of instance objects. By using fixed colors for unified environmental semantic labels, we improve the visualization of the global instance-level semantic map.

[0055] The comparison of the geometric-semantic fusion results of the embodiment of the present invention and the prior art is shown in the figure below: Figure 4 As shown, different colors of the point cloud represent different fused semantic categories. The point clouds in (a) and (b) are created in the same scene and at the same location. (a) shows the geometric-semantic fusion result of the prior art strategy for synchronously receiving point clouds and semantic information; (b) shows the geometric-semantic fusion result of the point cloud reception strategy using cross-frame accumulation and latest frame selection according to an embodiment of the present invention.

[0056] The comparison results show that for the same object at the same location, the number of points in Figure (a) is small and the point cloud is sparse, while the number of points in Figure (b) is greater and the point cloud is denser. This comparison of point cloud density demonstrates that the proposed point cloud reception strategy of cross-frame accumulation and latest frame selection can effectively increase point cloud density, thereby further improving map construction quality.

[0057] The ablation experiment results of the hierarchical fusion of environmental semantics and object instances in the embodiment of the present invention and the comparative example are shown in the figure. Figure 5 shown.

[0058] Figure 5(a) is a comparative ablation experiment result diagram without using the strategy proposed by the present invention; Figure (b) is an ablation experiment result diagram of an embodiment of the present invention. The images (1), (2), (3), and (4) on the right are detailed images of the red-framed areas corresponding to the map. Areas (1) and (2) are both tree and vegetation environments, and areas (3) and (4) are both ceiling environments. In Figure (a), as shown by the red frames (1) and (2), the environments of the two tree and vegetation semantic categories are not connected in space. In the constructed map, the two tree and vegetation use different global labels and colors, indicating that they belong to two different instances of the tree and vegetation class. For environmental semantics, it only detects semantic categories without distinguishing instances. Therefore, in the constructed global map, environmental semantics of the same category are not globally unique. In Figure (b), as shown by the red frames (1) and (2), the two tree and vegetation have the same color, indicating that they belong to the same global label and the same instance, ensuring the global uniqueness of environmental semantics. Similarly, comparing areas (3) and (4) in Figure (a) and (b), the two ceilings are not connected in space. Areas (3) and (4) in Figure (a) use different colors, indicating that they belong to multiple instances, while areas (3) and (4) in Figure (b) maintain the same color, indicating that they belong to the same instance. From the comparison results, it can be seen that the map constructed using the present invention has a greater advantage in expressing environmental semantics.

[0059] The outdoor instance-level semantic map constructed according to the method of the embodiment of the present invention is as follows: Figure 6 For instance objects, random colors are used in the map. For globally unified environmental semantics, a unified color is used across multiple experiments to ensure consistent colors across experiments, improving map understanding and visualization. Figure 6 Below is a legend for environmental semantic categories and their corresponding colors. Different colors are used for environmental semantic labels such as trees, ceilings, brick walls, grass, windows, buildings, ground, walls, and paths, which improves the visualization of the global instance-level semantic map.

Claims

1. A method for constructing outdoor instance-level semantic maps based on multi-sensor data fusion, characterized by Follow these steps: Step 1. Calibrate the external parameters of the RGB camera and solid-state lidar; Step 2. Perform panoptic segmentation on the RGB image to extract instance semantic information in the environment; Step 3. Use a positioning algorithm based on an RGB camera and solid-state lidar to provide real-time pose; Step 4. Process the received solid-state lidar point cloud based on the real-time pose and instance semantic information to obtain a 3D point cloud for semantic fusion. Step 5. Combining the calibration parameters of the RGB camera and solid-state LiDAR, the 3D point cloud is projected onto a 2D image plane. Based on the pixel positions of the projected 3D point cloud points within the 2D image plane, the corresponding instance semantic information is determined. This instance semantic information obtained from panoptic segmentation is then fused with the solid-state LiDAR point cloud to obtain a point cloud cluster with semantic information. Step 6. Based on the real-time pose, transform the point cloud cluster with semantic information into the global coordinate system to generate a global point cloud with semantic information. Step 7. Associate the single-frame information of the global point cloud containing semantic information with the global map data, determine the global labels of the environmental semantic instances and object instances in the global point cloud containing semantic information, implement hierarchical fusion of global labels on the global map, and then update the global map to obtain an outdoor instance-level semantic map.

2. The method for constructing an outdoor instance-level semantic map based on multi-sensor data fusion according to claim 1, characterized in that: The step 1 is to collect a segment of sensor data of the RGB camera and the solid-state laser radar as calibration data, and use a pixel-level automatic calibration method under target-free conditions to calibrate the external parameters of the RGB camera and the solid-state laser radar.

3. The method for constructing an outdoor instance-level semantic map based on multi-sensor data fusion according to claim 2 is characterized in that The step 2 is to use a panoramic segmentation module based on Mask2Former to extract semantic information from the RGB image and convert the semantic information into a custom data format, which is an image semantic information topic for ROS system communication.

4. The method for constructing an outdoor instance-level semantic map based on multi-sensor data fusion according to claim 3 is characterized in that Step 4 is performed in the following steps: Step 4.1 point cloud frame reception; Step 4.2: Update the latest point cloud frame; Step 4.3 receives the point cloud and monitors the pose of the current frame. If there is no pose, return to step 4.

1. If there is a pose, transfer the current frame to the global system and accumulate the point cloud to obtain the accumulated point cloud. Step 4.4: Receive the panoramic segmentation results and the accumulated point cloud, and monitor the current frame pose. If there is no pose, use the latest frame point cloud as the 3D point cloud for semantic fusion. If there is a pose, pre-process the accumulated point cloud and transfer it to the current frame as the 3D point cloud for semantic fusion.

5. The method for constructing an outdoor instance-level semantic map based on multi-sensor data fusion according to claim 4, characterized in that: The determination of the global labels of the environment semantic instances and object instances in the global point cloud containing semantic information in step 7 is performed according to the following steps: Step 7.1: For a single frame of global point cloud containing semantic information, determine whether it is an object instance. If not, generate a global label using the semantic label. If yes, proceed to step 7.

2. Step 7.2 determines whether a global label exists. If not, a new global label is generated using the semantic label. If yes, the new global label is matched with the global label.

Images