Autonomous exploration method based on multi-modal data fusion and large language model

Abstract

The invention discloses an autonomous exploration method based on multi-modal data fusion and a large language model, solves the problems of low autonomous exploration efficiency, insufficient decision intelligence, poor environmental adaptability and the like in the prior art, and belongs to the technical field of robots. The autonomous exploration method comprises the following steps: S1, deeply fusing laser radar geometric information and visual semantic information through cooperative work of a three-dimensional laser and a visual sensor, and constructing an accurate semantic map; s2, performing code conversion on the semantic map to enable a large language model to understand and perform reasoning; and S3, based on an output result of the large language model in the S2, selecting semantic-oriented exploration or geometric frontier exploration until the task is completed or all regions are explored. According to the method, accurate and detailed environment perception is realized, the intelligence and task correlation of exploration decision making are improved, and the method has wide application potential.

Description

TECHNICAL FIELD

[0001] The application relates to an autonomous exploration method based on multi-modal data fusion and a large language model, in particular to a robot autonomous exploration method based on multi-modal data fusion and a large language model, and belongs to the technical field of robots. BACKGROUND

[0002] With the increasing demand for autonomous navigation, exploration and operation of robots in complex environments, robot perception and navigation technology has become a research hotspot. At present, robot autonomous exploration generally relies on laser radar (LiDAR) and visual sensors to obtain environmental information to realize map construction and path planning.

[0003] The exploration technology based on laser radar mapping, such as GBPlanner and TARE, can construct two-dimensional or three-dimensional grid maps of the environment due to the high-precision ranging performance of laser radar, and realize the positioning and obstacle avoidance of robots through geometric structure information, but only contains geometric information, lacks semantic understanding of objects in the environment, and is difficult to support high-level intelligent decision-making and task-oriented exploration.

[0004] Pure visual methods use cameras to collect rich image information and obtain semantic information of the environment through visual semantic segmentation and target detection algorithms, making up for the lack of semantic understanding of laser radar. However, visual sensors are sensitive to environmental lighting conditions. In strong light, backlight, glare or night low-illumination conditions, the quality of visual information decreases significantly, which seriously limits the system performance and affects the stability of navigation and obstacle avoidance. SUMMARY

[0005] The technical problem to be solved by the application is to overcome the shortcomings of the prior art and solve the problems of low autonomous exploration efficiency, insufficient decision-making intelligence and poor environmental adaptability in the prior art.

[0006] The purpose of the application is achieved by the following technical solutions:

[0007] A robot autonomous exploration method based on multi-modal data fusion and a large language model, comprising:

[0008] S1, through the cooperative work of three-dimensional laser and visual sensors, deeply fuse laser radar geometric information and visual semantic information, and construct an accurate semantic map;

[0009] S2, encoding conversion is performed on the semantic map, so that the large language model can understand and reason;

[0010] S3, based on the output result of the large language model in S2, semantic guided exploration or geometric frontier exploration is selected until the task is completed or all areas are explored.

[0011] In an embodiment of the present application, in S1, the three-dimensional laser radar acquires point cloud data, subscribes to the point cloud data by using a SLAM method, and publishes a two-dimensional grid map; an image data is acquired by using a vision sensor, a target detection model is used for target detection, and a confidence of a corresponding target is given, a semantic segmentation model is used for semantic labeling of the target detection, and the two-dimensional grid map and the semantic labeling result are fused.

[0012] In an embodiment of the present application, in S1, a front point is extracted in the two-dimensional grid map, all the front points are clustered to form a plurality of front clusters, a centroid of each front cluster is calculated, and a basic priority is calculated to evaluate an exploration priority.

[0013] In an embodiment of the present application, the target detection model uses a lightweight open scene algorithm YoloE, an EfficientRep backbone network improved by using the YoloE algorithm and a SimOTA label allocation strategy are used, precise detection of a plurality of targets is realized under the premise of ensuring real-time performance, the target detection model is switched to an Owl-Vit model or a Grounding dino model for long-tail distribution or rare targets, and the detection capability for unknown targets is enhanced through a Transformer architecture and a text guiding mechanism.

[0014] In an embodiment of the present application, the fusion of the two-dimensional grid map and the semantic labeling result includes:

[0015] 1) for each object instance segmented by the semantic segmentation model, a pixel mask thereof is acquired;

[0016] 2) a center pixel coordinate (u, v) of the pixel mask is calculated;

[0017] 3) a depth value d of the depth image collected by the vision sensor at (u, v) is queried;

[0018] 4) the pixel coordinate (u, v, d) is converted into a three-dimensional point P_cam in a vision sensor coordinate system by using an intrinsic parameter of the vision sensor;

[0019] 5) P_cam is converted into P_map in a global map coordinate system by using a pre-calibrated tf transformation relationship; the tf transformation relationship refers to a transformation relationship from a vision sensor coordinate system to a robot body coordinate system to a global map coordinate system;

[0020] 6) Semantic map representation: Maintain a dynamic list SemanticObjects, where each element contains: {object_label, map_coordinates, detection_confidence, timestamp}, which are the label and category of the semantic information, the global coordinates of the semantic information, the confidence of the semantic information, and the timestamp of the discovery of the semantic information, respectively.

[0021] In one embodiment of the present invention, S2, the encoding and conversion of the semantic map includes: converting the semantic map into structured text; simultaneously integrating the user-input natural language task instructions with the structured text into a unified format, and then performing feature extraction to generate an input vector acceptable to a large language model.

[0022] In one embodiment of the present invention, S2 further includes semantic relevance calculation, specifically: inputting the input vector into the large language model, using the attention mechanism of the Transformer architecture inside the large language model, calculating the relevance between the task instruction and each semantic point in the semantic map, and outputting a list of semantic points sorted by relevance, providing a quantitative basis for exploration decision-making.

[0023] In one embodiment of the present invention, semantic-guided exploration refers to: when the relevance of the semantic points output by the large language model is higher than a set threshold, searching for leading points in the vicinity of the corresponding semantic points and raising their priority to the highest globally; using an improved A* algorithm, combined with the geometric information of the grid map, planning the optimal path to the priority leading points, and guiding the robot to prioritize the exploration of areas related to the task.

[0024] In one embodiment of the present invention, geometric front exploration refers to: if the relevance of all semantic points is lower than a threshold, switching to the traditional geometric front exploration mode, using a boundary extraction algorithm to identify global front points based on the two-dimensional grid map of LiDAR and exploration priority, and using the D*Lite algorithm for incremental path planning to respond to environmental changes in real time.

[0025] In one embodiment of the present invention, during the exploration process, the robot triggers the semantic map update and large language model recalculation process every time it moves a fixed distance or discovers new semantic information, dynamically adjusting the exploration strategy to ensure the timeliness and accuracy of decision-making.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] (1) This invention achieves accurate and detailed environmental perception: by deeply integrating LiDAR and visual semantic information, the limitations of a single sensor are effectively overcome. LiDAR ensures the geometric accuracy of map construction and can accurately depict information such as the distribution of obstacles and spatial dimensions in the environment; the visual sensor gives the robot the ability to understand environmental semantics and can identify various objects and area functions; the combination of the two enables the robot to form a more comprehensive and detailed environmental representation, whether it is the furniture layout and functional zoning in complex indoor scenes or the identification of obstacles and signs in outdoor environments, it can accurately grasp the situation and provide a reliable basis for subsequent exploration.

[0028] (2) This invention improves the intelligence and task relevance of exploration decision-making: It fully leverages the powerful semantic understanding and reasoning capabilities of the large language model to perform deep semantic matching analysis on the fused semantic map and task objectives. When performing exploration tasks, the robot can prioritize exploring environmental areas highly relevant to the task objectives based on the calculation results of the large language model. For example, in a rescue mission to search for trapped personnel, it can quickly locate semantic regions related to "people" and prioritize their exploration, avoiding blind searching, significantly improving exploration efficiency and task completion rate, and making the robot's exploration decisions more targeted and intelligent.

[0029] (3) This invention has broad application potential: With its excellent environmental adaptability and intelligent exploration performance, this method can be widely applied in many fields such as rescue missions and intelligent inspections. In rescue missions, it helps to quickly locate trapped personnel; in intelligent inspections, it ensures a comprehensive check of equipment status. This will powerfully promote robots towards a higher level of intelligent autonomous exploration, expanding their boundaries and value in practical applications.

[0030] (4) The present invention overcomes the problems of existing technologies, such as the ability of lidar-based mapping exploration to obtain only geometric structure information of the environment and lack of environmental semantic understanding; the great influence of lighting on pure visual semantic navigation and poor adaptability; and the lack of multimodal data fusion, making it difficult to form high-level semantic decision support. Attached Figure Description

[0031] Figure 1 This is an architecture diagram of a robot autonomous exploration method based on multimodal data fusion and a large language model.

[0032] Figure 2 This is a flowchart of a robot autonomous exploration method based on multimodal data fusion and a large language model.

[0033] Figure 3 Flowchart for building semantic maps. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0035] A robot autonomous exploration method based on multimodal data fusion and a large language model (LLM) is proposed. This method integrates LiDAR geometric information and visual semantic information, employing an exploration framework with high-level decision-making by a LLM. The framework utilizes LiDAR mapping algorithms to construct a two-dimensional grid map and extracts frontiers as potential exploration targets. Simultaneously, a visual semantic segmentation network identifies key objects in images acquired by a visual sensor, obtaining visual semantic information. This visual semantic information is then fused with the two-dimensional grid map to form a semantic map. The LLM parses the user-defined natural language task and matches it with the visual semantic information. If a highly relevant semantic object exists, the frontiers near that object in the semantic map are prioritized for exploration; otherwise, an information gain-based frontier exploration strategy in the two-dimensional grid map is adopted. The architecture of the robot autonomous exploration method based on multimodal data fusion and a large language model is as follows: Figure 1 As shown, the process is as follows: Figure 2 As shown.

[0036] A robot autonomous exploration method based on multimodal data fusion and a large language model, specifically including:

[0037] S1. Multimodal data acquisition and fusion mapping

[0038] Multimodal data acquisition and fusion mapping are the core foundation for realizing robot environmental cognition. By working collaboratively with 3D LiDAR and visual sensors, and deeply fusing geometric information from the LiDAR with semantic information from the vision, an accurate semantic map of the environment is constructed. The mapping process is as follows: Figure 3 As shown.

[0039] S11. Sensor Data Acquisition: The robot is equipped with a 3D LiDAR and a vision sensor (e.g., a high-resolution RGB-D camera). The 3D LiDAR scans the environment 360° at high frequency to acquire point cloud data containing geometric information such as obstacle distances and angles. The density of the point cloud data is reduced through a VoxelGrid filtering algorithm, reducing computational load while maintaining accuracy. The vision sensor is equipped with an automatic exposure and low-light enhancement module, combined with a wide-angle lens to achieve a large field of view coverage, acquiring environmental image sequences at a frequency of 30 frames per second to ensure the real-time performance and integrity of the data.

[0040] S12. Point Cloud Data Processing: SLAM methods (such as GMAPping, PointLio, FastLio, etc.) are used to subscribe to point cloud data acquired by 3D LiDAR and publish a 2D raster map ( / map). The value of a map cell is represented as -1 (unknown), 0 (free), or 100 (occupied). SLAM methods are computationally efficient, suitable for mapping in 2D / 3D environments, provide a basic geometric map for frontline point detection, and offer a unified global map coordinate system.

[0041] S13. Frontier Extraction and Evaluation: Frontiers are a core concept in autonomous robot exploration, defined as boundary points between known free space and unknown space. They represent locations the robot can move to, extending the boundaries of the known area, and maximizing the acquisition of new information by reaching these points. A grid cell is extracted as a frontier if its state is unknown and it has at least one adjacent grid cell that is occupied / free. However, since the extracted frontiers are usually numerous and dense, DBSCAN (density-based clustering algorithm) is used to cluster all frontiers into several frontier clusters. For each frontier cluster, its centroid p is calculated. f This serves as the representative front point of the cluster.

[0042] For the centroid p of each front cluster f Basic priority calculation (when there is no semantic information) is required, and the exploration priority score needs to be evaluated. base .

[0043] Score base (p f ) = w i InfoGain(p f )-w c Cost(probot,p f )

[0044] Information gain refers to the percentage of information received when the evaluation reaches p. f The size of the unknown region that may be observed later; its simplified calculation method is to statistically analyze p. f The number of unknown cells within a certain radius (e.g., 5 meters).

[0045] The movement cost (Cost) refers to the cost of moving from the robot's current position to position p. f The navigation path length (which can be estimated using the A* algorithm), and probot represents the energy consumption.

[0046] w i and w c It is a weighting factor used to balance exploration gains and movement costs.

[0047] S14. RGB Image Processing from Visual Sensor: Image data acquired by the visual sensor is input into the object detection model and the semantic segmentation model. The object detection model adopts the lightweight open-scene algorithm YOLOE, utilizing an improved EfficientRep backbone network and SimOTA label allocation strategy to achieve accurate detection of various targets while ensuring real-time performance. For long-tailed or rare targets, the object detection model switches to the Owl-Vit model or Groundingdino model, enhancing the detection capability of unknown targets through the Transformer architecture and text guidance mechanism. Based on object detection, the object detection model provides the confidence score for each target. The semantic segmentation model introduces the SegmentAnything model. The powerful zero-shot segmentation capability of the SegmentAnything model allows for pixel-level semantic annotation of the detection results output by the object detection model.

[0048] S15, Data Fusion Mapping:

[0049] 1) For each object instance segmented by the semantic segmentation model, obtain its pixel mask.

[0050] 2) Calculate the center pixel coordinates (u,v) of the pixel mask.

[0051] 3) Query the depth value d of the depth image acquired by the vision sensor at (u,v).

[0052] 4) Using the intrinsic parameters of the vision sensor, the pixel coordinates (u,v,d) are converted into a three-dimensional point P_cam in the vision sensor coordinate system.

[0053] 5) Using the pre-defined tf transformation relationship (referring to the transformation relationship from the vision sensor coordinate system to the robot body coordinate system to the global map coordinate system), P_cam is transformed to the global map coordinate system to obtain P_map. This achieves data fusion (i.e., semantic map) between the RGB image and depth image acquired by the vision sensor and the two-dimensional grid map acquired by the 3D LiDAR.

[0054] 6) Semantic map representation: Maintain a dynamic list SemanticObjects, where each element contains: {object_label, map_coordinates, detection_confidence, timestamp}, which are the label and category of the semantic information, the global coordinates of the semantic information, the confidence of the semantic information, and the timestamp of the discovery of the semantic information, respectively.

[0055] S2, Data Processing and Interaction with Large Language Models

[0056] Data processing and interaction with large language models are the key bridges to achieving intelligent decision-making. Through the encoding and conversion of fused data, large language models can understand environmental semantics and perform reasoning.

[0057] S21. Data encoding and conversion: Convert the semantic map into structured text; at the same time, integrate the natural language task instructions input by the user, such as "Find the first aid kit in the meeting room", with the structured text into a unified format, and use pre-trained language models such as BERT for feature extraction to generate input vectors acceptable to the large language model.

[0058] S22. Semantic relevance calculation: Input the input vector into a large language model (such as GPT series, Qianwen series, OpenAI series or a lightweight model trained independently), and use the attention mechanism of the internal Transformer architecture of the large language model to calculate the relevance between the task instruction and each semantic point in the semantic map. For example, for the "first aid kit" target, the model traverses all semantic points in the map, assigns higher relevance scores to the areas where objects such as "cabinet" and "table" that may store the first aid kit are located, and outputs a list of semantic points sorted by relevance, providing a quantitative basis for exploration and decision-making.

[0059] The prompt template is as follows:

[0060] You are an AI assistant for a robot exploring an unknown indoor environment. Your task is to select the best frontiers for the next exploration based on high-level commands and current sensor data. Current task: "{task_description}" Semantic objects detected in the environment: {semantic_list}, according to the task, which semantic objects are most relevant to the task? If there are any, please sort the objects in descending order of relevance. If there are no objects relevant to the task, please output None. Provide your answer in structured JSON format: {"Reasoning": "Your step-by-step thinking process is here.", "most_relevant_object_ids": <the id and relevance of the most relevant object, if none, then -1>}}.

[0061] Where: {task_description} is the natural language instruction input by the user, such as "Please help me find a bottle". {semantic_list} is a formatted list of SemanticObjects.

[0062] S3. Path planning and exploration decision-making

[0063] Path planning and exploration decision-making are the core of a robot's autonomous exploration, ensuring efficient task completion by dynamically adjusting exploration strategies. Based on the output of the large language model in S2, the robot chooses between semantic-guided exploration and geometric frontier exploration. The task ends when the task is completed (e.g., the target object is found) or all areas have been explored.

[0064] Semantic-guided exploration: When the relevance of semantic points output by the large language model exceeds a set threshold, a leading edge point is searched within a 3×3 grid area surrounding the corresponding semantic point, and its priority is elevated to the highest globally. An improved A* algorithm, combined with geometric information from the grid map, is used to plan the optimal path to the priority leading edge point, guiding the robot to prioritize the exploration of task-related areas.

[0065] Geometric Frontier Exploration: If the relevance of all semantic points is below a threshold, switch to the traditional geometric frontier exploration mode. Based on a two-dimensional grid map from LiDAR and exploration priorities, a boundary extraction algorithm is used to identify global frontier points, and the D*Lite algorithm is used for incremental path planning to respond to environmental changes in real time.

[0066] During the exploration process, every time the robot moves a fixed distance (e.g., 0.5 meters) or discovers new semantic information, it triggers the semantic map update and large language model recalculation process, dynamically adjusting the exploration strategy to ensure the timeliness and accuracy of decision-making.

[0067] The contents not described in detail in this specification are common knowledge to those skilled in the art.

[0068] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A robot autonomous exploration method based on multimodal data fusion and a large language model, characterized in that, include: S1. By working in collaboration with a 3D lidar and a visual sensor, the geometric information of the lidar and the semantic information of the vision are deeply integrated to construct an accurate semantic map; S2. Encode and convert the semantic map so that the large language model can understand and perform reasoning; S3. Based on the output of the large language model in S2, select semantic-guided exploration or geometric frontier exploration until the task is completed or all regions are explored.

2. The robot autonomous exploration method according to claim 1, characterized in that, In S1, a 3D lidar acquires point cloud data, subscribes to the point cloud data using the SLAM method, and publishes a 2D raster map; a visual sensor acquires image data, performs target detection using a target detection model, provides the confidence score of the corresponding target, performs semantic annotation on the target detection using a semantic segmentation model, and fuses the 2D raster map and the semantic annotation results.

3. The robot autonomous exploration method according to claim 2, characterized in that, In S1, frontier points are extracted from the two-dimensional grid map, and all frontier points are clustered to form several frontier clusters. For the centroid of each frontier cluster, the basic priority is calculated, and its exploration priority is evaluated.

4. The robot autonomous exploration method according to claim 2, characterized in that, The target detection model adopts the lightweight open-scene algorithm YOLOE, and utilizes the improved EfficientRep backbone network and SimOTA label allocation strategy of YOLOE algorithm to achieve accurate detection of multiple targets while ensuring real-time performance. For long-tailed or rare targets, the target detection model switches to the Owl-Vit model or Grounding dino model, and enhances the detection capability of unknown targets through Transformer architecture and text guidance mechanism.

5. The robot autonomous exploration method according to claim 2, characterized in that, The fusion of 2D raster maps and semantic annotation results includes: 1) For each object instance segmented by the semantic segmentation model, obtain its pixel mask; 2) Calculate the center pixel coordinates (u,v) of the pixel mask; 3) Query the depth value d of the depth image acquired by the vision sensor at (u,v); 4) Using the intrinsic parameters of the vision sensor, the pixel coordinates (u,v,d) are converted into a three-dimensional point P_cam in the vision sensor coordinate system; 5) Using the pre-defined tf transformation relationship, P_cam is transformed to the global map coordinate system to obtain P_map; the tf transformation relationship refers to the transformation relationship from the vision sensor coordinate system to the robot body coordinate system to the global map coordinate system; 6) Semantic map representation: Maintain a dynamic list SemanticObjects, where each element contains: {object_label, map_coordinates, detection_confidence, timestamp}, which are the label and category of the semantic information, the global coordinates of the semantic information, the confidence of the semantic information, and the timestamp of the discovery of the semantic information, respectively.

6. The robot autonomous exploration method according to claim 1, characterized in that, In S2, the encoding and transformation of the semantic map includes: converting the semantic map into structured text; simultaneously integrating the user's input natural language task instructions with the structured text into a unified format, and then performing feature extraction to generate an input vector acceptable to the large language model.

7. The robot autonomous exploration method according to claim 1, characterized in that, S2 also includes semantic relevance calculation, specifically: inputting the input vector into the large language model, using the attention mechanism of the Transformer architecture inside the large language model, calculating the relevance between the task instruction and each semantic point in the semantic map, and outputting a list of semantic points sorted by relevance, providing a quantitative basis for exploration decision-making.

8. The robot autonomous exploration method according to claim 1, characterized in that, Semantic-guided exploration refers to the following: when the relevance of semantic points output by the large language model is higher than a set threshold, frontier points are searched in the vicinity of the corresponding semantic point and their priority is raised to the highest globally; an improved A* algorithm is used, combined with the geometric information of the grid map, to plan the optimal path to the priority frontier points, guiding the robot to explore the task-related areas first.

9. The robot autonomous exploration method according to claim 1, characterized in that, Geometric frontier exploration refers to the following: if the relevance of all semantic points is below a threshold, switch to the traditional geometric frontier exploration mode. Based on the two-dimensional grid map of LiDAR and the exploration priority, use the boundary extraction algorithm to identify global frontier points, and use the D*Lite algorithm for incremental path planning to respond to environmental changes in real time.

10. The robot autonomous exploration method according to claim 1, characterized in that, During the exploration process, every time the robot moves a fixed distance or discovers new semantic information, it triggers the semantic map update and large language model recalculation process, dynamically adjusting the exploration strategy to ensure the timeliness and accuracy of decision-making.

Images