An autonomous exploration method based on multi-modal data fusion and large language model

Abstract

The application discloses an autonomous exploration method based on multi-modal data fusion and a large language model, solves the problems of low efficiency, insufficient decision intelligence and poor environmental adaptability in the prior art, and belongs to the technical field of robots. The autonomous exploration method comprises the following steps: S1, three-dimensional laser radar and a vision sensor are cooperatively used to deeply fuse laser radar geometric information and vision semantic information, and an accurate semantic map is constructed; S2, the semantic map is encoded and converted, so that a large language model can understand and perform reasoning; and S3, based on the output result of the large language model in S2, semantic-oriented exploration or geometric front exploration is selected until a task is completed or all regions are explored. The application realizes accurate and detailed environmental perception, improves the intelligence and task relevance of exploration decision, and has wide application potential.

Description

Technical Field

[0001] This invention relates to an autonomous exploration method based on multimodal data fusion and a large language model, and more particularly to an autonomous exploration method for robots based on multimodal data fusion and a large language model, belonging to the field of robotics technology. Background Technology

[0002] With the increasing demand for autonomous navigation, exploration, and operation of robots in complex environments, robot perception and navigation technologies have become a research hotspot. Currently, autonomous robot exploration generally relies on LiDAR and visual sensors to acquire environmental information in order to achieve map building and path planning.

[0003] Exploration technologies based on lidar mapping, such as GBPlanner and TARE, can construct two-dimensional or three-dimensional grid maps of the environment due to the high-precision ranging performance of lidar. They enable robots to locate and avoid obstacles through geometric structural information. However, they only contain geometric information and lack semantic understanding of objects in the environment, making it difficult to support high-level intelligent decision-making and task-oriented exploration.

[0004] Pure vision-based methods utilize cameras to capture rich image information and employ algorithms such as visual semantic segmentation and object detection to obtain semantic information about the environment, compensating for the semantic understanding deficiencies of LiDAR. However, visual sensors are sensitive to ambient lighting conditions. Under conditions of strong light, backlight, reflection, or low illumination at night, the quality of visual information deteriorates significantly, severely limiting system performance and affecting the stability of navigation and obstacle avoidance. Summary of the Invention

[0005] The technical problem to be solved by this invention is to overcome the shortcomings of the prior art and solve the problems of low efficiency of autonomous exploration, insufficient decision-making intelligence and poor environmental adaptability in the prior art.

[0006] The objective of this invention is achieved through the following technical solutions:

[0007] A robot autonomous exploration method based on multimodal data fusion and a large language model includes:

[0008] S1. By working in collaboration with a 3D lidar and a visual sensor, the geometric information of the lidar and the semantic information of the vision are deeply integrated to construct an accurate semantic map;

[0009] S2. Encode and convert the semantic map so that the large language model can understand and perform reasoning;

[0010] S3. Based on the output of the large language model in S2, select semantic-guided exploration or geometric frontier exploration until the task is completed or all regions are explored.

[0011] In one embodiment of the present invention, in S1, a three-dimensional laser acquires point cloud data, subscribes to the point cloud data using the SLAM method, and publishes a two-dimensional grid map; a visual sensor acquires image data, performs target detection using a target detection model, and provides the confidence level of the corresponding target; a semantic segmentation model performs semantic annotation on the target detection; and the two-dimensional grid map and the semantic annotation results are fused.

[0012] In one embodiment of the present invention, in S1, frontier points are extracted from a two-dimensional grid map, all frontier points are clustered to form several frontier clusters, and the basic priority is calculated for the centroid of each frontier cluster to evaluate and calculate its exploration priority.

[0013] In one embodiment of the present invention, the target detection model adopts the lightweight open-scene algorithm YOLOE, and utilizes the improved EfficientRep backbone network and SimOTA label allocation strategy of YOLOE algorithm to achieve accurate detection of multiple targets while ensuring real-time performance. For long-tailed or rare targets, the target detection model switches to the Owl-Vit model or the Grounding dino model, and enhances the detection capability of unknown targets through the Transformer architecture and text guidance mechanism.

[0014] In one embodiment of the present invention, the fusion of two-dimensional raster maps and semantic annotation results includes:

[0015] 1) For each object instance segmented by the semantic segmentation model, obtain its pixel mask;

[0016] 2) Calculate the center pixel coordinates (u,v) of the pixel mask;

[0017] 3) Query the depth value d of the depth image acquired by the vision sensor at (u,v);

[0018] 4) Using the intrinsic parameters of the vision sensor, the pixel coordinates (u,v,d) are converted into a three-dimensional point P_cam in the vision sensor coordinate system;

[0019] 5) Using the pre-defined tf transformation relationship, P_cam is transformed to the global map coordinate system to obtain P_map; the tf transformation relationship refers to the transformation relationship from the vision sensor coordinate system to the robot body coordinate system to the global map coordinate system;

[0020] 6) Semantic map representation: Maintain a dynamic list SemanticObjects, where each element contains: {object_label, map_coordinates, detection_confidence, timestamp}, which are the label and category of the semantic information, the global coordinates of the semantic information, the confidence of the semantic information, and the timestamp of the discovery of the semantic information, respectively.

[0021] In one embodiment of the present invention, S2, the encoding and conversion of the semantic map includes: converting the semantic map into structured text; simultaneously integrating the user-input natural language task instructions with the structured text into a unified format, and then performing feature extraction to generate an input vector acceptable to a large language model.

[0022] In one embodiment of the present invention, S2 further includes semantic relevance calculation, specifically: inputting the input vector into the large language model, using the attention mechanism of the Transformer architecture inside the large language model, calculating the relevance between the task instruction and each semantic point in the semantic map, and outputting a list of semantic points sorted by relevance, providing a quantitative basis for exploration decision-making.

[0023] In one embodiment of the present invention, semantic-guided exploration refers to: when the relevance of the semantic points output by the large language model is higher than a set threshold, searching for leading points in the vicinity of the corresponding semantic points and raising their priority to the highest globally; using an improved A* algorithm, combined with the geometric information of the grid map, planning the optimal path to the priority leading points, and guiding the robot to prioritize the exploration of areas related to the task.

[0024] In one embodiment of the present invention, geometric front exploration refers to: if the relevance of all semantic points is lower than a threshold, switching to the traditional geometric front exploration mode, using a boundary extraction algorithm to identify global front points based on the two-dimensional grid map of LiDAR and exploration priority, and using the D*Lite algorithm for incremental path planning to respond to environmental changes in real time.

[0025] In one embodiment of the present invention, during the exploration process, the robot triggers the semantic map update and large language model recalculation process every time it moves a fixed distance or discovers new semantic information, dynamically adjusting the exploration strategy to ensure the timeliness and accuracy of decision-making.

[0026] Compared with the prior art, the present invention has the following advantages:

[0027] (1) This invention achieves accurate and detailed environmental perception: by deeply integrating LiDAR and visual semantic information, the limitations of a single sensor are effectively overcome. LiDAR ensures the geometric accuracy of map construction and can accurately depict information such as the distribution of obstacles and spatial dimensions in the environment; the visual sensor gives the robot the ability to understand environmental semantics and can identify various objects and area functions; the combination of the two enables the robot to form a more comprehensive and detailed environmental representation, whether it is the furniture layout and functional zoning in complex indoor scenes or the identification of obstacles and signs in outdoor environments, it can accurately grasp the situation and provide a reliable basis for subsequent exploration.

[0028] (2) This invention improves the intelligence and task relevance of exploration decision-making: It fully leverages the powerful semantic understanding and reasoning capabilities of the large language model to perform deep semantic matching analysis on the fused semantic map and task objectives. When performing exploration tasks, the robot can prioritize exploring environmental areas highly relevant to the task objectives based on the calculation results of the large language model. For example, in a rescue mission to search for trapped personnel, it can quickly locate semantic regions related to "people" and prioritize their exploration, avoiding blind searching, significantly improving exploration efficiency and task completion rate, and making the robot's exploration decisions more targeted and intelligent.

[0029] (3) This invention has broad application potential: With its excellent environmental adaptability and intelligent exploration performance, this method can be widely applied in many fields such as rescue missions and intelligent inspections. In rescue missions, it helps to quickly locate trapped personnel; in intelligent inspections, it ensures a comprehensive check of equipment status. This will powerfully promote robots towards a higher level of intelligent autonomous exploration, expanding their boundaries and value in practical applications.

[0030] (4) The present invention overcomes the problems of existing technologies, such as the ability of lidar-based mapping exploration to obtain only geometric structure information of the environment and lack of environmental semantic understanding; the great influence of lighting on pure visual semantic navigation and poor adaptability; and the lack of multimodal data fusion, making it difficult to form high-level semantic decision support. Attached Figure Description

[0031] Figure 1 This is an architecture diagram of a robot autonomous exploration method based on multimodal data fusion and a large language model.

[0032] Figure 2 This is a flowchart of a robot autonomous exploration method based on multimodal data fusion and a large language model.

[0033] Figure 3 Flowchart for building semantic maps. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0035] A robot autonomous exploration method based on multimodal data fusion and a large language model (LLM) is proposed. This method integrates LiDAR geometric information and visual semantic information, employing an exploration framework with high-level decision-making by a LLM. The framework utilizes LiDAR mapping algorithms to construct a two-dimensional grid map and extracts frontiers as potential exploration targets. Simultaneously, a visual semantic segmentation network identifies key objects in images acquired by a visual sensor, obtaining visual semantic information. This visual semantic information is then fused with the two-dimensional grid map to form a semantic map. The LLM parses the user-defined natural language task and matches it with the visual semantic information. If a highly relevant semantic object exists, the frontiers near that object in the semantic map are prioritized for exploration; otherwise, an information gain-based frontier exploration strategy in the two-dimensional grid map is adopted. The architecture of the robot autonomous exploration method based on multimodal data fusion and a large language model is as follows: Figure 1 As shown, the process is as follows: Figure 2 As shown.

[0036] A robot autonomous exploration method based on multimodal data fusion and a large language model, specifically including:

[0037] S1. Multimodal data acquisition and fusion mapping

[0038] Multimodal data acquisition and fusion mapping are the core foundation for realizing robot environmental cognition. By working collaboratively with 3D LiDAR and visual sensors, and deeply fusing geometric information from the LiDAR with semantic information from the vision, an accurate semantic map of the environment is constructed. The mapping process is as follows: Figure 3 As shown.

[0039] S11. Sensor Data Acquisition: The robot is equipped with a 3D LiDAR and a vision sensor (e.g., a high-resolution RGB-D camera). The 3D LiDAR scans the environment 360° at high frequency to acquire point cloud data containing geometric information such as obstacle distances and angles. The density of the point cloud data is reduced through a VoxelGrid filtering algorithm, reducing computational load while maintaining accuracy. The vision sensor is equipped with an automatic exposure and low-light enhancement module, combined with a wide-angle lens to achieve a large field of view coverage, acquiring environmental image sequences at a frequency of 30 frames per second to ensure the real-time performance and integrity of the data.

[0040] S12. Point Cloud Data Processing: SLAM methods (such as GMAPping, PointLio, FastLio, etc.) are used to subscribe to point cloud data acquired by 3D LiDAR and publish a 2D raster map ( / map). The value of a map cell is represented as -1 (unknown), 0 (free), or 100 (occupied). SLAM methods are computationally efficient, suitable for mapping in 2D / 3D environments, provide a basic geometric map for frontline point detection, and offer a unified global map coordinate system.

[0041] S13. Frontier Extraction and Evaluation: Frontiers are a core concept in autonomous robot exploration, defined as boundary points between known free space and unknown space. They represent locations the robot can move to, extending the boundaries of the known area, and maximizing the acquisition of new information by reaching these points. A grid cell is extracted as a frontier if its state is unknown and it has at least one adjacent grid cell that is occupied / free. However, since the extracted frontiers are usually numerous and dense, DBSCAN (density-based clustering algorithm) is used to cluster all frontiers into several frontier clusters. For each frontier cluster, its centroid p is calculated. f This serves as the representative front point of the cluster.

[0042] For the centroid p of each front cluster f Basic priority calculation (when there is no semantic information) is required, and the exploration priority score needs to be evaluated. base .

[0043] Score base (p f ) = w i InfoGain(p f )-w c Cost(probot,p f )

[0044] Information gain refers to the percentage of information received when the evaluation reaches p. f The size of the unknown region that may be observed later; its simplified calculation method is to statistically analyze p. f The number of unknown cells within a certain radius (e.g., 5 meters).

[0045] The movement cost (Cost) refers to the cost of moving from the robot's current position to position p. f The navigation path length (which can be estimated using the A* algorithm), and probot represents the energy consumption.

[0046] w i and w c It is a weighting factor used to balance exploration gains and movement costs.

[0047] S14. RGB Image Processing from Visual Sensor: Image data acquired by the visual sensor is input into the object detection model and the semantic segmentation model. The object detection model adopts the lightweight open-scene algorithm YOLOE, utilizing an improved EfficientRep backbone network and SimOTA label allocation strategy to achieve accurate detection of various targets while ensuring real-time performance. For long-tailed or rare targets, the object detection model switches to the Owl-Vit model or Groundingdino model, enhancing the detection capability of unknown targets through the Transformer architecture and text guidance mechanism. Based on object detection, the object detection model provides the confidence score for each target. The semantic segmentation model introduces the SegmentAnything model. The powerful zero-shot segmentation capability of the SegmentAnything model allows for pixel-level semantic annotation of the detection results output by the object detection model.

[0048] S15, Data Fusion Mapping:

[0049] 1) For each object instance segmented by the semantic segmentation model, obtain its pixel mask.

[0050] 2) Calculate the center pixel coordinates (u,v) of the pixel mask.

[0051] 3) Query the depth value d of the depth image acquired by the vision sensor at (u,v).

[0052] 4) Using the intrinsic parameters of the vision sensor, the pixel coordinates (u,v,d) are converted into a three-dimensional point P_cam in the vision sensor coordinate system.

[0053] 5) Using the pre-defined tf transformation relationship (referring to the transformation relationship from the vision sensor coordinate system to the robot body coordinate system to the global map coordinate system), P_cam is transformed to the global map coordinate system to obtain P_map. This achieves data fusion (i.e., semantic map) between the RGB image and depth image acquired by the vision sensor and the two-dimensional grid map acquired by the 3D LiDAR.

[0054] 6) Semantic map representation: Maintain a dynamic list SemanticObjects, where each element contains: {object_label, map_coordinates, detection_confidence, timestamp}, which are the label and category of the semantic information, the global coordinates of the semantic information, the confidence of the semantic information, and the timestamp of the discovery of the semantic information, respectively.

[0055] S2, Data Processing and Interaction with Large Language Models

[0056] Interaction between data processing and large language models is a key bridge to realize intelligent decision-making. Through encoding and conversion of fused data, large language models can understand environmental semantics and perform reasoning.

[0057] S21. Data encoding conversion: convert a semantic map into structured text; meanwhile integrate a natural language task instruction input by a user, such as "find the first-aid kit in the meeting room", with the structured text into a unified format, adopt a pre-trained language model such as BERT to perform feature extraction, and generate an input vector acceptable to a large language model.

[0058] S22. Semantic relevance calculation: input the input vector into a large language model (such as GPT series, Qwen series, OpenAI series or self-trained lightweight model), use the attention mechanism of the internal Transformer architecture of the large language model to calculate and obtain the relevance between the task instruction and each semantic point in the semantic map. For example, for the target "first-aid kit", the model traverses all semantic points in the map, assigns higher relevance scores to regions where objects that may store the first-aid kit such as "cabinet" and "table" are located, and outputs a list of semantic points sorted by relevance, providing a quantitative basis for exploration decision-making.

[0059] The prompt template is as follows:

[0060] You are an AI assistant for a robot exploring an unknown indoor environment. Your task is to select the best frontier for next exploration according to advanced commands and current sensor data. Current task: "{task_description}". Semantic objects detected in the environment: {semantic_list}. According to the task, which semantic objects are most relevant to the task? If yes, sort the objects in descending order of relevance; if there is no object related to the task, output None. Provide your answer in a structured JSON format: { "Inference": "Your step-by-step thinking process is here.", "most_relevant_object_ids": <IDs and relevance of the most relevant objects, -1 if there are none>}.

[0061] Wherein: {task_description} is a natural language instruction input by a user, such as "please help me find a bottle". {semantic_list} is a formatted list of SemanticObjects.

[0062] S3. Path planning and exploration decision-making

[0063] Path planning and exploration decision-making are the core of a robot's autonomous exploration, ensuring efficient task completion by dynamically adjusting exploration strategies. Based on the output of the large language model in S2, the robot chooses between semantic-guided exploration and geometric frontier exploration. The task ends when the task is completed (e.g., the target object is found) or all areas have been explored.

[0064] Semantic-guided exploration: When the relevance of semantic points output by the large language model exceeds a set threshold, a leading edge point is searched within a 3×3 grid area surrounding the corresponding semantic point, and its priority is elevated to the highest globally. An improved A* algorithm, combined with geometric information from the grid map, is used to plan the optimal path to the priority leading edge point, guiding the robot to prioritize the exploration of task-related areas.

[0065] Geometric Frontier Exploration: If the relevance of all semantic points is below a threshold, switch to the traditional geometric frontier exploration mode. Based on a two-dimensional grid map from LiDAR and exploration priorities, a boundary extraction algorithm is used to identify global frontier points, and the D*Lite algorithm is used for incremental path planning to respond to environmental changes in real time.

[0066] During the exploration process, every time the robot moves a fixed distance (e.g., 0.5 meters) or discovers new semantic information, it triggers the semantic map update and large language model recalculation process, dynamically adjusting the exploration strategy to ensure the timeliness and accuracy of decision-making.

[0067] The contents not described in detail in this specification are common knowledge to those skilled in the art.

[0068] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A robot autonomous exploration method based on multimodal data fusion and a large language model, characterized in that, include: S1. By working in collaboration with a 3D lidar and a visual sensor, the geometric information of the lidar and the semantic information of the vision are deeply integrated to construct an accurate semantic map; S2. Encode and convert the semantic map so that the large language model can understand and perform reasoning; Encoding and transforming semantic maps includes converting them into structured text; Simultaneously, the user-input natural language task instructions and structured text are integrated into a unified format, and then feature extraction is performed to generate input vectors acceptable to large language models. S2 also includes semantic relevance calculation, specifically: inputting the input vector into the large language model, using the attention mechanism of the Transformer architecture inside the large language model, calculating the relevance between the task instruction and each semantic point in the semantic map, and outputting a list of semantic points sorted by relevance, providing a quantitative basis for exploration decision-making; S3. Based on the output of the large language model in S2, select semantic-guided exploration or geometric frontier exploration until the task is completed or all regions are explored.

2. The robot autonomous exploration method according to claim 1, characterized in that, In S1, a 3D lidar acquires point cloud data, subscribes to the point cloud data using the SLAM method, and publishes a 2D raster map; a visual sensor acquires image data, performs target detection using a target detection model, provides the confidence score of the corresponding target, performs semantic annotation on the target detection using a semantic segmentation model, and fuses the 2D raster map and the semantic annotation results.

3. The robot autonomous exploration method according to claim 2, characterized in that, In S1, frontier points are extracted from the two-dimensional grid map, and all frontier points are clustered to form several frontier clusters. For the centroid of each frontier cluster, the basic priority is calculated, and its exploration priority is evaluated.

4. The robot autonomous exploration method according to claim 2, characterized in that, The target detection model adopts the lightweight open-scene algorithm YOLOE, and utilizes the improved EfficientRep backbone network and SimOTA label allocation strategy of YOLOE algorithm to achieve accurate detection of multiple targets while ensuring real-time performance. For long-tailed or rare targets, the target detection model switches to the Owl-Vit model or Grounding dino model, and enhances the detection capability of unknown targets through Transformer architecture and text guidance mechanism.

5. The robot autonomous exploration method according to claim 2, characterized in that, The fusion of 2D raster maps and semantic annotation results includes: 1) For each object instance segmented by the semantic segmentation model, obtain its pixel mask; 2) Calculate the center pixel coordinates (u,v) of the pixel mask; 3) Query the depth value d of the depth image acquired by the vision sensor at (u,v); 4) Using the intrinsic parameters of the vision sensor, the pixel coordinates (u,v,d) are converted into a three-dimensional point P_cam in the vision sensor coordinate system; 5) Using the pre-defined tf transformation relationship, P_cam is transformed to the global map coordinate system to obtain P_map; the tf transformation relationship refers to the transformation relationship from the vision sensor coordinate system to the robot body coordinate system to the global map coordinate system; 6) Semantic map representation: Maintain a dynamic list SemanticObjects, where each element contains: {object_label, map_coordinates, detection_confidence, timestamp}, which are the label and category of the semantic information, the global coordinates of the semantic information, the confidence of the semantic information, and the timestamp of the discovery of the semantic information, respectively.

6. The robot autonomous exploration method according to claim 1, characterized in that, Semantic-guided exploration refers to the following: when the relevance of semantic points output by the large language model is higher than a set threshold, frontier points are searched in the vicinity of the corresponding semantic point and their priority is raised to the highest globally; an improved A* algorithm is used, combined with the geometric information of the grid map, to plan the optimal path to the priority frontier points, guiding the robot to explore the task-related areas first.

7. The robot autonomous exploration method according to claim 1, characterized in that, Geometric frontier exploration refers to the following: if the relevance of all semantic points is below a threshold, switch to the traditional geometric frontier exploration mode. Based on the two-dimensional grid map of LiDAR and the exploration priority, use the boundary extraction algorithm to identify global frontier points, and use the D*Lite algorithm for incremental path planning to respond to environmental changes in real time.

8. The robot autonomous exploration method according to claim 1, characterized in that, During the exploration process, every time the robot moves a fixed distance or discovers new semantic information, it triggers the semantic map update and large language model recalculation process, dynamically adjusting the exploration strategy to ensure the timeliness and accuracy of decision-making.

Images