An autonomous semantic mapping and navigation method and system for mobile robots

Abstract

The application belongs to the technical field of robot autonomous map construction and semantic target navigation, and particularly relates to a kind of autonomous semantic mapping navigation method and system of mobile robot, the method and system rely on asynchronous multithreading architecture, collaborative operation VLM decision thread, 3DGS SLAM navigation thread, combine 3DGS map and vector database to constitute double external memory, method is divided into two major stages of active exploration mapping, multimodal navigation query.The application realizes that environment geometric information and semantic information are completely decoupled, supports robot active semantic exploration, lifelong map iteration, is compatible with multimodal query, and navigation robustness is strong, and can be widely applied to service robot, industrial inspection, warehousing logistics, augmented reality tour and other fields.

Description

Technical Field

[0001] This invention belongs to the field of autonomous map building and semantic target navigation technology for robots. Specifically, it relates to an autonomous semantic mapping and navigation method and system for mobile robots, which deeply integrates 3D Gaussian Splatting (3DGS) mapping, Vision-Language Model (VLM) unified decision-making, and semantic memory. Background Technology

[0002] In recent years, with the rapid development of applications such as service robots and augmented reality, the ability for robots to understand semantics in complex environments and navigate using natural language has become an urgent need. Existing technologies mainly follow these approaches: 1) Embed semantic features into neural radiation fields (such as LERF, LangSplat). This type of method directly trains the semantic features of models such as CLIP into NeRF or 3D Gaussian fields, enabling the map itself to have semantic query capabilities. During a query, the target is located by rendering a semantic heatmap. Its main drawbacks are: tight coupling between semantics and geometry, requiring map retraining or fine-tuning to update semantics; inability to preserve first-person observation context; and difficulty in supporting abstract descriptive queries.

[0003] 2) Construct explicit semantic maps (such as VLMap, CLIP-Fields) This type of method first establishes a point cloud or raster map using traditional SLAM, and then backprojects and aggregates visual features into a 3D mesh to form semantic landmarks. Its drawbacks are: the feature aggregation process loses perspective information and descriptive richness; map updates also require re-aggregation; and it cannot handle semantic ambiguity from multiple perspectives at a single location.

[0004] 3) Structured semantic scene graphs (such as ConceptGraphs) This type of method abstracts the scene into a graph structure composed of object nodes and spatial relationship edges, and realizes navigation through graph queries. Its drawbacks are: it heavily relies on the accuracy of detection and segmentation, the construction and maintenance of the scene graph is fragile and computationally expensive; and it is difficult to express abstract and ambiguous overall scene semantics.

[0005] 4) Language model-driven zero-shot navigation (e.g., LLM-Grounder) This type of method uses a large language model to ground instructions to the current observations online, without requiring a persistent map. Its drawback is that it lacks memory, starting from scratch for each navigation and unable to reuse historical knowledge.

[0006] In addition, all of the above solutions share a common drawback: the robot's exploration path is preset or determined by traditional coverage planning, which is completely separated from the semantic understanding process, resulting in low mapping efficiency, redundant or missing semantic information, and an inability to form proactive intelligence that "observes with a purpose". Summary of the Invention

[0007] To address the aforementioned technical problems, this invention provides an autonomous semantic mapping and navigation method and system for mobile robots. It utilizes a Visual Language Model (VLM) as a unified brain, simultaneously undertaking tasks such as scene understanding, semantic memory, and proactive exploration decision-making. Three-dimensional Gaussian sputtering (3DGS) serves as geometric appearance memory, and a vector database serves as semantic experience memory, forming a closed-loop, organic intelligent agent system. This system, driven by natural language, proactively and efficiently explores unknown environments, constructs a persistent, cross-modal retrieveable environmental memory, and supports a unified framework for traditional coordinate navigation, natural language navigation, and image navigation.

[0008] This invention is implemented as follows: It provides an autonomous semantic mapping and navigation method for mobile robots. Based on an asynchronous multi-threaded architecture, it collaboratively runs a VLM decision thread and a 3DGS SLAM navigation thread, and relies on a dual external memory storage module to complete environment mapping and multimodal navigation. The dual external memory storage module includes a 3DGS map for storing scene geometry and appearance information, and a vector database for storing semantic records. The mapping part of the mapping and navigation method specifically includes the following steps: S1. System initialization: Start the mobile robot, complete the initialization of the 3DGS SLAM navigation thread, and the vision sensor on the mobile robot collects the initial RGB image and initial pose of the environment, and sends them to the VLM decision thread and the 3DGS SLAM navigation thread respectively. S2, Actively explore and build a graph in a loop: S201, Perception: The VLM decision thread receives the current keyframe RGB image and real-time pose sent by the 3DGS SLAM navigation thread; S202, Recall: The VLM decision thread retrieves the historical semantic record with the highest matching degree from the vector database based on the current pose; S203, Thinking and Decision-Making: The VLM decision thread combines preset tasks, historical semantic records and the current keyframe RGB image to output structured decision content, which includes a natural language description of the scene, the intention of the next action, the target description and the basis for the decision; S204, Memory Writing: The VLM decision thread associates and combines real-time pose, timestamp, image features, scene natural language description and original scene natural language description to generate new semantic records and write them into the vector database. S205, Command Issuance: The VLM decision thread parses the next action intention into the target pose or movement command and sends it to the 3DGS SLAM navigation thread; S206. Motion Execution and Real-time Mapping: The 3DGS SLAM navigation thread receives target pose or movement commands from the VLM decision thread and controls the movement of the mobile robot; at the same time, it updates the globally consistent 3DGS map based on real-time image data collected by the vision sensor and continuously calculates the pose of the mobile robot. S207, Keyframe Feedback: The 3DGS SLAM navigation thread filters keyframes based on pose changes and visual overlap, and feeds back the keyframe RGB images and updated poses to the VLM decision thread. S208. Termination Judgment: The VLM decision thread determines whether the current area has been explored. If not, it returns to step S201 to continue the loop. If the exploration is completed, the active exploration and mapping phase ends. At this time, the 3DGS map and vector database complete the environmental data storage.

[0009] Preferably, the navigation component of the mapping and navigation method specifically includes the following steps: S3, Multimodal Navigation Query Execution: S301, Query Reception and Encoding: The mobile robot receives the query command input by the user and generates a unified query embedding vector through the VLM decision thread; S302. Similarity retrieval: Using the query embedding vector, perform similarity retrieval in the vector database and return the top K candidate semantic records with the highest similarity. Each candidate semantic record corresponds to the associated pose and scene natural language description text. S303, Viewpoint Verification: Based on the 3DGS map, render a virtual view at the pose corresponding to each candidate semantic record; input the virtual view and the user query command into the VLM decision thread, and the VLM decision thread outputs a matching score in the range of 0 to 1, and filters low-scoring candidate results based on the matching score; S304. Target Determination and Navigation: The 3DGS SLAM navigation thread selects the pose corresponding to the candidate semantic record with the highest matching score that exceeds a preset threshold as the final navigation target point. It plans a global path by combining the obstacle occupancy information extracted from the 3DGS map and controls the mobile robot to move along the planned path to the navigation target point to complete the navigation task.

[0010] Further preferably, the VLM decision thread is a low-frequency operating thread with an operating frequency of 1Hz; the 3DGS SLAM navigation thread is a high-frequency operating thread with an operating frequency of 30Hz.

[0011] Further preferred, the VLM decision thread deployment method includes two types: the first is cloud deployment, where the 3DGS SLAM navigation thread is deployed on the mobile robot's onboard computing unit, and the VLM decision thread and vector database are deployed on a cloud server, with the VLM decision thread and 3DGS SLAM navigation thread transmitting data through a wireless communication module; the second is fully offline deployment, where the quantized VLM decision thread, vector database, and 3DGS SLAM navigation thread are all deployed on the mobile robot's onboard computing unit.

[0012] Further preferably, in step 301, the query instruction is in the form of natural language text, example image, or a mixture of natural language text and example image.

[0013] Further preferred, in step 303, if the matching scores of all candidate semantic records are lower than a preset threshold, the VLM decision thread requests supplementary query information from the user or prompts the user to re-explore the environment.

[0014] The present invention also provides an autonomous semantic mapping and navigation system for mobile robots. Based on the above-mentioned autonomous semantic mapping and navigation method, the system includes a VLM decision thread, a 3DGS SLAM navigation thread, and a dual external memory storage module. The three modules work together in an asynchronous multi-threaded architecture. The VLM decision thread is used to receive images and poses, retrieve historical semantic records from the vector database, generate structured decisions by combining task and observation information, complete the writing of semantic records, parse and issue movement commands, and at the same time complete multimodal query encoding and viewpoint matching verification. The 3DGS SLAM navigation thread is used to receive movement commands and complete path planning, obstacle avoidance and robot motion control, build and update 3DGS maps in real time, calculate pose, filter key frames and feed data back to the VLM decision thread module; The dual external memory storage module includes a 3DGS map and a vector database; the 3DGS map is used to store Gaussian point cloud data of the scene to realize geometric positioning, virtual view rendering and path occupancy information extraction; the vector database is used to store semantic records with temporal sequence, pose, image feature embedding, text feature embedding and original text, and supports cross-modal similarity retrieval.

[0015] Compared with the prior art, the advantages of the present invention are as follows: 1. Complete decoupling of geometry and semantics, with semantics capable of independent evolution: 3DGS maps only concern themselves with geometric appearance, while all semantic memories are stored in a vector database. Replacing or upgrading the VLM model, or adding new language descriptions, requires no map reconstruction or fine-tuning, achieving true lifelong learning; 2. Unified intelligent agent actively explores, improving both mapping efficiency and semantic quality: The VLM decision thread controls the mapping process with semantic curiosity or task objectives, actively eliminating semantic blind spots and verifying hypotheses, avoiding information redundancy or missing information in traditional passive mapping. 3. Retain first-person "appearance memory" and support richer semantic queries: Store memory in a way that records "pose-description" independently rather than aggregated features, and support natural language queries related to specified viewpoints; 4. Unified framework for multimodal query and cross-modal retrieval: The same database and the same retrieval mechanism support text, image and mixed queries, without the need to build separate indexes for each modality; 5. Lightweight edge + cloud intelligence, flexible deployment: The edge only needs to run the real-time 3DGS SLAM navigation thread, while the VLM decision thread and vector retrieval can be completed in the cloud or on the edge server, with low bandwidth requirements; 6. Closed-loop verification improves navigation robustness: After retrieval, the new perspective rendering of the 3DGS SLAM navigation thread and the secondary verification of the VLM decision thread effectively reduce the risk of navigation failure caused by erroneous retrieval. Attached Figure Description

[0016] Figure 1 This is a diagram illustrating the overall architecture of the autonomous semantic mapping and navigation system of this invention. Figure 2 is a flowchart illustrating the active exploration and mapping part of the autonomous semantic mapping and navigation method of the present invention; Figure 3 is a flowchart illustrating the multimodal navigation query part of the autonomous semantic mapping and navigation method of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0018] refer to Figures 1-3 This invention provides an autonomous semantic mapping and navigation system for mobile robots, including a VLM decision thread, a 3DGS SLAM navigation thread, and a dual external memory storage module. The three modules work together in an asynchronous multi-threaded architecture. The VLM decision thread is used to receive images and poses, retrieve historical semantic records from the vector database, generate structured decisions by combining task and observation information, complete the writing of semantic records, parse and issue movement commands, and at the same time complete multimodal query encoding and viewpoint matching verification. The 3DGS SLAM navigation thread is used to receive movement commands and complete path planning, obstacle avoidance and robot motion control, build and update 3DGS maps in real time, calculate pose, filter key frames and feed data back to the VLM decision thread module; The dual external memory storage module includes a 3DGS map and a vector database; the 3DGS map is used to store Gaussian point cloud data of the scene to realize geometric positioning, virtual view rendering and path occupancy information extraction; the vector database is used to store semantic records with temporal sequence, pose, image feature embedding, text feature embedding and original text, and supports cross-modal similarity retrieval.

[0019] The VLM decision thread deployment methods include two types: the first is cloud deployment, where the 3DGS SLAM navigation thread is deployed on the mobile robot's onboard computing unit, and the VLM decision thread and vector database are deployed on a cloud server, with the VLM decision thread and 3DGS SLAM navigation thread transmitting data through a wireless communication module; the second is fully offline deployment, where the quantized VLM decision thread, vector database, and 3DGS SLAM navigation thread are all deployed on the mobile robot's onboard computing unit.

[0020] Using the above-mentioned mapping and navigation system, the autonomous semantic mapping and navigation method for mobile robots specifically includes the following steps: S1. System initialization: Start the mobile robot, complete the initialization of the 3DGS SLAM navigation thread, and the vision sensor on the mobile robot collects the initial RGB image and initial pose of the environment, and sends them to the VLM decision thread and the 3DGS SLAM navigation thread respectively. S2, Actively explore and build a graph in a loop: S201, Perception: The VLM decision thread receives the current keyframe RGB image and real-time pose sent by the 3DGS SLAM navigation thread; S202, Recall: The VLM decision thread retrieves the historical semantic record with the highest matching degree from the vector database based on the current pose; S203, Thinking and Decision-Making: The VLM decision thread combines preset tasks, historical semantic records and the current keyframe RGB image to output structured decision content, which includes a natural language description of the scene, the intention of the next action, the target description and the basis for the decision; S204, Memory Writing: The VLM decision thread associates and combines real-time pose, timestamp, image features, scene natural language description and original scene natural language description to generate new semantic records and write them into the vector database. S205, Command Issuance: The VLM decision thread parses the next action intention into the target pose or movement command and sends it to the 3DGS SLAM navigation thread; S206. Motion Execution and Real-time Mapping: The 3DGS SLAM navigation thread receives target pose or movement commands from the VLM decision thread and controls the movement of the mobile robot; at the same time, it updates the globally consistent 3DGS map based on real-time image data collected by the vision sensor and continuously calculates the pose of the mobile robot. S207, Keyframe Feedback: The 3DGS SLAM navigation thread filters keyframes based on pose changes and visual overlap, and feeds back the keyframe RGB images and updated poses to the VLM decision thread. S208. Termination Judgment: The VLM decision thread determines whether the current area has been explored. If not, it returns to step S201 to continue the loop. If the exploration is completed, the active exploration and mapping phase ends. At this time, the 3DGS map and vector database complete the environmental data storage. S3, Multimodal Navigation Query Execution: S301, Query Reception and Encoding: The mobile robot receives the query command input by the user and generates a unified query embedding vector through the VLM decision thread; the query command is in the form of natural language text, example image, or a mixture of natural language text and example image; S302. Similarity retrieval: Using the query embedding vector, perform similarity retrieval in the vector database and return the top K candidate semantic records with the highest similarity. Each candidate semantic record corresponds to the associated pose and scene natural language description text. S303, Viewpoint Verification: Based on the 3DGS map, render a virtual view at the pose corresponding to each candidate semantic record; input the virtual view and the user query command into the VLM decision thread, and the VLM decision thread outputs a matching score in the range of 0 to 1, and filters low-scoring candidate results based on the matching score; if the matching score of all candidate semantic records is lower than a preset threshold, the VLM decision thread requests the user to supplement the query information, or prompts the user to re-explore the environment.

[0021] S304. Target Determination and Navigation: The 3DGS SLAM navigation thread selects the pose corresponding to the candidate semantic record with the highest matching score that exceeds a preset threshold as the final navigation target point. It plans a global path by combining the obstacle occupancy information extracted from the 3DGS map and controls the mobile robot to move along the planned path to the navigation target point to complete the navigation task.

[0022] The technical solution of this invention will be described in detail below with reference to a specific application scenario. This embodiment uses a wheeled service robot as the hardware carrier, and the application scenario is indoor office building environment mapping and target navigation.

[0023] 1. Hardware platform configuration This embodiment provides two hardware deployment schemes, which can be selected according to the usage scenario: Option 1 (Cloud-based Collaborative Deployment): The wheeled service robot is equipped with an RGB-D vision camera, an NVIDIA Jetson Orin onboard computing unit, and a wireless communication module; the onboard computing unit runs a 3DGS SLAM navigation thread; the VLM decision thread and vector database are deployed on a cloud server, and the wheeled service robot interacts with the cloud via a wireless network.

[0024] Option 2 (Fully Offline Deployment): The wheeled service robot is equipped with an RGB-D vision camera and a high-performance onboard computing unit; the quantized and compressed VLM decision thread, vector database, and 3DGS SLAM navigation thread are all deployed on the onboard computing unit, so that the robot can complete mapping and navigation independently without the network.

[0025] The robot adopts an asynchronous multi-threaded architecture. The VLM decision-making thread runs at a low frequency (1Hz), while the 3DGS SLAM navigation thread runs at a high frequency (30Hz). The two threads run independently and interact asynchronously.

[0026] 2. VLM Structured Decision Hint Configuration To ensure that the output of the VLM decision thread can be parsed by the system, standardized system prompts are pre-configured to guide the VLM decision thread to output structured decisions in a fixed JSON format. The prompts are as follows: "You are a robot's visual brain with proactive exploration capabilities. You will see images from the current perspective and a summary of historical memories. Your tasks are: 1) Describe the current scene in natural language, including as much as possible the spatial relationships between objects; 2) Determine whether the current area has been sufficiently explored; if there are semantically ambiguous objects or unobserved areas, generate the next exploration target; otherwise, suggest moving to other unexplored areas. Please strictly adhere to the following JSON format for output:" {"description":"Natural language description of the scene","intent":"explore"|"navigate_to"|"done","target_description":"Natural language description of the target, such as 'the red book on the table'","reasoning":"Brief explanation of the decision-making reason"} In task-driven mode, user task instructions are appended to the above prompts, and the VLM decision thread continuously judges the task completion status.

[0027] 3. Vector Database Construction and Retrieval Implementation (1) Data entry process: After each keyframe image is processed by the VLM decision thread, a single semantic record is generated and written to the vector database. A single semantic record includes: a 7-dimensional pose vector (3D coordinates xyz + quaternions), a 768-dimensional or higher image feature embedding, a text embedding of the same dimension as the image feature embedding, the original scene description text, and a timestamp. The image feature embedding is generated by the VLM visual encoder, and the text embedding is generated by the VLM text encoder encoding the scene description text.

[0028] (2) Similarity retrieval process: After receiving the user's multimodal query, the VLM decision thread calls the corresponding encoder to generate a unified query embedding vector according to the query type; the cosine similarity algorithm is used to retrieve the Top-K (K=5 in this embodiment) semantic records with the highest similarity in the vector database, and the camera pose and original scene description text corresponding to the record are returned.

[0029] 4. Implementation of Viewpoint Verification and Target Pose Selection After completing the vector database retrieval, initiate the viewpoint verification process to improve navigation accuracy. Specific steps include: The first step is to iterate through the 5 candidate semantic records, extract the camera pose corresponding to each record, and call the 3DGS map to render the corresponding virtual view at that pose. The second step involves inputting the virtual view and the user's original query command into the VLM decision thread. The VLM decision thread outputs a matching score of 0 to 1, with a preset matching threshold of 0.6. The third step is to filter candidate poses with a matching score greater than 0.6 and select the pose with the highest score as the final navigation target point. Fourth, if all candidate pose matching scores are below 0.6, the system will pop up a window to request additional query information from the user, or prompt the user to restart the environment exploration.

[0030] 5. Implementation of motion execution and safety verification After receiving the target pose from the VLM decision thread, the 3DGS SLAM navigation thread executes the motion control process: (1) Safety verification: First, the target pose is verified. If the pose is in an inaccessible location such as inside an obstacle or in a suspended area, the target pose is automatically corrected to the nearest accessible coordinate point. (2) Path planning: The occupied grid map is extracted in real time from the 3DGS Gaussian point cloud data, and the global path planning is completed by the A* algorithm; (3) Local obstacle avoidance and motion: The dynamic window method (DWA) is used to complete local dynamic obstacle avoidance, generate a smooth motion trajectory, and control the wheeled robot to move along the trajectory.

[0031] 6. Complete Scene Execution Example Application scenario: In an office building environment, the user issues the command "Complete the floor plan and locate the meeting room with a whiteboard." The complete workflow is as follows: Step 1: System Initialization and Initial Exploration The robot starts, completes 3DGS SLAM initialization, and the RGB-D camera acquires initial images and poses of the corridor and uploads them to the VLM decision thread. The VLM decision thread outputs a decision based on the prompt: {"description":"An indoor corridor with multiple wooden doors, flat floor, and walls on both sides","intent":"explore","target_description":"Go to the position 1 meter in front of the first door on the left side of the corridor","reasoning":"The current area has not been explored; prioritize exploring the rooms on both sides of the corridor"}. This semantic record is written to the vector database, and the VLM decision thread sends the target pose to the 3DGS SLAM navigation thread.

[0032] Step 2: Iterative Active Exploration and Map Building The robot moves to the first door, and the 3DGS SLAM navigation thread updates the 3DGS map in real time, filters keyframes, and feeds them back to the VLM decision thread. The VLM decision thread identifies the room scene and outputs a description: "The room contains a desk and bookshelves, and there are no whiteboards on the walls." The decision intent is "Return to the corridor and explore the next door," and the corresponding semantic record is stored in the vector database.

[0033] The robot explores the second door in turn and determines that there is still no whiteboard in the room; then it moves to the third door. The VLM decision thread recognizes the scene and outputs a description: "Large conference room, a large whiteboard is hanging on the wall, and a marker slot is set below the whiteboard". The decision intent is "continue to explore the remaining rooms to confirm whether there are other conference rooms with whiteboards". This key semantic record is stored in the vector database.

[0034] Step 3: Exploration terminated After the robot completes the exploration of all rooms on the floor, the VLM decision thread determines that the exploration of the entire area is complete and outputs the intent "done". The active exploration and mapping phase ends. At this time, the 3DGS map stores the geometric data of the entire floor environment, and the vector database stores the semantic records of all areas.

[0035] Step 4: Multimodal navigation query The user enters the text query: "Take me to the meeting room with the whiteboard." The system then executes the multimodal navigation process: 1. The VLM decision thread encodes the query text to generate a query embedding vector and retrieves the Top-5 semantic records from the vector database; 2. Extract candidate poses, render virtual views using 3DGS maps, and verify them through VLM decision thread matching. The third meeting room has a matching score of 0.92 (higher than the threshold of 0.6), and is therefore determined as the navigation target point. 3.3 The DGS SLAM navigation thread plans the path, avoids obstacles, and controls the robot's movement. Finally, the robot accurately arrives at the target conference room, and the navigation task is completed.

[0036] In addition to indoor office building scenarios, this invention can also be applied to industrial plant inspections, large-scale warehousing and logistics, shopping mall navigation, and outdoor park robots. For different scenarios, only the VLM decision thread prompts, path planning algorithm parameters, and matching score thresholds need to be adjusted; no modification to the overall system architecture and core methods is required, demonstrating strong adaptability. Furthermore, the geometrically and semantically decoupled architecture of this invention allows for direct upgrades of the VLM model in subsequent iterations without the need to rebuild the 3DGS map, resulting in a long lifespan.

Claims

1. A method for autonomous semantic mapping and navigation of a mobile robot, characterized in that, The VLM decision thread and 3DGS SLAM navigation thread operate collaboratively based on an asynchronous multi-threaded architecture, and environmental mapping and multimodal navigation are completed by relying on a dual external memory storage module. The dual external memory storage module includes a 3DGS map for storing scene geometry and appearance information, and a vector database for storing semantic records. The mapping part of the mapping and navigation method specifically includes the following steps: S1. System initialization: Start the mobile robot, complete the initialization of the 3DGS SLAM navigation thread, and the vision sensor on the mobile robot collects the initial RGB image and initial pose of the environment, and sends them to the VLM decision thread and the 3DGS SLAM navigation thread respectively. S2, Actively explore and build a graph in a loop: S201, Perception: The VLM decision thread receives the current keyframe RGB image and real-time pose sent by the 3DGS SLAM navigation thread; S202, Recall: The VLM decision thread retrieves the historical semantic record with the highest matching degree from the vector database based on the current pose; S203, Thinking and Decision-Making: The VLM decision thread combines preset tasks, historical semantic records and the current keyframe RGB image to output structured decision content, which includes a natural language description of the scene, the intention of the next action, the target description and the basis for the decision; S204, Memory Writing: The VLM decision thread associates and combines real-time pose, timestamp, image features, scene natural language description and original scene natural language description to generate new semantic records and write them into the vector database. S205, Command Issuance: The VLM decision thread parses the next action intention into the target pose or movement command and sends it to the 3DGS SLAM navigation thread; S206. Motion Execution and Real-time Mapping: The 3DGS SLAM navigation thread receives target pose or movement commands from the VLM decision thread and controls the movement of the mobile robot; at the same time, it updates the globally consistent 3DGS map based on real-time image data collected by the vision sensor and continuously calculates the pose of the mobile robot. S207, Keyframe Feedback: The 3DGS SLAM navigation thread filters keyframes based on pose changes and visual overlap, and feeds back the keyframe RGB images and updated poses to the VLM decision thread. S208. Termination Judgment: The VLM decision thread determines whether the current area has been explored. If not, it returns to step S201 to continue the loop. If the exploration is completed, the active exploration and mapping phase ends. At this time, the 3DGS map and vector database complete the environmental data storage.

2. The autonomous semantic mapping and navigation method for mobile robots according to claim 1, characterized in that, The navigation component of the mapping and navigation method specifically includes the following steps: S3, Multimodal navigation query execution: S301, Query Reception and Encoding: The mobile robot receives the query command input by the user and generates a unified query embedding vector through the VLM decision thread; S302. Similarity retrieval: Using the query embedding vector, perform similarity retrieval in the vector database and return the top K candidate semantic records with the highest similarity. Each candidate semantic record corresponds to the associated pose and scene natural language description text. S303, Viewpoint Verification: Based on the 3DGS map, render a virtual view at the pose corresponding to each candidate semantic record; input the virtual view and the user query command into the VLM decision thread, and the VLM decision thread outputs a matching score in the range of 0 to 1, and filters low-scoring candidate results based on the matching score; S304. Target Determination and Navigation: The 3DGS SLAM navigation thread selects the pose corresponding to the candidate semantic record with the highest matching score that exceeds a preset threshold as the final navigation target point. It plans a global path by combining the obstacle occupancy information extracted from the 3DGS map and controls the mobile robot to move along the planned path to the navigation target point to complete the navigation task.

3. The autonomous semantic mapping and navigation method for mobile robots according to claim 1 or 2, characterized in that, The VLM decision thread is a low-frequency operating thread with an operating frequency of 1Hz; the 3DGS SLAM navigation thread is a high-frequency operating thread with an operating frequency of 30Hz.

4. The autonomous semantic mapping and navigation method for a mobile robot according to claim 1 or 2, characterized in that, The VLM decision thread deployment methods include two types: the first is cloud deployment, where the 3DGS SLAM navigation thread is deployed on the mobile robot's onboard computing unit, and the VLM decision thread and vector database are deployed on a cloud server, with the VLM decision thread and 3DGS SLAM navigation thread transmitting data through a wireless communication module; the second is fully offline deployment, where the quantized VLM decision thread, vector database, and 3DGS SLAM navigation thread are all deployed on the mobile robot's onboard computing unit.

5. The autonomous semantic mapping and navigation method for mobile robots according to claim 2, characterized in that, In step 301, the query instruction is in the form of natural language text, example image, or a mixture of natural language text and example image.

6. The autonomous semantic mapping and navigation method for mobile robots according to claim 2, characterized in that, In step 303, if the matching scores of all candidate semantic records are lower than a preset threshold, the VLM decision thread requests supplementary query information from the user or prompts the user to re-explore the environment.

7. An autonomous semantic mapping and navigation system for a mobile robot, characterized in that, Based on the autonomous semantic mapping and navigation method according to any one of claims 1-6, the system includes a VLM decision thread, a 3DGS SLAM navigation thread, and a dual external memory storage module, and the three modules work together in an asynchronous multi-threaded architecture. The VLM decision thread is used to receive images and poses, retrieve historical semantic records from the vector database, generate structured decisions by combining task and observation information, complete the writing of semantic records, parse and issue movement commands, and at the same time complete multimodal query encoding and viewpoint matching verification. The 3DGS SLAM navigation thread is used to receive movement commands and complete path planning, obstacle avoidance and robot motion control, build and update 3DGS maps in real time, calculate pose, filter key frames and feed data back to the VLM decision thread module; The dual external memory storage module includes a 3DGS map and a vector database; the 3DGS map is used to store Gaussian point cloud data of the scene to realize geometric positioning, virtual view rendering and path occupancy information extraction; the vector database is used to store semantic records with temporal sequence, pose, image feature embedding, text feature embedding and original text, and supports cross-modal similarity retrieval.

Images