Robot navigation method based on plane layout diagram

Abstract

The invention discloses a robot navigation method based on a plane layout. The method comprises the following steps: S1, based on a building plane layout map, analyzing an image by using a multi-modal large model, extracting topological and geometric features in an environment, and constructing a semantic topological structure; s2, acquiring a navigation task instruction, and analyzing corresponding initial topological node and target topological node information; s3, inputting the semantic topological structure and the node information into a large language model, and generating a global path node sequence by utilizing the spatial reasoning capability of the large language model; s4, on the basis of real-time data acquired by the environment sensing equipment and prior information of the plane layout, the real-time pose of the robot is determined in a mode of fusing semantic observation matching and geometric observation matching; and S5, based on the node sequence and the real-time pose, planning a global geometric path, and using a local planner and a controller to drive the robot to move to a target point along the path. The method does not need to construct an SLAM map in advance, and is low in cost and good in flexibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of robot autonomous navigation and artificial intelligence, in particular to a method and system for understanding environment semantics by combining visual language large models and large language models, and realizing robot intelligent path planning, positioning and navigation based on building layout maps, and a computer readable storage medium. BACKGROUND

[0002] Currently, the autonomous navigation of indoor mobile robots mainly relies on pre-constructed environment maps, such as Simultaneous Localization and Mapping (SLAM) technology. This method constructs an environment map in real time and simultaneously locates by sensors (such as lidar, camera) mounted on the robot. Although SLAM technology is relatively mature, it has significant drawbacks: first, the mapping process needs to be carried out in the field in the deployment environment, which is time-consuming and labor-intensive for large-scale scenarios (such as office buildings, hospitals), and the deployment cost is high; second, the constructed map is usually a collection of geometric features (such as point cloud, grid), lacking high-level semantic information (such as "room function", "house number", "passage connection relationship"), making it difficult for the robot to understand and execute natural language instructions containing rich semantics (such as "please go to room 315 on the third floor of the east side") issued by humans.

[0003] In recent years, some research has attempted to use building layout maps for navigation, as layout maps contain rich prior spatial structure and semantic information. However, existing methods often rely on manual annotation or simple image processing rules in the information extraction stage, which is low in intelligence and cannot adapt to diverse drawing styles. At the path planning level, they lack deep understanding and reasoning ability for spatial relationships, and cannot handle complex cross-area navigation tasks.

[0004] Therefore, there is an urgent need in the art for a new paradigm of robot navigation that can automatically understand the prior environment information of layout maps and intelligently respond to high-level semantic instructions, to overcome the deficiencies of high deployment cost, poor flexibility and insufficient intelligence level in existing technologies. SUMMARY

[0005] The present application aims to overcome the deficiencies of the prior art and provide a robot navigation method that does not require pre-SLAM mapping and can directly understand the semantics of layout maps and execute natural language tasks.

[0006] In a first aspect, the present application provides a robot navigation method based on layout maps, comprising the following steps: 1. Environment semantic construction step: obtaining the building layout map of the target environment, using a multi-modal model to analyze the layout map, extracting topological nodes and geometric features in the environment, and constructing a semantic topology structure; 2. Navigation task parsing steps: Obtain navigation task instructions, parse the starting information and target information in the instructions, and determine the corresponding starting topology node and target topology node in the semantic topology structure; 3. Global path reasoning step: Input the semantic topology structure, the starting topology node, and the target topology node into the large language model, and use the spatial reasoning capability of the large language model to generate a global path node sequence from the starting topology node to the target topology node; 4. Navigation execution steps: Based on the real-time data obtained by the environmental perception device and the prior information of the planar layout, the robot's real-time pose is determined, and a geometric path is planned according to the global path node sequence and the real-time pose to control the robot to move to the target point.

[0007] Furthermore, in the environmental semantic construction step, the multimodal model parsing specifically includes: guiding the visual language large model to identify functional areas (such as rooms) and connecting channels (such as corridors) in the image through preset prompt words, and outputting structured text containing node types and approximate locations; then using image processing algorithms to perform edge calibration on the approximate locations and mapping the pixel coordinates to physical environment coordinates.

[0008] Furthermore, the positioning in the navigation execution step adopts multimodal fusion positioning: using semantic observation (such as door number text recognition) for coarse positioning and semantic matching, and using geometric observation (such as registration of laser point cloud and planar map features) for precise positioning.

[0009] In a second aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect.

[0010] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0011] The beneficial effects of this invention are as follows: 1. Zero deployment cost: It directly utilizes widely available building floor plans as prior knowledge, eliminating the need for on-site SLAM mapping and enabling the robot to be used "out of the box". 2. High-level semantic understanding: By combining a large visual language model and a large language model, the robot is equipped with human-like image reading and spatial reasoning abilities, and can accurately respond to complex natural language commands. 3. Accuracy and robustness: Through the fusion of semantic and geometric observations, and a hierarchical planning control architecture, positioning accuracy and navigation reliability are guaranteed even without SLAM maps. Attached Figure Description

[0012] Figure 1 This is a flowchart of a robot navigation method based on a planar layout diagram provided in an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of the parsing process of a planar layout diagram in one embodiment of the present invention, showing the semantic topology structure (taking JSON format as an example) generated by a multimodal large model and the corresponding topological connection relationship.

[0014] Figure 3 This is an example diagram of the prompt word construction and output results of a large language model for spatial path reasoning in one embodiment of the present invention.

[0015] Figure 4 This is a schematic diagram of a robot localization process based on the fusion of semantic observation and geometric observation in one embodiment of the present invention. Detailed Implementation

[0016] The implementation of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0017] Reference Figure 1 This embodiment demonstrates a complete robot navigation process based on a planar layout diagram.

[0018] S101, Environmental semantic construction steps: First, obtain the digital floor plan I of the target building (e.g., Figure 2 As shown in Figure A, the format can be JPG, PNG, or PDF. This image is input into a trained multimodal large model (e.g., GPT-4V, Qwen-VL, or Gemini Pro Vision). In this embodiment, the model is guided by specific prompts. The prompt strategy is as follows: "You are an architectural drawing analysis expert. Please identify all key areas (rooms, corridors, lobbies, elevators) in the input image and extract their names and pixel bounding boxes. Please output in JSON format, example: {'nodes':[{'id':'1','name':'A101','type':'room','bbox':[x1,y1,x2,y2]}...]}". After the model outputs structured JSON text, to improve coordinate accuracy, this system introduces a coordinate calibration mechanism: using the Canny edge detection algorithm to extract wall lines from the original image, the coarse bounding boxes output by the multimodal large model are snapped together with the nearest wall pixel edges. Finally, based on a pre-defined scale (e.g., 0.05 meters per pixel), the pixel coordinates are converted into physical coordinates, and the final semantic topology is constructed (e.g., ...). Figure 2 (as shown in B).

[0019] S102, Navigation Task Parsing Steps: The robot receives the user's navigation task instruction. This instruction can be voice (e.g., "Go to Meeting Room A101"), text input, or screen clicks. If it is a voice instruction, it is first converted into text using a speech recognition service. Subsequently, using natural language processing technology or by calling a large language model interface, the starting entity Node is extracted from the text. start (e.g., "front end") and target entity Node goal (e.g., “A101”), and find the corresponding node ID in the semantic topology structure constructed by S101.

[0020] S103, Global Path Reasoning Step: Input the semantic topology structure (including node connection relationships) generated in S101 and the task nodes parsed in S102 into a large language model (such as GPT-4, Llama 3, etc.). Construct the following prompt: "Based on the provided topology connection list, please plan an optimal path from 'front desk' to 'A101'. Please consider the logical rationality of the path (prioritize main roads and avoid passing through private rooms), and output the sequence of node IDs traversed." The large language model performs reasoning based on its pre-trained spatial common sense, outputting the global path node sequence π = [v1, v2, ..., v g ]. Where, v i This represents the i-th topological node in the path (corresponding to a specific room or passage area in the semantic topology graph), where v1 is the starting topological node, v g The target topology node is (e.g., Front Desk → Main Corridor → East Passage → A101).

[0021] S104, Navigation Execution Steps – Positioning: (e.g., ...) Figure 4 As shown, the robot needs to determine its precise pose x in the coordinate system of the planar layout diagram. t This embodiment employs a semantic-geometric fusion localization strategy: 1. Semantic observation: Use an RGB camera to capture environmental images and use an OCR model to identify house numbers (such as "A101"), providing a rough positional constraint. 2. Geometric Observation: Environmental point clouds are acquired using 2D or 3D LiDAR to generate local feature maps. These local feature maps are then registered with a prior feature map (as a reference map) generated based on the building floor plan. The registration process employs an iterative nearest-point algorithm or a normal distribution transformation algorithm, and calculates the local feature map by solving the following optimization problem. The optimal transformation relationship between the feature map and the reference map: Where R is the rotation matrix, t is the translation vector, and p i Let q be the i-th point in the local feature map. c(i) is the prior feature map (reference map) and p i The corresponding closest point, M is the number of optimization iterations. 3. Bayesian filtering: Using particle filtering or extended Kalman filtering to fuse the above observation and odometry data, output a high-frequency, smooth robot pose estimate.

[0022] S105, Navigation Execution Steps – Global Planning: Based on the determined robot pose and the node sequence output in S103, a collision-free global geometric path τ is generated on the rasterized map using the A* algorithm between adjacent topological nodes (e.g., from "main corridor" to "A101 doorway"). global The cost function of the A* algorithm is f(n) = g(n) + h(n), where g(n) is the actual movement cost and h(n) is the heuristic cost (such as Euclidean distance).

[0023] S106, Navigation Execution Steps – Local Control: Robot Tracks Global Geometric Path τ global Simultaneously, a local planner is used to avoid dynamic obstacles. This embodiment employs a hierarchical control architecture: 1. Dynamic Window Local Programming: This method uses a dynamic window approach to sample data in the velocity space (v, ω) and applies a cost function. G(v,ω) selects the optimal velocity: G(v,ω)=α·heading(v,ω)+β·dist(v,ω)+γ·velocity(v,ω) The parameters are defined as follows: v is the robot velocity, ω is the robot angular velocity, heading(·) is the azimuth cost function, dist(·) is the distance cost function, velocity(·) is the azimuth cost function, α is the weight of the azimuth cost function used to make the robot face the target; β is the weight of the distance cost function used to ensure a safe distance from obstacles; and γ is the weight of the velocity cost function used to encourage the robot to travel at high speed. In this embodiment, the preferred parameters are set as follows: α = 0.05, β = 0.2, and γ = 0.1. 2. Model Predictive Control Trajectory Tracking: To achieve smoother control, the trajectory generated by the dynamic window method can be used as a reference, and model predictive control can be used to generate the final low-level motor commands. Model predictive control tracks the trajectory of each motor. The control cycle is used to solve the following finite-time optimization problem to generate the control sequence: The meanings and values ​​of the symbols are defined as follows: J: Total cost function; N: Prediction time domain length, which is set to 10 in this embodiment; xk+1 : The predicted state vector at time k+1 (including position x, y and heading angle θ); x ref,k+1 : The corresponding reference status on the reference path; u k : The control input vector at time k (containing linear velocity v and angular velocity ω); Q: State error weight matrix, used to adjust tracking accuracy. In this embodiment, it is set as a diagonal matrix Q = diag(10, 10, 0.5), indicating that the penalty for position error is higher than that for angle error; W: Control input weight matrix, used to adjust the smoothness and energy consumption of control. In this embodiment, it is set as a diagonal matrix W = diag(0.1, 0.1). By solving this quadratic programming problem, the first term of the optimal control sequence is obtained and sent to the robot chassis as the control command at the current moment.

[0024] Example 2

[0025] This embodiment provides an electronic device for performing the above-described method. The device includes at least one processor (such as a CPU, GPU, or NPU), memory, and a communication interface. The memory stores a computer program; when the program is executed by the processor, the system can load building floor plans, call multimodal large models deployed in the cloud or locally via API for environmental analysis, and process sensor data in real time to complete all the navigation steps in Embodiment 1 above.

[0026] Those skilled in the art will understand that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A robot navigation method based on a planar layout diagram, characterized in that, Includes the following steps: Environmental semantic construction steps: Obtain the building floor plan of the target environment, use a multimodal model to parse the floor plan, extract the topological nodes and geometric features in the environment, and construct a semantic topology graph; Navigation task parsing steps: Obtain navigation task instructions, parse the starting information and target information in the instructions, and determine the corresponding starting topology node and target topology node in the semantic topology structure; Global path reasoning steps: Input the semantic topology structure, the starting topology node, and the target topology node into the large language model, and use the spatial reasoning capability of the large language model to generate a global path node sequence from the starting topology node to the target topology node; Navigation execution steps: Based on the real-time data acquired by the environmental perception device and the prior information of the planar layout, the robot's real-time pose is determined, and a geometric path is planned according to the global path node sequence and the real-time pose to control the robot to move to the target point.

2. The method according to claim 1, characterized in that, The environmental semantic construction steps specifically include: inputting the building floor plan into the visual language big model, extracting image features using a visual encoder; guiding the visual language big model to identify functional areas and connecting channels in the image through prompt words, defining them as topological nodes, and outputting structured text describing the connection relationships between nodes; parsing the structured text to establish the semantic topology structure containing node IDs, types, and geometric coordinates.

3. The method according to claim 2, characterized in that, The structured text is in one of the following formats: JSON, XML, or YAML; the semantic topology is a semantic topology graph.

4. The method according to claim 1, characterized in that, In the navigation task parsing step, the navigation task instruction is a voice instruction; parsing the instruction specifically includes: converting the voice instruction into text using speech recognition technology, and extracting the starting entity and target entity from the text using natural language processing technology or a large language model.

5. The method according to claim 1, characterized in that, The navigation execution steps for determining the robot's real-time pose specifically include: semantic observation matching: using camera equipment to perceive key semantic markers in the environment, and combining optical character recognition technology to extract the text information of the markers, and matching it with the semantic information in the planar layout map; geometric observation matching: using lidar or depth camera to acquire environmental point cloud data, and registering the generated local environmental feature map with the prior feature map generated based on the planar layout map; fusion localization: based on the semantic observation matching results and geometric observation matching results, using a probability filtering algorithm to calculate the robot's precise pose.

6. The method according to claim 5, characterized in that, The registration process employs either the iterative nearest point algorithm or the normal distribution transformation algorithm; the probabilistic filtering algorithm is either the Monte Carlo localization algorithm or the particle filter algorithm.

7. The method according to claim 1, characterized in that, The navigation execution steps also include a hierarchical planning process: geometric path planning: based on the global path node sequence, a collision-free global geometric path is generated between adjacent topological nodes using a search algorithm; local control: during the process of the robot tracking the global geometric path, an obstacle avoidance trajectory is generated by a local planner based on real-time perception data, and the obstacle avoidance trajectory is tracked by a controller.

8. The method according to claim 7, characterized in that, The search algorithm is A* algorithm, Dijkstra's algorithm or RRT algorithm; the local planner uses dynamic window method; the controller uses model predictive control algorithm.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images