A mobile robot target navigation system and method of navigation thereof

Abstract

The application discloses a kind of mobile robot target navigation system and navigation method thereof, comprising: map construction module, responsible for establishing the grid map and three-dimensional point cloud map for navigation positioning and the environment topological map for planning to seek path;Base navigation module, using grid map and point cloud map realizes fixed-point navigation function;Instruction analysis module is responsible for parsing natural language input and extracting key information etc.;Path planning module, with the aid of a variety of models and algorithms calculates the optimal search path;Target detection module, object three-dimensional positioning is realized by combining vision model and depth camera.The application can improve the target navigation effect of mobile robot in known environment, improve the deployability and safety of target navigation, and improve target navigation from approaching target to reaching target specified distance, realize more accurate target navigation.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robot navigation control, and in particular to a mobile robot target navigation system and a navigation method thereof that combine a large language model with an environment topology map. Background Art

[0002] With the rapid development of deep learning technology, large language models and visual models have shown excellent results, leading many robotics researchers to explore combining them. Because large language models can handle complex language tasks, they can be used for robot planning, while visual models focus on analyzing and understanding images and videos. They can identify objects and scenes, perform image classification, and detect targets, which can be used by robots to acquire external information. Robotic research methods that acquire basic information through sensors, process environmental information using visual models, and finally combine large language models for decision-making have become increasingly popular. This combination leverages the environmental perception capabilities of visual models and the decision-making capabilities of large language models, enabling robots to more accurately understand and respond to complex environmental information.

[0003] Mobile robots are widely used in various industries mentioned above, such as sweeping robots, welcoming guests, and delivery robots. Navigation is a fundamental function of mobile robots. Fixed-point navigation uses a coordinate in a map as the navigation location, allowing the mobile robot to reach the specified location based on sensor information and map data. This process involves multiple modules, including sensing, positioning, mapping, obstacle avoidance, path planning, and execution. Commonly used sensors include cameras, IMUs (inertial measurement units), single-line lasers, and multi-line lasers. Traditional navigation methods are only suitable for fixed-point navigation and are less suitable for more intelligent target navigation and visual language navigation.

[0004] With the advancement of artificial intelligence, research is growing on how to achieve more intelligent navigation, such as visual language navigation and goal-based navigation. Compared to traditional navigation based on fixed coordinates, goal-based navigation uses semantically defined target descriptions as navigational directions for mobile robots. Similar to how traditional navigation serves as the foundation for mobile robots, goal-based navigation can also be considered a foundation for task planning and execution for intelligent mobile robots. By providing goal-oriented navigation, mobile robots can perform more intelligent functions, such as delivering objects to designated targets or manipulating designated targets within an environment. Based on the type of task, goal-based navigation can be categorized into two main types: one for relatively known environments and the other for exploring completely unknown environments. Based on implementation methods, goal-based navigation can also be categorized into two main types: end-to-end approaches, where deep learning models directly output the final action; and engineering approaches, where a navigation system is built and logically determines the final action.

[0005] The existing target navigation methods have the following shortcomings:

[0006] First, existing target navigation methods are mostly designed for completely unknown environments. This is primarily due to the fact that many research datasets in this field are based on static scanned data, which cannot accurately simulate real-world working environments. Real-world robot working environments are often relatively fixed, yet also dynamic. For example, while a robot operates in a specific location, the general information about the location remains constant, while the information about objects and people within can change. However, many existing studies assume that the environment is completely unknown. This assumption ignores a lot of important information, reducing the effectiveness of mobile robot target navigation.

[0007] Second, most existing target navigation methods remain at the planning level and still have shortcomings in specific implementation and deployment. For example, the actions output by many methods are discrete, and the robot's kinematic characteristics and obstacle avoidance are not considered when moving. In addition, many methods only involve finding or approaching the target, but do not have the function of accurately reaching the specified distance of the target.

[0008] Third, the target navigation method based on deep learning has high training costs and is difficult to generalize in different environments. Summary of the Invention

[0009] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a mobile robot target navigation system and a navigation method thereof that combine a large language model with an environmental topology map, which can improve the effect of target navigation in a known environment, improve the deployability and safety of target navigation, and improve target navigation from approaching the target to reaching the specified distance of the target, thereby achieving more accurate target navigation.

[0010] To achieve the above objectives, the present invention provides a technical solution: a mobile robot target navigation system, comprising:

[0011] The map construction module is responsible for creating a 3D point cloud map based on the point cloud data collected by multi-line lasers, projecting the 3D point cloud map into a 2D grid map, and collecting semantic and visual information at set path points, thereby constructing an environmental topology map. The environmental topology map refers to the abstraction of the environment as a topological map, where the nodes of the topological map record the semantic and visual information of specific locations in the environment.

[0012] The basic navigation module uses a two-dimensional grid map and a three-dimensional point cloud map for global positioning and fixed-point navigation, encapsulates basic navigation functions, provides interfaces, and records operation logs;

[0013] The command analysis module analyzes the natural language input by the user based on a large language model and extracts effective navigation targets and auxiliary information;

[0014] The path planning module plans a suitable navigation path in the scene based on the navigation target, auxiliary information and environment topology map output by the instruction analysis module based on the large language model, that is, the optimal path;

[0015] The target detection module is responsible for detecting targets in visual information while searching along the navigation path, and calculating the position of objects based on the detection results and depth information.

[0016] Furthermore, the map construction module involves three types of maps, including three-dimensional point cloud maps, two-dimensional grid maps and environmental topology maps; among them, the three-dimensional point cloud map is used for the global positioning of the mobile robot, and it uses point clouds to accurately describe environmental information; the two-dimensional grid map is used for obstacle avoidance planning of the mobile robot, and it records obstacles in the environment from a top-down perspective; the environmental topology map is used for planning and finding paths. It is an abstraction of the environment, consisting of nodes and edges. In the environmental topology map, nodes represent an area, and edges represent the connectivity and distance between two areas. Nodes can record the visual information, semantic information and location of a certain area. The above three maps all use a unified map coordinate system.

[0017] Furthermore, the map construction module specifically performs the following operations:

[0018] First, all laser beams that can cover the entire height of the mobile robot are selected, and a three-dimensional point cloud map is established using the three-dimensional SLAM algorithm. The three-dimensional point cloud map is then reduced to a two-dimensional grid map. Then, according to the manually set path points, the map is divided into different navigation areas. Then, the path points are taken in turn to take photos of the surrounding environment. The photos of the surrounding environment are then processed using the visual language model VLM to summarize the description information of the surrounding environment. The labels of the path points are summarized based on the description information or manually set labels, where the description information and labels are semantic information and the photos taken are visual information. Then, the same number of nodes are created in the environment topology map to represent the path points, and the semantic information and visual information contained in the path points are recorded in the corresponding nodes in the environment topology map. Finally, edges are set between the nodes based on the connectivity relationship between the path points, and the distance between the nodes is calculated according to the two-dimensional grid map and assigned to the corresponding edges, thereby constructing a complete environment topology map.

[0019] Furthermore, the basic navigation module uses the Navigation2 navigation framework in the robot framework ROS2 as the underlying implementation; the basic navigation module uses the three-dimensional point cloud map and two-dimensional grid map created in the map construction module for positioning and navigation, and provides the following basic navigation functions: fixed-point navigation, navigation along trajectory points, movement of a specified distance forward and backward, rotation at a specified angle, and providing real-time posture information for the mobile robot during navigation; the basic navigation module converts the multiple feedback and waiting calls in ROS2 into a calling method that uses callback functions for notification. The caller does not need to consider the specific process in ROS2, but only needs to give instructions. The parameters of the callback function will eventually tell whether the instruction is executed successfully. In addition, the calling process is transmitted in JSON format data and recorded using the log system in ROS2.

[0020] Furthermore, the instruction analysis module receives natural language input and, by giving pre-adjusted prompt words, enables the large language model to extract the navigation target, auxiliary information and distance information in the natural language, wherein the navigation target is the final object to be found, the auxiliary information is additional information to assist in the search, and the distance information represents the specified distance that the mobile robot needs to move to the target after finding the target; the initial navigation target may not be directly detected by the target detection model, so after obtaining the initial navigation target, the large language model will be used again to convert the navigation target into the target category that is semantically closest and can be detected.

[0021] Furthermore, the path planning module receives the navigation target x, the auxiliary information y, the initial position t of the robot in the map coordinate system and the environment topology map m. Each node in the environment topology map m represents an area in the environment. The node stores the visual information and semantic information of the area, where the semantic information includes the description of the area and the label corresponding to the area. Let [p1, p2, p3, ..., p i ,p (i+1) ,...] is a node in the environment topology graph m, p i is the i-th node in the environment topology map, which is also a path point. The path planning module needs to calculate the following four values ​​to plan the final path: the similarity between the node and the navigation target x, the probability value of the navigation target x being near the node, the matching degree between the navigation target x and the visual information of each node, and the distance between each node and the position t. The specific situation is as follows:

[0022] First, the large language model is used to analyze the correlation between the navigation target x and the node label in the environment topology map m, and the auxiliary information y is combined to make corrections, and finally the similarity between the node and the navigation target x is given. The specific approach is: collect the labels of all nodes, send the node label, navigation target x and auxiliary information y into the large language model, and use the prompt word to guide the large language model to compare the similarity between the navigation target x and each label under the given auxiliary information y, where the similarity is a value between 0 and 1, and then assign the similarity of each label to the node containing the label, that is, p i The sum of all label similarities is p i The similarity A(x,y,p) with the navigation target x under given auxiliary information y i );

[0023] Subsequently, the large language model is used again to determine the correlation between the description of the navigation target x and the node in the environment topology map m, and is corrected in combination with the auxiliary information y, thereby giving the probability value of the navigation target x being near the node. The specific approach is: the descriptions corresponding to all nodes, the navigation target x and the auxiliary information y are sent to the large language model, and the prompt word is used to guide the large language model to analyze whether the navigation target x is near a certain node when the auxiliary information y is given, and finally the large language model outputs a value between 0 and 1 for each node to represent the probability value of the navigation target being near the node. Assume that under the given auxiliary information y, the navigation target x is at p i The probability of being near is B(x,y,p i );

[0024] Next, the visual text matching model CLIP is used to match the navigation target x with the visual information of each node in the environment topology map m, and then the matching degree between the navigation target x and the visual information of each node is given. The specific method is: the visual information of all nodes, that is, the pictures taken at the nodes, and the navigation target x are used as the input of the visual text matching model CLIP. The visual text matching model CLIP outputs the similarity between each picture and the navigation target x, then p i The sum of the similarities between all corresponding images and the navigation target x is the similarity between the navigation target x and p i The visual similarity C(x,p i ), and calculate the distance between each node and position t, let p i The distance from position t is D(t,p i );

[0025] Finally, perform a weighted sum operation on the above four values ​​to get p iThe final priority of the node is set as a, b is the probability value weight of the navigation target being near the node, c is the weight of the matching degree between the navigation target and the visual information in the environment topology map, and d is the weight of the distance between the current position and each node in the environment topology map. Then the path point p i The final priority S(x,y,t,p i ) is:

[0026] S(x,y,t,p i )=aA(x,y,p i )+bB(x,y,p i )+cC(x,p i )-dD(t,p i )

[0027] After finding the final priority of all nodes, the Dijkstra algorithm is used to calculate the shortest path from the current position to each node, thereby planning an optimal search path that traverses all path points.

[0028] Furthermore, the target detection module is implemented using a depth camera, including visual detection and position estimation, as follows:

[0029] Visual detection provides two object detection methods. The first is open semantic detection, which uses the open semantic object detection model Grounding-Vino. Open semantic detection can detect more categories and distinguish objects based on their position and color. Because the open semantic object detection model Grounding-Vino cannot be deployed on embedded platforms, it is deployed on a server and implemented remotely. The second is conventional object detection, which uses the YOLO11 series model and can only detect 80 predefined categories. Because the YOLO11 series model is lightweight, it can be directly deployed on mobile robots.

[0030] After the target is detected successfully, the position is estimated. Specifically, the depth camera is used to obtain point cloud information, and the instance segmentation model is used to divide the point cloud in the target area. Since the instance segmentation model cannot perfectly segment the target, the initial segmentation point cloud will have a large error, and the point cloud needs to be preprocessed. The processing flow is: remove independent points from the point cloud, downsample, and cluster using the DBSCAN algorithm and select the largest category to obtain accurate target point cloud information. Then, the directed bounding box of the point cloud is calculated to obtain the position of the target in the camera coordinate system.

[0031] The present invention also provides a navigation method for the mobile robot target navigation system, the details of which are as follows:

[0032] First, the map construction module is used to create a 3D point cloud map, a 2D raster map, and an environmental topology map. For the same environment, it only needs to be constructed once. The 3D point cloud map and the 2D raster map are used as input to the basic navigation module to activate the basic navigation function. Next, the natural language input is converted into navigation targets, auxiliary information, and the final distance through the command analysis module. The path planning module is used to plan a suitable search path, namely the optimal search path. The basic navigation module is used to sequentially move to the planned path points. At the same time, the target detection module is activated to detect the search target. When the target is detected, the robot stops moving and obtains the target's position in the camera coordinate system through the target detection module in the stationary state. Based on the relative pose of the camera coordinate system and the robot coordinate system, the target position is first converted from the camera coordinate system to the robot coordinate system. The basic navigation module is then used to obtain the robot's position in the map coordinate system. The target position is then converted from the robot coordinate system to the map coordinate system. Combined with the preset distance, a suitable stop point is determined in the map coordinate system. Finally, the basic navigation module guides the mobile robot to the stop point, thus completing a single navigation task.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. This invention proposes using tags to enhance the effectiveness of large language models for path planning, thereby improving target navigation. Specifically, this involves recording tag information for each location in the environment in the map construction module to enrich the semantic topology map, capturing user prompts for the target in the instruction analysis module, and using a large language model in the path planning module to analyze the correlation between human-generated tags in the environment topology map and navigation targets.

[0035] 2. This invention overcomes the safety issues of previous target navigation algorithms, which focused solely on planning and neglected deployment and execution. Specifically, it uses ROS2 to implement the basic navigation module, while the upper layer uses navigation point control. This system leverages the obstacle avoidance capabilities of basic navigation to avoid the safety issues associated with the direct use of discrete speed control in existing algorithms. Furthermore, because this system is built on the ROS2 framework, it can be easily deployed on mobile robots using the ROS2 framework.

[0036] 3. This invention improves existing target navigation systems from approaching a target to reaching a specified target location, providing a precondition for subsequent mobile robot operations. Specifically, this involves using a large language model to extract distance information in the command analysis module, combining the target detection model with a depth camera to achieve three-dimensional positioning of the target in the target detection module, and integrating it with the basic navigation module to obtain the mobile robot's global positioning and find the final navigation position within the specified distance on the map. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 This is an architectural diagram of the system of the present invention.

[0038] Figure 2 This is an example diagram of a mobile robot.

[0039] Figure 3 This is an example of an environment topology diagram.

[0040] Figure 4 Analyze the flow chart for the instruction.

[0041] Figure 5 Flowchart for route planning.

[0042] Figure 6 This is a navigation flow chart of the system of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0044] like Figure 1 As shown, this embodiment discloses a target navigation system based on a large language model and an environment topology map. The system integrates different modules in a modular way to achieve a highly available target navigation method, which includes a map construction module, a basic navigation module, a command analysis module, a path planning module, and a target detection module. This example is carried out in a real scene, and the mobile robot used is as follows Figure 2 As shown in the figure, the mobile robot uses a four-wheel differential car as the main body, is equipped with a depth camera and a multi-line laser, and runs on the Ubuntu 22.04 operating system with ROS2 (version HUMBLE).

[0045] The basic navigation module is built using the Navigation2 framework in ROS2. The module provides functions such as fixed-point navigation, rotation of a specific angle, forward and backward a specific distance, and global positioning. Among them, global positioning adopts the positioning solution of AUTOWARE (open source autonomous driving framework), which combines IMU, vehicle speed, ndt point cloud matching algorithm and extended Kalman filter algorithm. The basic navigation module takes into account the kinematic characteristics of the robot, such as the maximum speed of the robot's size, etc., and combines sensors to perceive the surrounding environment to ensure the safety of the robot's movement. In addition, the module encapsulates the navigation function and provides a simple callback function-based interface, and transmits data and records logs in JSON format.

[0046] The map construction module constructs three types of maps for this system, namely a three-dimensional point cloud map for positioning, a two-dimensional grid map for obstacle avoidance planning, and an environmental topology map for path planning. The three-dimensional point cloud map and the two-dimensional grid map will be used for basic navigation, and the environmental topology map will be used for the path planning module of target navigation. The three-dimensional point cloud map is constructed using multi-line laser and LIO_SAM algorithm, and then the three-dimensional point cloud map is projected into a two-dimensional grid map using the Octomap_Server library. After constructing the two-dimensional grid map and the three-dimensional point cloud map, key points are determined in the map through manual selection, and then basic navigation is used to control the mobile robot to go to these key points in turn, obtain the surrounding environment information and semantic information of the key points, and finally construct the information into an environmental topology map. The content of the environmental topology map is as follows: Figure 3 shown.

[0047] The instruction analysis module is responsible for processing natural language instructions. The system is expected to receive natural language input, which should mention the target to be found, and can optionally add additional auxiliary information and set the distance between the robot's final position and the target. Its workflow is as follows Figure 4 As shown. The instruction analysis module will first use the Tongyi Qianwen language model to process the language input, extract the target, auxiliary information and the final distance. The extracted target will then be converted into a recognizable target category. Assuming the target is a blue cup, it will be converted into blue cup and cup. The blue cup comes with color information and can only be processed using the open semantic detection model. Cup is one of the 80 categories supported by the conventional target detection model YOLO. Once the target cannot be recognized by the existing target detection model, the parsing failure will be returned. If the final distance is set, the mobile robot will navigate to the distance range after finding the target. Otherwise, it will navigate to a position closest to the target without colliding with obstacles.

[0048] The path planning module is responsible for generating a search path for the navigation target, such as Figure 5 As shown, the path planning module receives the navigation target x, auxiliary information y, the initial position t of the robot in the map coordinate system and the environment topology map m. Each node in the environment topology map m represents an area in the environment. The node stores the visual information and semantic information of the area, where the semantic information includes the description of the area and the label corresponding to the area. Let [p1, p2, p3, ..., p i ,p (i+1) ,...] is a node in the environment topology graph m, p iis the i-th node in the environment topology map, which is also a path point. The path planning module needs to calculate the following four values ​​to plan the final path: the similarity between the node and the navigation target x, the probability value of the navigation target x being near the node, the matching degree between the navigation target x and the visual information of each node, and the distance between each node and the position t. The specific situation is as follows:

[0049] First, the large language model is used to analyze the correlation between the navigation target x and the node label in the environment topology map m, and the auxiliary information y is combined to make corrections, and finally the similarity between the node and the navigation target x is given. The specific approach is: collect the labels of all nodes, send the node label, navigation target x and auxiliary information y into the large language model, and use the prompt word to guide the large language model to compare the similarity between the navigation target x and each label under the given auxiliary information y, where the similarity is a value between 0 and 1, and then assign the similarity of each label to the node containing the label, that is, p i The sum of all label similarities is p i The similarity A(x,y,p) with the navigation target x under given auxiliary information y i );

[0050] Subsequently, the large language model is used again to determine the correlation between the description of the navigation target x and the node in the environment topology map m, and is corrected in combination with the auxiliary information y, thereby giving the probability value of the navigation target x being near the node. The specific approach is: the descriptions corresponding to all nodes, the navigation target x and the auxiliary information y are sent to the large language model, and the prompt word is used to guide the large language model to analyze whether the navigation target x is near a certain node when the auxiliary information y is given, and finally the large language model outputs a value between 0 and 1 for each node to represent the probability value of the navigation target being near the node. Assume that under the given auxiliary information y, the navigation target x is at p i The probability of being near is B(x,y,p i );

[0051] Next, the visual text matching model CLIP is used to match the navigation target x with the visual information of each node in the environment topology map m, and then the matching degree between the navigation target x and the visual information of each node is given. The specific method is: the visual information of all nodes, that is, the pictures taken at the nodes, and the navigation target x are used as the input of the visual text matching model CLIP. The visual text matching model CLIP outputs the similarity between each picture and the navigation target x, then p i The sum of the similarities between all corresponding images and the navigation target x is the similarity between the navigation target x and p i The visual similarity C(x,p i ), and calculate the distance between each node and position t, let p i The distance from position t is D(t,p i);

[0052] Finally, perform a weighted sum operation on the above four values ​​to get p i The final priority of the node is set as a, b is the probability value weight of the navigation target being near the node, c is the weight of the matching degree between the navigation target and the visual information in the environment topology map, and d is the weight of the distance between the current position and each node in the environment topology map. Then the path point p i The final priority S(x,y,t,p i ) is:

[0053] S(x,y,t,p i )=aA(x,y,p i )+bB(x,y,p i )+cC(x,p i )-dD(t,p i )

[0054] After finding the final priority of all nodes, the Dijkstra algorithm is used to calculate the shortest path from the current position to each node, thereby planning an optimal search path that traverses all path points.

[0055] The target detection module is responsible for detecting whether a navigation target appears in the camera's field of view during operation. This module provides two target detection methods: open semantic detection, which uses the open semantic target detection model Grounding-Vino, and conventional target detection, which uses the YOLO11 series model. The open semantic target detection model Grounding-Vino is too large to be deployed directly on the mobile robot's host computer, so it is deployed on a server and implemented remotely. After deployment on the RTX3090, a single detection takes 0.5 seconds. Therefore, open detection cannot be performed in real time and can only stop and detect after reaching a path point. The YOLO11 series model can be deployed on a mobile robot and perform real-time detection. Because the target needs to be located, the system needs to be used with a depth camera. The depth camera used in this example is the RealSense D435i. The deep camera can obtain the point cloud corresponding to the image. When the target is detected, the module will segment the target area through the instance segmentation model. Since the segmentation model cannot achieve perfect segmentation, the initially segmented point cloud often contains irrelevant points. It is necessary to first perform downsampling and independent point removal, and then perform DBSCAN clustering. The cluster with the largest number of points is regarded as the target surface point cloud, and then the directed bounding box of the point cloud is calculated. The center of the bounding box is used as the target position in the camera coordinate system. Finally, the coordinate transformation is performed to obtain the position of the target in the map.

[0056] like Figure 6As shown, the navigation method of the mobile robot target navigation system in this embodiment is as follows:

[0057] First, use the map construction module to create a three-dimensional point cloud map, a two-dimensional raster map, and an environmental topology map. For the same environment, it only needs to be constructed once, and the three-dimensional point cloud map and the two-dimensional raster map are used as inputs of the basic navigation module to activate the basic navigation function. The two-dimensional raster map and the three-dimensional point cloud map correspond to the same coordinate system. In the navigation task, the coordinate system corresponding to the two-dimensional raster map will be used as the map coordinate system, and the two-dimensional raster map will be used as the global map. Then, the natural language input is converted into navigation targets, auxiliary information, and the final distance through the instruction analysis module, and the path planning module is used to plan a suitable search path, that is, the optimal search path. The basic navigation module is used to go to the planned path points in sequence, and the target detection module is started to detect the search target. When the target is detected, it will stop moving and obtain the target's position in the camera coordinate system through the target detection module in the stationary state. According to the relative posture of the camera coordinate system and the robot coordinate system, the target position is first converted from the camera coordinate system to the robot coordinate system, and then the basic navigation module is used to obtain the robot's position in the map coordinate system. The target position is then converted from the robot coordinate system to the map coordinate system, and combined with the preset distance, a suitable stop point is determined on the map coordinate system. Finally, the basic navigation module is used to guide the mobile robot to the stop point, thereby completing a single navigation task.

[0058] In summary, this paper combines a large language model with an environmental topology map to implement a target navigation system built on ROS2. This system improves target navigation in known environments, addressing the shortcomings of existing target navigation algorithms, which lack consideration of deployment and robot kinematics. Furthermore, it addresses the shortcomings of existing target navigation algorithms, which lack consideration of the distance to the final target. This system has broad research and practical application value and is worthy of promotion.

[0059] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A mobile robot target navigation system, characterized in that: include: The map construction module is responsible for creating a 3D point cloud map based on the point cloud data collected by multi-line lasers, projecting the 3D point cloud map into a 2D grid map, and collecting semantic and visual information at set path points, thereby constructing an environmental topology map. The environmental topology map refers to the abstraction of the environment as a topological map, where the nodes of the topological map record the semantic and visual information of specific locations in the environment. The basic navigation module uses a two-dimensional grid map and a three-dimensional point cloud map for global positioning and fixed-point navigation, encapsulates basic navigation functions, provides interfaces, and records operation logs; The command analysis module analyzes the natural language input by the user based on a large language model and extracts effective navigation targets and auxiliary information; The instruction analysis module receives natural language input and, by providing pre-adjusted prompt words, enables the large language model to extract the navigation target, auxiliary information, and distance information from the natural language. The navigation target is the final object to be found, the auxiliary information is additional information to assist in the search, and the distance information represents the specified distance the mobile robot needs to move to the target after finding it. The initial navigation target may not be directly detectable by the target detection model. Therefore, after obtaining the initial navigation target, the large language model is used again to convert the navigation target into the target category that is semantically closest and can be detected. The path planning module plans a suitable navigation path in the scene based on the navigation target, auxiliary information and environment topology map output by the instruction analysis module based on the large language model, that is, the optimal path; The target detection module is responsible for detecting targets in visual information while searching along the navigation path, and calculating the position of objects based on the detection results and depth information.

2. A mobile robot target navigation system according to claim 1, characterized in that: The map construction module involves three types of maps, including three-dimensional point cloud maps, two-dimensional grid maps and environmental topology maps; among them, the three-dimensional point cloud map is used for the global positioning of the mobile robot, and it uses point clouds to accurately describe environmental information; the two-dimensional grid map is used for obstacle avoidance planning of the mobile robot, and it records obstacles in the environment from a top-down perspective; the environmental topology map is used for planning and finding paths. It is an abstraction of the environment and consists of nodes and edges. In the environmental topology map, nodes represent an area, and edges represent the connectivity and distance between two areas. Nodes can record the visual information, semantic information and location of a certain area. The above three maps all use a unified map coordinate system.

3. A mobile robot target navigation system according to claim 2, characterized in that: The map construction module specifically performs the following operations: First, all laser beams that can cover the entire height of the mobile robot are selected, and a three-dimensional point cloud map is established using the three-dimensional SLAM algorithm. The three-dimensional point cloud map is then reduced to a two-dimensional grid map. Then, according to the manually set path points, the map is divided into different navigation areas. Then, the path points are taken in turn to take photos of the surrounding environment. The photos of the surrounding environment are then processed using the visual language model VLM to summarize the description information of the surrounding environment. The labels of the path points are summarized based on the description information or manually set labels, where the description information and labels are semantic information and the photos taken are visual information. Then, the same number of nodes are created in the environment topology map to represent the path points, and the semantic information and visual information contained in the path points are recorded in the corresponding nodes in the environment topology map. Finally, edges are set between the nodes based on the connectivity relationship between the path points, and the distance between the nodes is calculated according to the two-dimensional grid map and assigned to the corresponding edges, thereby constructing a complete environment topology map.

4. A mobile robot target navigation system according to claim 3, characterized in that: The basic navigation module uses the Navigation2 navigation framework in the robot framework ROS2 as the underlying implementation; the basic navigation module uses the three-dimensional point cloud map and two-dimensional grid map created in the map construction module for positioning and navigation, and provides the following basic navigation functions: fixed-point navigation, navigation along trajectory points, movement of a specified distance forward and backward, rotation at a specified angle, and providing real-time posture information for the mobile robot during navigation; the basic navigation module converts the multiple feedback and waiting calls in ROS2 into a calling method that uses callback functions for notification. The caller does not need to consider the specific process in ROS2, but only needs to give an instruction. The parameters of the callback function will eventually tell whether the instruction is executed successfully. In addition, the calling process is transmitted in JSON format data and recorded using the log system in ROS2.

5. A mobile robot target navigation system according to claim 4, characterized in that: The path planning module receives the navigation target , auxiliary information , the robot's initial position in the map coordinate system and environment topology diagram , environment topology map Each node in the CNN represents an area in the environment. The node stores the visual information and semantic information of the area, where the semantic information includes the description of the area around it and the label corresponding to the area. Environment topology diagram The nodes in The first Nodes, also known as path points; the path planning module needs to calculate the following four values ​​to plan the final path: nodes and navigation targets Similarity, navigation target Probability value near the node, navigation target The matching degree with each node’s visual information and the matching degree between each node and position The details are as follows: First, use a large language model to analyze navigation goals and environment topology diagram The association degree of the node labels in the Make corrections and finally give the node and navigation target The specific approach is to collect the labels of all nodes, and then combine the node labels and navigation targets With auxiliary information Send it to the large language model, and use the prompt word to guide the large language model to compare the given auxiliary information Navigation target in case of The similarity with each label, where the similarity is The value between them is then assigned the similarity of each label to the node containing the label, that is, The sum of all label similarities is Given auxiliary information Next and navigation target Similarity ; Then, the large language model is used again to determine the navigation target and environment topology diagram The relevance of the description of the node in the Make corrections and then give the navigation target The probability value of being near a node is as follows: the description and navigation target corresponding to all nodes are With auxiliary information Send it to the big language model and guide the big language model to analyze the navigation target through prompt words Given auxiliary information When it is near a node, it determines whether it meets the description of the node, and finally the large language model outputs a The value between represents the probability value of the navigation target being near the node. Given the auxiliary information Next, navigation target in The probability near ; Then, the visual text matching model CLIP is used to match the navigation target and environment topology diagram Match the visual information of each node in the image to give the navigation target The matching degree with the visual information of each node is as follows: the visual information of all nodes, that is, the pictures taken at the nodes, are matched with the navigation target. As the input of the visual text matching model CLIP, the visual text matching model CLIP outputs each picture and the navigation target The similarity of All corresponding pictures and navigation targets The sum of the similarities is the navigation target and Visual similarity , and calculate each node and position The distance, set and location The distance is ; Finally, perform a weighted sum operation on the above four values ​​to obtain The final priority of is the weight of the similarity between the node and the navigation target, is the probability weight of the navigation target being near the node, is the weight of the matching degree between the navigation target and the visual information in the environment topology map, is the weight of the distance between the current position and each node in the environment topology graph, then the path point The final priority The expression is: ； After finding the final priority of all nodes, the Dijkstra algorithm is used to calculate the shortest path from the current position to each node, thereby planning an optimal search path that traverses all path points.

6. A mobile robot target navigation system according to claim 5, characterized in that: The target detection module is implemented using a depth camera, including visual detection and position estimation, as follows: Visual detection provides two object detection methods. The first is open semantic detection, which uses the open semantic object detection model Grounding-Vino. Open semantic detection can detect more categories and distinguish objects based on their position and color. Because the open semantic object detection model Grounding-Vino cannot be deployed on embedded platforms, it is deployed on a server and implemented remotely. The second is conventional object detection, which uses the YOLO11 series model and can only detect 80 predefined categories. Because the YOLO11 series model is lightweight, it can be directly deployed on mobile robots. After the target is detected successfully, the position is estimated. Specifically, the depth camera is used to obtain point cloud information, and the instance segmentation model is used to divide the point cloud in the target area. Since the instance segmentation model cannot perfectly segment the target, the initial segmentation point cloud will have a large error, and the point cloud needs to be preprocessed. The processing flow is: remove independent points from the point cloud, downsample, and cluster using the DBSCAN algorithm and select the largest category to obtain accurate target point cloud information. Then, the directed bounding box of the point cloud is calculated to obtain the position of the target in the camera coordinate system.

7. The navigation method of the mobile robot target navigation system according to any one of claims 1 to 6, characterized in that: First, the map construction module is used to create a 3D point cloud map, a 2D raster map, and an environmental topology map. For the same environment, it only needs to be constructed once. The 3D point cloud map and the 2D raster map are used as input to the basic navigation module to activate the basic navigation function. Next, the natural language input is converted into navigation targets, auxiliary information, and the final distance through the command analysis module. The path planning module is used to plan a suitable search path, namely the optimal search path. The basic navigation module is used to sequentially move to the planned path points. At the same time, the target detection module is activated to detect the search target. When the target is detected, the robot stops moving and obtains the target's position in the camera coordinate system through the target detection module in the stationary state. Based on the relative pose of the camera coordinate system and the robot coordinate system, the target position is first converted from the camera coordinate system to the robot coordinate system. The basic navigation module is then used to obtain the robot's position in the map coordinate system. The target position is then converted from the robot coordinate system to the map coordinate system. Combined with the preset distance, a suitable stop point is determined in the map coordinate system. Finally, the basic navigation module guides the mobile robot to the stop point, thus completing a single navigation task.

Images