Mobile robot target navigation system and navigation method thereof

Abstract

The invention discloses a mobile robot target navigation system and a navigation method thereof, and the system comprises a map building module which is responsible for building a grid map and a three-dimensional point cloud map which are used for navigation and positioning, and an environment topological graph which is used for planning and searching a path; the basic navigation module is used for realizing a fixed-point navigation function by using a grid map and a point cloud map; the instruction analysis module is responsible for analyzing natural language input, extracting key information and the like; the path planning module is used for calculating an optimal searching path by means of various models and algorithms; and the target detection module combines the visual model and the depth camera to realize three-dimensional positioning of the object. According to the invention, the target navigation effect of the mobile robot in a known environment can be improved, the deployability and safety of target navigation are improved, the target navigation is improved from approaching the target to reaching the specified distance of the target, and more accurate target navigation is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of mobile robot navigation control, and in particular to a mobile robot target navigation system and a navigation method thereof which combine a large language model with an environmental topology map. Background Art

[0002] With the rapid development of deep learning technology, large language models and visual models have shown excellent results, so many robot researchers have also tried to combine them. Because large language models can handle complex language tasks, they can be used for robot planning, while visual models focus on the analysis and understanding of images and videos. They can identify objects and scenes, perform image classification and target detection, etc., which can be used for robots to obtain external information. Robot research methods that obtain basic information through sensors, use visual models to process environmental information, and finally combine large language models for decision-making have gradually become a hot topic. This combination utilizes the visual model's ability to perceive the environment and the decision-making ability of the large language model, allowing robots to more accurately understand and respond to complex environmental information.

[0003] Mobile robots are widely used in various industries mentioned above, such as sweeping robots, welcoming robots and delivery robots. Among them, navigation is a very basic function of mobile robots. Fixed-point navigation refers to using a certain coordinate in the map as the navigation position, allowing the mobile robot to reach the specified location based on sensor information and map information. This process involves multiple modules including sensing, positioning, mapping, obstacle avoidance, path planning and execution. Commonly used sensors include cameras, IMU (inertial measurement unit), single-line laser and multi-line laser. Traditional navigation methods are only applicable to fixed-point navigation, and are not very applicable to more intelligent target navigation and visual language navigation.

[0004] With the development of artificial intelligence, more and more people are studying how to achieve more intelligent navigation, such as visual language navigation, target navigation, etc. Compared with the original navigation based on fixed-point coordinates, target navigation uses semantic target description as the navigation direction of mobile robots. Analogous to traditional navigation as the basis of mobile robots, target navigation can also be regarded as a basis for task planning and execution of intelligent mobile robots. By providing target-oriented navigation, mobile robots can perform more intelligent functions, such as delivering items to designated targets or operating designated targets in the environment. From the perspective of task type, target navigation can be divided into two main types, one is for relatively known environments, and the other is to explore completely unknown environments. From the perspective of implementation methods, target navigation can also be divided into two main types, one is the end-to-end method, that is, the final action is directly output through a deep learning model. The other is the engineering method, that is, building a navigation system and selecting the final action through logical judgment.

[0005] The existing target navigation methods have the following shortcomings:

[0006] First, most existing target navigation methods are oriented towards completely unknown environments. This is mainly because many research data sets in this field are static scan data, which cannot accurately simulate the real working environment. The working environment of real robots is often relatively fixed, but also has a certain degree of dynamics. For example, when a robot works in a certain place, the general information of the place remains unchanged, but the information of objects and people in the place can change. However, many existing studies believe that the environment is completely unknown. This assumption ignores a lot of important information and reduces the target navigation effect of mobile robots.

[0007] Second, most existing target navigation methods remain at the planning level and are still insufficient in terms of specific implementation and deployment. For example, the actions output by many methods are discrete, and the robot's kinematic characteristics and obstacle avoidance are not considered when moving. In addition, many methods only involve finding or approaching the target, but do not have the function of accurately reaching the specified distance of the target.

[0008] Third, the target navigation method based on deep learning has a high training cost and is difficult to generalize to different environments. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and to provide a mobile robot target navigation system and a navigation method thereof that combine a large language model with an environmental topology map, which can improve the effect of target navigation in a known environment, improve the deployability and safety of target navigation, and improve target navigation from approaching the target to reaching a specified distance to the target, thereby achieving more accurate target navigation.

[0010] To achieve the above object, the technical solution provided by the present invention is: a mobile robot target navigation system, comprising:

[0011] The map construction module is responsible for building a 3D point cloud map based on the point cloud data collected by multi-line lasers, projecting the 3D point cloud map into a 2D grid map, and collecting semantic information and visual information at the set path points, thereby realizing the construction of an environmental topology map. The environmental topology map refers to the abstraction of the environment into a topology map, and the nodes of the topology map record the semantic information and visual information of specific locations in the environment.

[0012] The basic navigation module uses two-dimensional grid maps and three-dimensional point cloud maps for global positioning and fixed-point navigation, encapsulates basic navigation functions, provides interfaces, and records operation logs;

[0013] The instruction analysis module analyzes the natural language input by the user based on the large language model and extracts effective navigation targets and auxiliary information;

[0014] The path planning module plans a suitable navigation path in the scene based on the navigation target, auxiliary information and environmental topology map output by the instruction analysis module based on the large language model, that is, the optimal search path;

[0015] The target detection module is responsible for detecting targets in visual information while searching along the navigation path, and calculating the position of objects based on the detection results and depth information.

[0016] Furthermore, the map construction module involves three types of maps, including three-dimensional point cloud maps, two-dimensional grid maps and environmental topology maps; among them, the three-dimensional point cloud map is used for the global positioning of the mobile robot, and it uses point clouds to accurately describe environmental information; the two-dimensional grid map is used for obstacle avoidance planning of the mobile robot, and it records obstacles in the environment from a top-down perspective; the environmental topology map is used for planning and finding paths, and it is an abstraction of the environment, consisting of nodes and edges. In the environmental topology map, nodes represent an area, and edges represent the connectivity and distance between two areas. Nodes can record the visual information, semantic information and location of a certain area, and the above three maps all use a unified map coordinate system.

[0017] Furthermore, the map construction module specifically performs the following operations:

[0018] First, all laser beams that can cover the entire height of the mobile robot are selected, and a three-dimensional point cloud map is established using a three-dimensional SLAM algorithm. The three-dimensional point cloud map is then reduced to a two-dimensional grid map. Then, according to the manually set path points, the map is divided into different navigation areas, and then the path points are taken in turn to take photos of the surrounding environment. Then, the visual language model VLM is used to process the photos of the surrounding environment, and the description information of the surrounding environment is summarized. The labels of the path points are summarized in combination with the description information or manually set labels, where the description information and labels are semantic information, and the photos taken are visual information; then, the same number of nodes are created in the environmental topology map to represent the path points, and the semantic information and visual information contained in the path points are recorded in the corresponding nodes in the environmental topology map. Finally, edges are set between the nodes according to the connectivity between the path points, and the distances between the nodes are calculated according to the two-dimensional grid map and assigned to the corresponding edges, thereby constructing a complete environmental topology map.

[0019] Furthermore, the basic navigation module uses the Navigation2 navigation framework in the robot framework ROS2 as the underlying implementation; the basic navigation module uses the three-dimensional point cloud map and the two-dimensional grid map created in the map construction module for positioning and navigation, and provides the following basic navigation functions: fixed-point navigation, navigation along trajectory points, movement of a specified distance forward and backward, rotation of a specified angle, and providing real-time posture information for the mobile robot during navigation; the basic navigation module converts the multiple feedback and waiting calls in ROS2 into a calling method that notifies with a callback function. The caller does not need to consider the specific process in ROS2, but only needs to give an instruction, and the parameters of the callback function will eventually tell whether the instruction is executed successfully. In addition, the calling process is transmitted in JSON format data and recorded using the log system in ROS2.

[0020] Furthermore, the instruction analysis module receives natural language input, and by giving pre-adjusted prompt words, the large language model extracts the navigation target, auxiliary information and distance information in the natural language, wherein the navigation target is the final object to be found, the auxiliary information is additional information to assist in the search, and the distance information represents the specified distance that the mobile robot needs to move to the target after finding the target; the initial navigation target may not be directly detected by the target detection model, so after obtaining the initial navigation target, the large language model will be used again to convert the navigation target into the target category that is semantically closest and can be detected.

[0021] Furthermore, the path planning module receives the navigation target x, the auxiliary information y, the initial position t of the robot in the map coordinate system and the environment topology map m, each node in the environment topology map m represents an area in the environment, and the node stores the visual information and semantic information of the area, wherein the semantic information includes the description of the area and the label corresponding to the area. Let [p1, p2, p3, ..., p i ,p (i+1) ,...] is a node in the environment topology graph m, p i is the i-th node in the environment topology map, and is also a path point. The path planning module needs to calculate the following four values ​​to plan the final path: the similarity between the node and the navigation target x, the probability value of the navigation target x being near the node, the matching degree between the navigation target x and the visual information of each node, and the distance between each node and the position t. The specific situation is as follows:

[0022] First, the large language model is used to analyze the correlation between the navigation target x and the node label in the environment topology map m, and the auxiliary information y is combined to make corrections, and finally the similarity between the node and the navigation target x is given. The specific method is as follows: collect the labels of all nodes, send the node labels, navigation target x and auxiliary information y into the large language model, and use the prompt words to guide the large language model to compare the similarity between the navigation target x and each label under the given auxiliary information y, where the similarity is a value between 0 and 1, and then assign the similarity of each label to the node containing the label, that is, p i The sum of all label similarities is p i The similarity A(x,y,p) with the navigation target x under given auxiliary information y i );

[0023] Subsequently, the large language model is used again to determine the correlation between the description of the navigation target x and the node in the environment topology map m, and is corrected in combination with the auxiliary information y, thereby giving the probability value of the navigation target x being near the node. The specific method is as follows: the descriptions corresponding to all nodes, the navigation target x and the auxiliary information y are sent to the large language model, and the large language model is guided by prompt words to analyze whether the navigation target x is near a certain node when the auxiliary information y is given, and whether it meets the description of the node. Finally, the large language model outputs a value between 0 and 1 for each node to represent the probability value of the navigation target being near the node. Suppose that under the given auxiliary information y, the navigation target x is near p i The probability of nearby is B(x,y,p i );

[0024] Next, the visual text matching model CLIP is used to match the navigation target x with the visual information of each node in the environment topology map m, and then the matching degree between the navigation target x and the visual information of each node is given. The specific method is: the visual information of all nodes, that is, the pictures taken at the nodes, and the navigation target x are used as the input of the visual text matching model CLIP. The output of the visual text matching model CLIP is the similarity between each picture and the navigation target x, then p i The sum of the similarities between all corresponding images and the navigation target x is the similarity between the navigation target x and p i The visual similarity C(x,p i ), and calculate the distance between each node and position t, let p i The distance from position t is D(t,p i );

[0025] Finally, perform a weighted sum operation on the above four values ​​to get p iThe final priority of the node is, let a be the weight of the similarity between the node and the navigation target, b be the weight of the probability value of the navigation target being near the node, c be the weight of the matching degree between the navigation target and the visual information in the environment topology map, d be the weight of the distance between the current position and each node in the environment topology map, then the path point p i The final priority S(x,y,t,p i ) is:

[0026] S(x,y,t,p i )=aA(x,y,p i )+bB(x,y,p i )+cC(x,p i )-dD(t,p i )

[0027] After finding the final priority of all nodes, the Dijkstra algorithm is used to calculate the shortest path from the current position to each node, thereby planning an optimal search path that traverses all path points.

[0028] Furthermore, the target detection module is implemented using a depth camera, including visual detection and position estimation, as follows:

[0029] Visual detection provides two target detection methods. The first is open semantic detection, which uses the open semantic target detection model Grounding-Vino. Open semantic detection can detect more categories and distinguish objects based on their position and color. Since the open semantic target detection model Grounding-Vino cannot be deployed on embedded platforms, it is deployed on a server and implemented by remote calling. The second is conventional target detection, which uses the YOLO11 series model and can only detect 80 predefined categories. Since the YOLO11 series model is lightweight, it can be directly deployed on mobile robots.

[0030] After the target is detected successfully, the position is estimated. Specifically, the depth camera is used to obtain point cloud information, and the instance segmentation model is used to divide the point cloud in the target area. Since the instance segmentation model cannot perfectly segment the target, the point cloud segmented for the first time will have a large error, and the point cloud needs to be preprocessed. The processing flow is: remove independent points of the point cloud, downsample, cluster using the DBSCAN algorithm and select the largest category, so as to obtain accurate target point cloud information, and then calculate the directed bounding box of the point cloud to obtain the position of the target in the camera coordinate system.

[0031] The present invention also provides a navigation method for the mobile robot target navigation system, the specific situation is as follows:

[0032] First, the map construction module is used to create a three-dimensional point cloud map, a two-dimensional grid map and an environmental topology map. For the same environment, it only needs to be constructed once, and the three-dimensional point cloud map and the two-dimensional grid map are used as the input of the basic navigation module to activate the basic navigation function. Then, the natural language input is converted into navigation targets, auxiliary information and final distances through the instruction analysis module, and a suitable search path, that is, the optimal search path, is planned using the path planning module. The basic navigation module is used to go to the planned path points in sequence, and the target detection module is started to detect the search target. When the target is detected, the mobile action will be stopped, and the target position in the camera coordinate system will be obtained by the target detection module in a stationary state. According to the relative position of the camera coordinate system and the robot coordinate system, the target position is first converted from the camera coordinate system to the robot coordinate system, and then the basic navigation module is used to obtain the robot's position in the map coordinate system, and then the target position is converted from the robot coordinate system to the map coordinate system. Combined with the preset distance, a suitable stop point is determined on the map coordinate system. Finally, the mobile robot is guided to the stop point through the basic navigation module to complete a single navigation task.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. The present invention proposes to use tags to improve the effect of path planning by a large language model, thereby improving the effect of target navigation. Specifically, the map construction module records the tag information of each location in the environment to enrich the semantic topology map, the instruction analysis module captures the user's prompts for the target, and the path planning module uses a large language model to analyze the correlation between the artificial tags in the environment topology map and the navigation target.

[0035] 2. The present invention overcomes the problem that the target navigation in the past only stays at the planning layer without considering the safety issues during deployment and execution. Specifically, it uses ROS2 to implement the basic navigation module, uses navigation point control in the upper layer, and uses the obstacle avoidance function of the basic navigation to avoid the safety issues caused by the direct use of discrete speed control in the existing algorithm. At the same time, since this system is built on the ROS2 framework, it can be easily deployed on a mobile robot using the ROS2 framework.

[0036] 3. The present invention improves the existing target navigation system from approaching the target to reaching the specified target position, which provides a precondition for the subsequent operation of the mobile robot. Specifically, it uses a large language model to extract distance information in the instruction analysis module, combines the target detection model with the depth camera in the target detection module to achieve three-dimensional positioning of the target, and combines the basic navigation module to obtain the global positioning of the mobile robot and then find the final navigation position that meets the specified distance in the map. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the system of the present invention.

[0038] Figure 2 This is an example diagram of a mobile robot.

[0039] Figure 3 This is an example of an environment topology diagram.

[0040] Figure 4 Analyze the flow chart for the instruction.

[0041] Figure 5 Flowchart for route planning.

[0042] Figure 6 The navigation flow chart of the system of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0044] like Figure 1 As shown, this embodiment discloses a target navigation system based on a large language model and an environmental topology map. The system integrates different modules in a modular way to achieve a highly available target navigation method, which includes a map construction module, a basic navigation module, a command analysis module, a path planning module, and a target detection module. This example is carried out in a real scene, and the mobile robot used is as follows Figure 2 As shown, the mobile robot uses a four-wheel differential car as the main body, is equipped with a depth camera and a multi-line laser, and runs on the Ubuntu 22.04 operating system with ROS2 (version HUMBLE).

[0045] The basic navigation module is built using the Navigation2 framework in ROS2, which provides functions such as fixed-point navigation, rotation of a specific angle, forward and backward specific distance, and global positioning. The global positioning adopts the positioning solution of AUTOWARE (open source autonomous driving framework), which combines IMU, vehicle speed, ndt point cloud matching algorithm and extended Kalman filter algorithm. The basic navigation module takes into account the kinematic characteristics of the robot, such as the maximum speed of the robot's size, etc., and combines sensors to perceive the surrounding environment to ensure the safety of the robot's movement. In addition, the module encapsulates the navigation function and provides a simple callback function-based interface, and transmits data and records logs in JSON format.

[0046] The map construction module constructs three types of maps for this system, namely a three-dimensional point cloud map for positioning, a two-dimensional grid map for obstacle avoidance planning, and an environmental topology map for path planning. The three-dimensional point cloud map and the two-dimensional grid map will be used for basic navigation, and the environmental topology map will be used in the path planning module for target navigation. The three-dimensional point cloud map is constructed using multi-line laser and LIO_SAM algorithm, and then the Octomap_Server library is used to project the three-dimensional point cloud map into a two-dimensional grid map. After the two-dimensional grid map and the three-dimensional point cloud map are constructed, the key points are determined in the map through manual selection, and then the basic navigation is used to control the mobile robot to go to these key points in turn, obtain the surrounding environment information and semantic information of the key points, and finally construct the information into an environmental topology map. The content of the environmental topology map is as follows: Figure 3 shown.

[0047] The command analysis module is responsible for processing natural language commands. The system is expected to receive natural language input, which should mention the target to be found, and can optionally add additional auxiliary information and set the distance between the robot's final position and the target. The workflow is as follows Figure 4 As shown. The instruction analysis module will first use the Tongyi Qianwen language model to process the language input and extract the target, auxiliary information and final distance. The extracted target will then be converted into a recognizable target category. Assuming that the target is a blue cup, it will be converted into blue cup and cup. The blue cup comes with color information and can only be processed using the open semantic detection model. Cup is one of the 80 categories supported by the conventional target detection model YOLO. Once the target cannot be recognized by the existing target detection model, the parsing failure will be returned. If the final distance is set, the mobile robot will navigate to the distance range after finding the target. Otherwise, it will navigate to a position closest to the target without colliding with obstacles.

[0048] The path planning module is responsible for generating a path for the navigation target, such as Figure 5 As shown in FIG. 1 , the path planning module receives the navigation target x, the auxiliary information y, the initial position t of the robot in the map coordinate system, and the environment topology map m. Each node in the environment topology map m represents an area in the environment. The node stores the visual information and semantic information of the area, wherein the semantic information includes the description of the area around and the label corresponding to the area. Let [p1, p2, p3, ..., p i ,p (i+1) ,...] is a node in the environment topology graph m, p iis the i-th node in the environment topology map, and is also a path point. The path planning module needs to calculate the following four values ​​to plan the final path: the similarity between the node and the navigation target x, the probability value of the navigation target x being near the node, the matching degree between the navigation target x and the visual information of each node, and the distance between each node and the position t. The specific situation is as follows:

[0049] First, the large language model is used to analyze the correlation between the navigation target x and the node label in the environment topology map m, and the auxiliary information y is combined to make corrections, and finally the similarity between the node and the navigation target x is given. The specific method is as follows: collect the labels of all nodes, send the node labels, navigation target x and auxiliary information y into the large language model, and use the prompt words to guide the large language model to compare the similarity between the navigation target x and each label under the given auxiliary information y, where the similarity is a value between 0 and 1, and then assign the similarity of each label to the node containing the label, that is, p i The sum of all label similarities is p i The similarity A(x,y,p) with the navigation target x under given auxiliary information y i );

[0050] Subsequently, the large language model is used again to determine the correlation between the description of the navigation target x and the node in the environment topology map m, and is corrected in combination with the auxiliary information y, thereby giving the probability value of the navigation target x being near the node. The specific method is as follows: the descriptions corresponding to all nodes, the navigation target x and the auxiliary information y are sent to the large language model, and the large language model is guided by prompt words to analyze whether the navigation target x is near a certain node when the auxiliary information y is given, and whether it meets the description of the node. Finally, the large language model outputs a value between 0 and 1 for each node to represent the probability value of the navigation target being near the node. Suppose that under the given auxiliary information y, the navigation target x is near p i The probability of nearby is B(x,y,p i );

[0051] Next, the visual text matching model CLIP is used to match the navigation target x with the visual information of each node in the environment topology map m, and then the matching degree between the navigation target x and the visual information of each node is given. The specific method is: the visual information of all nodes, that is, the pictures taken at the nodes, and the navigation target x are used as the input of the visual text matching model CLIP. The output of the visual text matching model CLIP is the similarity between each picture and the navigation target x, then p i The sum of the similarities between all corresponding images and the navigation target x is the similarity between the navigation target x and p i The visual similarity C(x,p i ), and calculate the distance between each node and position t, let p i The distance from position t is D(t,p i);

[0052] Finally, perform a weighted sum operation on the above four values ​​to get p i The final priority of the node is, let a be the weight of the similarity between the node and the navigation target, b be the weight of the probability value of the navigation target being near the node, c be the weight of the matching degree between the navigation target and the visual information in the environment topology map, d be the weight of the distance between the current position and each node in the environment topology map, then the path point p i The final priority S(x,y,t,p i ) is:

[0053] S(x,y,t,p i )=aA(x,y,p i )+bB(x,y,p i )+cC(x,p i )-dD(t,p i )

[0054] After finding the final priority of all nodes, the Dijkstra algorithm is used to calculate the shortest path from the current position to each node, thereby planning an optimal search path that traverses all path points.

[0055] The target detection module is responsible for detecting whether a navigation target appears in the camera's field of view during operation. This module provides two target detection methods, one is open semantic detection, which uses the open semantic target detection model Grounding-Vino, and the other is conventional target detection, which uses the YOLO11 series model. Among them, the open semantic target detection model Grounding-Vino is too large to be directly deployed on the host of the mobile robot, so it is implemented by server deployment and remote call. After deployment on RTX3090, a single detection takes 0.5s, so open detection cannot be performed in real time, and can only stop for detection after reaching the path point. The YOLO11 series model can be deployed on a mobile robot and perform real-time detection. Since the target needs to be located, the system needs to be used with a depth camera. The depth camera used in this example is RealSense D435i. The deep camera can obtain the point cloud corresponding to the image. When the target is detected, the module will segment the target area through the instance segmentation model. Since the segmentation model cannot achieve perfect segmentation, the initially segmented point cloud often contains irrelevant points. It is necessary to first perform downsampling and independent point removal, and then perform DBSCAN clustering. The cluster with the largest number of points is regarded as the target surface point cloud, and then calculate the directed bounding box of the point cloud. The center of the bounding box is used as the target position in the camera coordinate system, and finally the coordinate transformation is performed to obtain the position of the target in the map.

[0056] like Figure 6As shown, the navigation method of the mobile robot target navigation system in this embodiment is as follows:

[0057] First, use the map construction module to create a three-dimensional point cloud map, a two-dimensional grid map and an environmental topology map. For the same environment, it only needs to be built once, and the three-dimensional point cloud map and the two-dimensional grid map are used as the input of the basic navigation module to activate the basic navigation function. The two-dimensional grid map and the three-dimensional point cloud map correspond to the same coordinate system. In the navigation task, the coordinate system corresponding to the two-dimensional grid map will be used as the map coordinate system, and the two-dimensional grid map will be used as the global map. Then, the natural language input is converted into navigation targets, auxiliary information and final distances through the instruction analysis module, and a suitable search path, that is, the optimal search path, is planned using the path planning module. The basic navigation module is used to go to the planned path points in sequence, and the target detection module is started to detect the search target. When the target is detected, it will stop moving and obtain the position of the target in the camera coordinate system through the target detection module in a stationary state. According to the relative posture of the camera coordinate system and the robot coordinate system, the target position is first converted from the camera coordinate system to the robot coordinate system, and then the basic navigation module is used to obtain the robot's position in the map coordinate system, and then the target position is converted from the robot coordinate system to the map coordinate system, and combined with the preset distance, a suitable stop point is determined on the map coordinate system, and finally the basic navigation module is used to guide the mobile robot to the stop point, thereby completing a single navigation task.

[0058] In summary, the present invention combines a large language model with an environmental topology map to realize a target navigation system, which is built on ROS2. It can improve the target navigation effect in a known environment, solve the shortcomings of existing target navigation algorithms that do not consider deployment and robot kinematic characteristics, and make up for the shortcomings of existing target navigation algorithms that do not consider the distance to the final target. It has extensive research and practical application value and is worth promoting.

[0059] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A mobile robot target navigation system, characterized in that: include: The map construction module is responsible for building a 3D point cloud map based on the point cloud data collected by multi-line lasers, projecting the 3D point cloud map into a 2D grid map, and collecting semantic information and visual information at the set path points, thereby realizing the construction of an environmental topology map. The environmental topology map refers to the abstraction of the environment into a topology map, and the nodes of the topology map record the semantic information and visual information of specific locations in the environment. The basic navigation module uses two-dimensional grid maps and three-dimensional point cloud maps for global positioning and fixed-point navigation, encapsulates basic navigation functions, provides interfaces, and records operation logs; The instruction analysis module analyzes the natural language input by the user based on the large language model and extracts effective navigation targets and auxiliary information; The path planning module plans a suitable navigation path in the scene based on the navigation target, auxiliary information and environmental topology map output by the instruction analysis module based on the large language model, that is, the optimal search path; The target detection module is responsible for detecting targets in visual information while searching along the navigation path, and calculating the position of objects based on the detection results and depth information.

2. A mobile robot target navigation system according to claim 1, characterized in that: The map construction module involves three types of maps, including three-dimensional point cloud maps, two-dimensional grid maps and environmental topology maps; among them, the three-dimensional point cloud map is used for the global positioning of the mobile robot, and it uses point clouds to accurately describe environmental information; the two-dimensional grid map is used for obstacle avoidance planning of the mobile robot, and it records obstacles in the environment from a top-down perspective; the environmental topology map is used for planning and finding paths, and it is an abstraction of the environment, consisting of nodes and edges. In the environmental topology map, nodes represent an area, and edges represent the connectivity and distance between two areas. Nodes can record the visual information, semantic information and location of a certain area. The above three maps all use a unified map coordinate system.

3. A mobile robot target navigation system according to claim 2, characterized in that: The map construction module specifically performs the following operations: First, all laser beams that can cover the entire height of the mobile robot are selected, and a three-dimensional point cloud map is established using a three-dimensional SLAM algorithm. The three-dimensional point cloud map is then reduced to a two-dimensional grid map. Then, according to the manually set path points, the map is divided into different navigation areas, and then the path points are taken in turn to take photos of the surrounding environment. Then, the visual language model VLM is used to process the photos of the surrounding environment, and the description information of the surrounding environment is summarized. The labels of the path points are summarized in combination with the description information or manually set labels, where the description information and labels are semantic information, and the photos taken are visual information; then, the same number of nodes are created in the environmental topology map to represent the path points, and the semantic information and visual information contained in the path points are recorded in the corresponding nodes in the environmental topology map. Finally, edges are set between the nodes according to the connectivity between the path points, and the distances between the nodes are calculated according to the two-dimensional grid map and assigned to the corresponding edges, thereby constructing a complete environmental topology map.

4. A mobile robot target navigation system according to claim 3, characterized in that: The basic navigation module uses the Navigation2 navigation framework in the robot framework ROS2 as the underlying implementation; the basic navigation module uses the three-dimensional point cloud map and the two-dimensional grid map created in the map construction module for positioning and navigation, and provides the following basic navigation functions: fixed-point navigation, navigation along trajectory points, movement of a specified distance forward and backward, rotation at a specified angle, and providing real-time posture information for the mobile robot during navigation; the basic navigation module converts the multiple feedback and waiting calls in ROS2 into a calling method that notifies with a callback function. The caller does not need to consider the specific process in ROS2, but only needs to give an instruction, and the parameters of the callback function will eventually tell whether the instruction is executed successfully. In addition, the calling process is transmitted in JSON format data and recorded using the log system in ROS2.

5. A mobile robot target navigation system according to claim 4, characterized in that: The instruction analysis module receives natural language input and enables the large language model to extract navigation targets, auxiliary information and distance information in the natural language by giving pre-adjusted prompt words, wherein the navigation target is the final object to be found, the auxiliary information is additional information to help find the object, and the distance information represents the specified distance that the mobile robot needs to move to the target after finding the target; the initial navigation target may not be directly detected by the target detection model, so after obtaining the initial navigation target, the large language model will be used again to convert the navigation target into the target category that is semantically closest and can be detected.

6. A mobile robot target navigation system according to claim 5, characterized in that: The path planning module receives the navigation target x, auxiliary information y, the initial position t of the robot in the map coordinate system and the environment topology map m. Each node in the environment topology map m represents an area in the environment. The node stores the visual information and semantic information of the area, where the semantic information includes the description of the area and the label corresponding to the area. Let [p1, p2, p3, ..., p i ,p (i+1) ,...] is a node in the environment topology graph m, p i is the i-th node in the environment topology map, and is also a path point. The path planning module needs to calculate the following four values ​​to plan the final path: the similarity between the node and the navigation target x, the probability value of the navigation target x being near the node, the matching degree between the navigation target x and the visual information of each node, and the distance between each node and the position t. The specific situation is as follows: First, the large language model is used to analyze the correlation between the navigation target x and the node label in the environment topology map m, and the auxiliary information y is combined to make corrections, and finally the similarity between the node and the navigation target x is given. The specific method is as follows: collect the labels of all nodes, send the node labels, navigation target x and auxiliary information y into the large language model, and use the prompt words to guide the large language model to compare the similarity between the navigation target x and each label under the given auxiliary information y, where the similarity is a value between 0 and 1, and then assign the similarity of each label to the node containing the label, that is, p i The sum of all label similarities is p i The similarity A(x,y,p) with the navigation target x under given auxiliary information y i ); Subsequently, the large language model is used again to determine the correlation between the description of the navigation target x and the node in the environment topology map m, and is corrected in combination with the auxiliary information y, thereby giving the probability value of the navigation target x being near the node. The specific method is as follows: the descriptions corresponding to all nodes, the navigation target x and the auxiliary information y are sent to the large language model, and the large language model is guided by prompt words to analyze whether the navigation target x is near a certain node when the auxiliary information y is given, and whether it meets the description of the node. Finally, the large language model outputs a value between 0 and 1 for each node to represent the probability value of the navigation target being near the node. Suppose that under the given auxiliary information y, the navigation target x is near p i The probability of nearby is B(x,y,p i ); Next, the visual text matching model CLIP is used to match the navigation target x with the visual information of each node in the environment topology map m, and then the matching degree between the navigation target x and the visual information of each node is given. The specific method is: the visual information of all nodes, that is, the pictures taken at the nodes, and the navigation target x are used as the input of the visual text matching model CLIP. The output of the visual text matching model CLIP is the similarity between each picture and the navigation target x, then p i The sum of the similarities between all corresponding images and the navigation target x is the similarity between the navigation target x and p i The visual similarity C(x,p i ), and calculate the distance between each node and position t, let p i The distance from position t is D(t,p i ); Finally, perform a weighted sum operation on the above four values ​​to get p i The final priority of the node is, let a be the weight of the similarity between the node and the navigation target, b be the probability value weight of the navigation target being near the node, c be the weight of the matching degree between the navigation target and the visual information in the environment topology map, d be the weight of the distance between the current position and each node in the environment topology map, then the path point p i The final priority S(x,y,t,p i ) is: S(x,y,t,p i )=aA(x,y,p i )+bB(x,y,p i )+cC(x,p i )-dD(t,p i ) After finding the final priority of all nodes, the Dijkstra algorithm is used to calculate the shortest path from the current position to each node, thereby planning an optimal search path that traverses all path points.

7. A mobile robot target navigation system according to claim 6, characterized in that: The target detection module is implemented using a depth camera, including visual detection and position estimation, as follows: Visual detection provides two target detection methods. The first is open semantic detection, which uses the open semantic target detection model Grounding-Vino. Open semantic detection can detect more categories and distinguish objects based on their position and color. Since the open semantic target detection model Grounding-Vino cannot be deployed on embedded platforms, it is deployed on a server and implemented by remote calling. The second is conventional target detection, which uses the YOLO11 series model and can only detect 80 predefined categories. Since the YOLO11 series model is lightweight, it can be directly deployed on mobile robots. After the target is detected successfully, the position is estimated. Specifically, the depth camera is used to obtain point cloud information, and the instance segmentation model is used to divide the point cloud in the target area. Since the instance segmentation model cannot perfectly segment the target, the point cloud segmented for the first time will have a large error, and the point cloud needs to be preprocessed. The processing flow is: remove independent points of the point cloud, downsample, cluster using the DBSCAN algorithm and select the largest category, so as to obtain accurate target point cloud information, and then calculate the directed bounding box of the point cloud to obtain the position of the target in the camera coordinate system.

8. The navigation method of the mobile robot target navigation system according to any one of claims 1 to 7, characterized in that: First, the map construction module is used to create a three-dimensional point cloud map, a two-dimensional grid map and an environmental topology map. For the same environment, it only needs to be constructed once, and the three-dimensional point cloud map and the two-dimensional grid map are used as the input of the basic navigation module to activate the basic navigation function. Then, the natural language input is converted into navigation targets, auxiliary information and final distances through the instruction analysis module, and a suitable search path, that is, the optimal search path, is planned using the path planning module. The basic navigation module is used to go to the planned path points in sequence, and the target detection module is started to detect the search target. When the target is detected, the mobile action will be stopped, and the target position in the camera coordinate system will be obtained by the target detection module in a stationary state. According to the relative position of the camera coordinate system and the robot coordinate system, the target position is first converted from the camera coordinate system to the robot coordinate system, and then the basic navigation module is used to obtain the robot's position in the map coordinate system, and then the target position is converted from the robot coordinate system to the map coordinate system. Combined with the preset distance, a suitable stop point is determined on the map coordinate system. Finally, the mobile robot is guided to the stop point through the basic navigation module to complete a single navigation task.

Images