Mobile robot for planning path based on large language model and movement control method
Through a mobile robot system based on large language model, the problems of feature points limitations and inaccuracy in SLAM technology are solved, and more accurate and robust mobile planning is achieved, robot navigation performance and computing efficiency are improved, and human-computer Interaction and technical applications.
Patent Information
- Application Number
- CN202411031276.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-15
AI Technical Summary
The existing SLAM technology has limitations, inaccuracy and high computing power requirements in mobile planning, resulting in a decline in robot navigation and positioning performance, limiting its application in complex environments.
A mobile robot system based on a large language model is adopted to obtain image and motion data through cameras and IMU sensors, combined with NeRF neural network algorithm, maps are generated and planned using natural language processing to achieve more accurate and robust mobile path planning.
It improves robots' understanding of the environment and navigation accuracy, reduces the demand for computing resources, enhances reliability and controllability in complex environments, and promotes the popularization and application of human-computer interaction and robotics technology.
Smart Images

Figure CN118960740A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot control technology, and in particular to a mobile robot and a mobile control method for planning a path based on a large language model. Background Art
[0002] One of the main challenges currently faced by the field of mobile robotics technology is the technical problems existing in its mobility planning. Currently, the commonly used robot mobility planning methods mainly rely on simultaneous localization and mapping (SLAM) technology. However, the current SLAM technology is mainly based on image feature points, through which environmental maps are formed. There are some key problems with this approach, one of which is the limitation of the feature points themselves, which lack a deep understanding of the scene. Since feature points are only abstract points on the image, the robot is limited in understanding the substantive information of the environment, which makes the accuracy of mobility planning challenging.
[0003] In addition, another limitation of current SLAM technology is the inaccuracy it may have in practical applications. Since the selection and tracking of feature points may be affected by lighting, occlusion and environmental changes, there are errors in the construction of the map. This inaccuracy may have a negative impact on the robot's navigation and positioning, reducing its performance and reliability in complex environments.
[0004] In addition, the high computing power requirements of current SLAM technology are also a significant problem. The complex calculations in the image processing and map construction process require a lot of computing resources, which may be a constraint for some mobile robots and limit their widespread application in practical applications. Therefore, seeking a more accurate, robust and computationally efficient mobile robot motion planning method is an urgent task for current research and development. Therefore, the existing technology needs to be improved and enhanced.
[0005] Therefore, the prior art has defects and needs to be improved. Summary of the invention
[0006] The technical problem to be solved by the present invention is to provide a mobile robot and a mobile control method for planning paths based on a large language model, so as to solve the technical problem mentioned in the background technology.
[0007] The technical solution of the present invention is as follows: a mobile robot for planning a path based on a large language model is provided, comprising: a camera, an IMU sensor, a mobile device in the hardware part, and an image recognition unit, an odometer, a map generator, a high-level planner, a low-level planner and a large language model in the control part. The IMU sensor is an image and motion perception sensor.
[0008] The camera transmits the current image to the image recognition unit, which performs segmentation and recognition on the current image, identifies the outline and classification of objects, such as beds, tables, etc., and related location information; at the same time, the camera's image and the data of the motion sensing IMU sensor are sent to the odometer; the odometer calculates the current position movement information in real time; preferably, the fusion algorithm of the visual odometer and IMU is used. The map generator collects the calculated data of the image recognition unit and the odometer, and gradually generates a map based on the location information, and updates and maintains the map; the map description uses language to update the database of the large language model in real time.
[0009] The high-level planner interacts with the large language model based on the map data and real-time location information of the map generator and the task instructions received by the mobile robot, describes the current state and requests new information or instructions; after obtaining the content or instructions issued by the large language, it converts the mobile instructions into specific mobile task requirements, which are then sent to the low-level planner to be converted into motor drive instructions to drive the mobile device to complete the mobile task.
[0010] Combining the azimuth movement and angle change of the mobile robot, multi-angle image data, and NeRF (Neutral Radiance Field) and other neural network algorithms, the outline and relative position of the classified object are calculated to form the basic coordinates of the area occupied by the object, and the area occupied by the object is formed on the plane map. For example, [(x1, y1), (x2, y2), (x3, y3), (x4, y4)]. In this way, the area occupied by the object can be formed on the plane map.
[0011] A mobile control method adopts the aforementioned mobile robot that plans paths based on a large language model, and its workflow includes the following steps.
[0012] S1: The robot receives an instruction to move toward a certain target.
[0013] S2: Obtain the camera image of the current location and perform image analysis.
[0014] S3: Use language to consult the large language model and obtain the current position information.
[0015] S4: Based on the current position, consult the large language model and obtain the instructions for the current move.
[0016] S5: Determine whether the current movement instruction is completed; if not, re-enter step S2; if yes, enter step S6.
[0017] S6: Determine whether the current moving instruction is completed, that is, whether the target has been reached; if not, proceed to step S2; if yes, end.
[0018] A map plan is generated by a map generator; the plan includes the four corner coordinates of each object in the plan; a data structure is expressed using a map language described in a language; the data structure is used to reflect the connection relationship.
[0019] The following is the process of creating a map.
[0020] A1: The robot is placed in the initial room, and receives an instruction: move to the target room and shoot a video for several seconds. The target room is a certain room or all rooms.
[0021] A2: The robot in the initial room recognizes the current image and consults the large language model. The large language model gives the direction of movement, and the robot drives into a room according to the direction of movement.
[0022] A3: After the robot moves to the room, it recognizes the current image and consults the large language model.
[0023] A4: The natural language processing (NLP) function of the large language model analyzes the description provided by the robot, determines whether the room is the target room, and returns a language instruction; if the large language model determines that the room is the target room, the language instruction returned to the robot is: take pictures and return after completion; go to step A7; if the large language model determines that the room is not the target room, the language instruction returned to the robot is: the current location is not the target room, you need to find the exit and pass through; go to step A5.
[0024] A5: The robot converts the language instruction into actual action of the robot through the high-level planner 110 and the low-level planner 125; according to the instruction, the robot rotates, recognizes the current image and locks the exit, moves through the exit, and submits the information obtained in this process to the map generator 105, so that it can gradually build and improve the map.
[0025] A6: After step A5 is completed, the robot passes through the exit, recognizes the current image, and consults the large language model. The large language model gives the direction of movement. The robot drives into the next room according to the direction of movement and enters step A3.
[0026] A7: Mission completed.
[0027] Each time the robot enters a room, it marks the current position as the coordinate zero point (0, 0), and the initial angle is also 0. Alternatively, the robot marks the current position in the initial room as the coordinate zero point (0, 0), and the initial angle is also 0.
[0028] After the map is established, the large language model plans the robot's movement trajectory more effectively based on the map.
[0029] In step A4, if the large language model determines that the room is the target room, the language instruction returned to the robot also includes: movement end.
[0030] By adopting the above scheme, the present invention provides a mobile robot and a mobile control method for planning a path based on a large language model, which has the following beneficial effects:
[0031] (1) The training of large language models enables machines to understand more complex and abstract language structures, not only recognizing the superficial meaning of words, but also understanding the logic and common sense contained therein. This provides a richer information basis for robots to reason and make decisions in real environments. By using natural language as input, robots can more comprehensively understand human intentions and needs, and thus perform tasks more intelligently.
[0032] (2) The language-based control of robots also has a certain degree of certainty. Compared with traditional image processing-based methods, language provides a more structured and standardized input method. Through reasonable language expression, tolerance for ambiguity and ambiguity can be reduced, thereby improving the controllability and predictability of robot behavior. This is a very valuable control method for robots that need to perform tasks in a specific environment.
[0033] (3) Language-based robot control also helps bridge the communication barrier of human-machine interaction. Users can communicate with robots through natural language without having to understand complex technical details. This makes robot technology more popular and can be more widely used in daily life and industrial production.
[0034] (4) With the help of large language models and language-controlled robots, more intelligent, flexible and controllable robot systems can be realized, promoting the widespread application of artificial intelligence technology in real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A framework diagram of an embodiment of a mobile robot based on large language model path planning provided by the present invention.
[0036] Figure 2A An embodiment of capturing images by a robot camera.
[0037] Figure 2B For the corresponding Figure 2A The original image is segmented and classified into object categories described by language.
[0038] Figure 3The present invention provides a workflow of a mobile robot based on large language model path planning.
[0039] Figure 4 An example of map formation by a mobile robot based on large language model path planning provided by the present invention.
[0040] Figure 5 A schematic diagram of the structure of the map language expression of a mobile robot based on large language model path planning provided by the present invention.
[0041] Figure 6 A flowchart of an embodiment of a map-free exploration mode of a mobile robot based on large language model path planning provided by the present invention.
[0042] Figure 7 A flowchart of an embodiment of a mobile robot's map patrol mode based on large language model path planning provided by the present invention. DETAILED DESCRIPTION
[0043] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0044] Please refer to Figure 1 This embodiment provides a mobile robot that plans a path based on a large language model. Figure 1 The framework diagram of the mobile robot based on large language model path planning provided by the present invention. The robot comprises:
[0045] The hardware part includes the camera 160, IMU sensor 161, mobile device 162, and the control part includes the image recognition unit 115, odometer 135, map generator 105, high-level planner 110, low-level planner 125 and large language model 101.
[0046] The camera 160 transmits the current image to the image recognition unit 115, performs segmentation and recognition on the current image, identifies the outline and classification of objects, such as beds, tables, etc., and related location information. At the same time, the image of the camera 160 and the data of the motion sensing IMU sensor 161 are sent to the odometer 135. The odometer 135 calculates the current position movement information in real time. This embodiment uses a fusion algorithm of visual odometer (Visual Odometry) and IMU. The map generator 105 collects the calculated data of the image recognition unit 115 and the odometer 135, and gradually generates a map based on the location information, and updates and maintains the map. The map description updates the knowledge base of the large language model 101 in real time through language.
[0047] The high-level planner 110 interacts with the large language model 101 based on the map data and real-time location information of the map generator 105 and the task instructions received by the robot, describes the current state and requests new information or instructions. After obtaining the content or instructions issued by the large language 101, the mobile quality is converted into specific mobile task requirements, which are then sent to the low-level planner 125 to be converted into motor drive instructions to drive the mobile device 162 to complete the mobile task.
[0048] Specifically, Figure 2A is the image of the current robot camera, showing that there is a bed 1101V and a table 1104V in the field of view. After the image recognition unit 115 segments and classifies the image, Figure 2B As shown, corresponding to the original image, the image is segmented and classified and labeled into object categories described by language, such as single bed 1101, small table 1104, etc.
[0049] Furthermore, by combining the azimuth movement and angle change of the mobile robot, multi-angle image data, and neural network algorithms such as NeRF (Neutral Radiance Field), the outline and relative position of the classified object can be calculated to form the basic coordinates of the area occupied by the object, such as [(x1, y1), (x2, y2), (x3, y3), (x4, y4)]. In this way, the area occupied by the object can be formed on the plane map.
[0050] When the robot can describe the scene it sees with words, the large language model 101 can use language to guide the robot to complete many tasks, including moving to the target.
[0051] Figure 3 The invention provides a workflow of a mobile robot based on large language model path planning, that is, a robot movement control method, using the aforementioned mobile robot based on large language model path planning, and its workflow includes the following steps.
[0052] S1: The robot receives an instruction to move toward a certain target.
[0053] S2: Obtain the camera image of the current location and perform image analysis.
[0054] S3: Use language to consult the large language model and obtain the current position information.
[0055] S4: Based on the current position, consult the large language model and obtain the instructions for the current move.
[0056] S5: Determine whether the current movement instruction is completed; if not, re-enter step S2; if yes, enter step S6.
[0057] S6: Determine whether the current moving instruction is completed, that is, whether the target has been reached; if not, proceed to step S2; if yes, end.
[0058] Figure 4 It is a schematic diagram of the floor plan of the home map generated by the map generator 105 of the robot. Correspondingly, the robot generates a map language description data structure 501 using a language description. The map includes a bedroom 401, which includes a door D1, a single bed 1101, a bedside table 1102, a wardrobe 1103, a small table 1104, etc. It includes the four corner coordinates of each object in the floor plan. The data structure reflects the connection relationship, such as door D1 is connected to aisle 406; aisle 406 is connected to door D1, door D2 and living room 403; living room 403 is connected to door D4, door D5, etc.
[0059] In the actual operation of the robot, there are two situations: the map has been established and the map has not been established. Figure 6 As shown, an example of the process of the robot working in the map-free exploration mode 601.
[0060] The robot receives the command: "Move to the living room and shoot a 13-second video."
[0061] The robot identifies the current position as the coordinate zero point (0, 0), and the initial angle is also 0. The robot recognizes the current image and consults the large language model 101 .
[0062] "There is a table in front of me in my current field of vision. There is a stool on the left side of the table, a bed 40 degrees to the left, a small bedside table on the right side of the bed, a wardrobe on the left side of the bed, a photo frame above the bed, and three decorations on the left side of the photo frame. What is my current location? I need to move to the living room."
[0063] The natural language processing (NLP) function of the large language model can analyze the description provided by the robot and return the correct instructions: "According to the vision, the current location is in the bedroom. You need to find the door and pass through it."
[0064] The robot converts the language instruction into actual actions of the robot through the high-level planner 110 and the low-level planner 125. According to the previous instruction, the robot performs rotation, image recognition and locks the door, moves through the door, and submits the information obtained in this process to the map generator 105, so that it gradually builds and improves the map.
[0065] After the last action is completed, the robot recognizes the current image and consults the large language model 101 again. "In the current field of view, there is a high table with a picture frame on the top at 90 degrees, and a table and several chairs at 270 degrees. What is the next direction to move?"
[0066] Consult the Big Language Model 101 and it will reply: "Based on vision, the 90-degree direction is a dead end. Move 2 meters to the 270-degree direction."
[0067] According to the command, the robot moves 2 meters in the direction of 270 degrees. Then it repeats the previous steps, recognizes the current image and consults the large language model 101 again. "In the current field of view, there is a table and some chairs in front, a sofa at 30 degrees to the left, and two doors on the right. What is the next direction to move?"
[0068] The large language model 101 believes that the robot has reached the predetermined goal based on the description, and then responds: "Movement completed."
[0069] The robot completes the movement command, takes the photo, and returns after completion. The mission is over.
[0070] like Figure 7 As shown, an example of the process of the robot working in the map cruise mode 701.
[0071] After the map is established, the large language model 101 can plan the robot's movement trajectory more effectively according to the map.
[0072] At this time, the large language model 101 does not need to let the robot implement the exploration mode. When the robot consults the large language model 101 for the first time, its response will be more accurate and effective. "According to vision, the current location is bedroom 1. You need to find the door, pass through it, and move 2 meters in the 90-degree direction."
[0073] In summary, the present invention provides a mobile robot and a mobile control method for planning a path based on a large language model, which have the following beneficial effects:
[0074] (1) The training of large language models enables machines to understand more complex and abstract language structures, not only recognizing the superficial meaning of words, but also understanding the logic and common sense contained therein. This provides a richer information basis for robots to reason and make decisions in real environments. By using natural language as input, robots can more comprehensively understand human intentions and needs, and thus perform tasks more intelligently.
[0075] (2) The language-based control of robots also has a certain degree of certainty. Compared with traditional image processing-based methods, language provides a more structured and standardized input method. Through reasonable language expression, tolerance for ambiguity and ambiguity can be reduced, thereby improving the controllability and predictability of robot behavior. This is a very valuable control method for robots that need to perform tasks in a specific environment.
[0076] (3) Language-based robot control also helps bridge the communication barrier of human-machine interaction. Users can communicate with robots through natural language without having to understand complex technical details. This makes robot technology more popular and can be more widely used in daily life and industrial production.
[0077] (4) With the help of large language models and language-controlled robots, more intelligent, flexible and controllable robot systems can be realized, promoting the widespread application of artificial intelligence technology in real-world scenarios.
[0078] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A mobile robot that plans a path based on a large language model, characterized in that: include: The hardware part includes the camera, IMU sensor, mobile device, and the control part includes the image recognition unit, odometer, map generator, high-level planner, low-level planner, and large language model; The camera transmits the current image to the image recognition unit, which segments and recognizes the current image, identifies the outline and classification of the object, and related location information; at the same time, the data of the camera's IMU sensor is sent to the odometer; The odometer calculates the current position movement information in real time; The map generator collects the calculation data of the image recognition unit and the odometer, and gradually generates a map based on the location information, and updates and maintains the map; the map description uses language to update the database of the large language model in real time; The high-level planner interacts with the large language model based on the map data and real-time location information of the map generator and the task instructions received by the mobile robot to describe the current state and request new information or instructions. After obtaining the content or instructions issued by the large language, the mobile instructions are converted into specific mobile task requirements, which are then sent to the low-level planner to be converted into motor drive instructions to drive the mobile device to complete the mobile task.
2. A mobile robot for planning a path based on a large language model according to claim 1, characterized in that: Combining the mobile robot's azimuth movement, angle changes, and multi-angle image data, a neural network algorithm is used to calculate the classified object outline and relative position to form the basic coordinates of the object's footprint, thereby forming the area occupied by the object on a plane map.
3. A mobile control method, characterized in that: The mobile robot for planning a path based on a large language model according to claim 1 or 2 has a workflow comprising the following steps: S1: The robot receives an instruction and needs to move toward a certain target; S2: Obtain the camera image of the current location and perform image analysis; S3: Use language to consult the large language model and obtain current position information; S4: Based on the current position, consult the large language model and obtain the current movement instructions; S5: Determine whether the current movement instruction is completed; if not, re-enter step S2; if yes, proceed to step S6; S6: Determine whether the current moving instruction is completed, that is, whether the target has been reached; if not, proceed to step S2; if yes, end.
4. A mobile control method according to claim 3, characterized in that: A map generator generates a map plan; the plan includes the four-corner coordinates of each object in the plan; a map language described in a language is used to express a data structure; the data structure is used to reflect the connection relationship.
5. A mobile control method according to claim 3, characterized in that: The process of creating a map is: A1: The robot is placed in the initial room. The robot receives instructions: move to the target room and shoot a few seconds of video. A2: The robot in the initial room recognizes the current image and consults the large language model. The large language model gives the direction of movement, and the robot drives into a room according to the direction of movement; A3: After the robot moves to the room, it recognizes the current image and consults the large language model; A4: The natural language processing (NLP) function of the large language model analyzes the description provided by the robot, determines whether the room is the target room, and returns language instructions; If the large language model determines that the room is the target room, the language instruction returned to the robot is: take pictures and return after completion; go to step A7; If the large language model determines that the room is not the target room, the language instruction returned to the robot is: the current position is not the target room, and the exit needs to be found and passed; go to step A5; A5: The robot converts the language instruction into actual actions of the robot through the high-level planner 110 and the low-level planner 125; according to the instruction, the robot rotates, recognizes the current image and locks the exit, moves through the exit, and submits the information obtained in this process to the map generator 105, so that the map is gradually established and improved; A6: After step A5 is completed, the robot passes through the exit, recognizes the current image, and consults the large language model. The large language model gives the direction of movement. The robot drives into the next room according to the direction of movement and enters step A3; A7: Mission completed.
6. A mobile control method according to claim 5, characterized in that: Every time the robot enters a room, it marks its current position as the coordinate zero point (0, 0), and the initial angle is also 0.
7. A mobile control method according to claim 5, characterized in that: The robot marks its current position in the initial room as the coordinate zero point (0, 0), and the initial angle is also 0.
8. A mobile control method according to claim 5, characterized in that: After the map is established, the large language model plans the robot's movement trajectory more effectively based on the map.
9. A mobile control method according to claim 5, characterized in that: In step A4, if the large language model determines that the room is the target room, the language instruction returned to the robot also includes: movement end.
10. A mobile control method according to claim 5, characterized in that: The target room is a certain room or all rooms.
Citation Information
Cited By
Robot autonomous exploration mapping method, device and equipment and medium
CN121297816A
