Robot path planning perception method and system based on large language model

Abstract

The invention discloses a robot path planning perception method and system based on a large language model. The method comprises the following steps: acquiring an initial data set; selecting an SLAM algorithm to process the initial data set; performing rasterization processing on the first map; performing semantic annotation processing on the second map; performing path planning according to the large language model to generate a path planning scheme; performing conversion according to the path planning scheme to obtain execution information of the robot; target sensing information and environment information of the robot in the moving process are obtained, and a motion planning adjustment scheme of the robot is generated through a large language model according to the target sensing information and the environment information; and a man-machine interaction module is constructed to realize information interaction between the user and the robot. According to the embodiment of the invention, a more reliable and smoother path can be generated, the navigation and perception capabilities of the robot in a complex environment are improved, and the method can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a robot path planning perception method and system based on a large language model. Background Art

[0002] At present, large language models (LLMs) have made significant progress in language understanding and reasoning, but they also face challenges in applying them to robot navigation and perception, such as how to enable robots to understand the environment and perform reliable path planning. Existing research mainly focuses on using LLMs for language tasks or as decision-making agents, lacking an end-to-end framework that inputs relevant natural language instructions from users and map information perceived by the robot itself, outputs a path to the target location, and completes the corresponding perception task. Traditional motion planning algorithms, such as BFS and A* algorithms, have problems such as large computational complexity and uneven paths in complex environments. In addition, the challenge of virtual-to-real migration also limits the performance of robots in real environments. Therefore, it is necessary to develop a robot planning and perception framework that can combine the advantages of LLMs and solve the limitations of existing technologies to achieve the goal of automatically completing navigation and perception tasks based on human instructions.

[0003] Existing robot planning and perception technologies face problems such as the lack of full integration of LLMs’ semantic understanding and decision-making capabilities for robot navigation and perception tasks, limitations of traditional planning algorithms, challenges in virtual-to-real migration, and insufficient planning capabilities of large language models (LLMs). Traditional path planning algorithms are inefficient in complex environments and the generated paths are not smooth. The difference between the simulated environment and the real environment also makes it difficult for robots to apply learned strategies to actual scenarios. In addition, how to fully utilize the understanding and reasoning capabilities of LLMs to generate better path planning solutions is also a challenge. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to propose a robot path planning perception method and system based on a large language model, which can generate a more reliable and smoother path and improve the robot's navigation and perception capabilities in complex environments.

[0005] To achieve the above object, an embodiment of the present invention provides a robot path planning perception method based on a large language model, comprising the following steps:

[0006] Acquire an initial data set, wherein the initial data set includes robot sensor information and robot posture information;

[0007] Selecting a SLAM algorithm according to an actual application scenario to process the initial data set to obtain a first map constructed by SLAM;

[0008] rasterizing the first map to obtain a second map;

[0009] Performing semantic annotation processing on the second map to obtain a third map;

[0010] According to the large language model, path planning is performed on the third map after the semantic annotation processing is completed to generate a path planning solution;

[0011] Obtaining execution information of the robot according to the path planning scheme, wherein the execution information is used to control the robot to move along the target path using corresponding motion parameters;

[0012] Acquire target perception information and environmental information of the robot during movement, and generate a motion planning adjustment plan of the robot through the large language model according to the target perception information and the environmental information;

[0013] Build a human-computer interaction module to realize information interaction between users and robots.

[0014] In some embodiments, the process of selecting a SLAM algorithm according to an actual application scenario to process the initial data set to obtain a first map constructed by SLAM includes the following steps:

[0015] Selecting a Karto SLAM algorithm, and adjusting parameters of the Karto SLAM algorithm to obtain an optimal parameter combination; the parameter adjustment includes but is not limited to adjusting a node adding strategy and adjusting a threshold of loop detection;

[0016] According to the adjusted optimal parameter combination, the initial data set is processed by the Karto SLAM algorithm to obtain a first map constructed by SLAM;

[0017] Wherein, the adjusting node adding strategy includes selecting whether to add a node according to the distance between nodes;

[0018] The adjusting the loop detection threshold includes: determining whether a loop occurs according to point cloud similarity.

[0019] In some embodiments, the rasterizing the first map to obtain the second map comprises the following steps:

[0020] Dividing the continuous map space on the first map into grid units of equal size, each grid unit representing an area on the map, and converting the first map into a grid map of a specified resolution;

[0021] The grid map is binarized, and the value of each grid cell is converted into 0 or 1 to obtain a second map.

[0022] In some embodiments, performing semantic annotation processing on the second map to obtain a third map includes the following steps:

[0023] For the second map, each grid cell is assigned an important semantic label; specifically:

[0024] The Yo lov8 target detection model is used as the robot perception module algorithm. Through the calibration relationship between the camera and the lidar, the positions of important objects in reality are projected onto the grid map. Finally, each grid unit is given a corresponding semantic label and a passable attribute.

[0025] In some embodiments, performing path planning on the third map after semantic annotation processing according to the large language model to generate a path planning solution includes the following steps:

[0026] converting the binary map of the second map into a natural language description;

[0027] Determine the robot's mission objectives based on the user's preference information or control instructions;

[0028] According to the task objective, a few-shot refine method is introduced into the large language model to iteratively correct the path planning scheme to obtain the final path planning scheme. The specific step is as follows: corresponding examples are added to the prompt words. When the planning result output by the large language model is judged to be wrong, the path planning is continued until the planning result is judged to be correct.

[0029] In some embodiments, the adding of corresponding examples to the prompt word includes:

[0030] Design multiple representative path planning cases, which include factors of different map types, obstacle distributions, and target locations;

[0031] Obtain path planning examples through expert design and annotation; or automatically generate path planning examples through path planning algorithms;

[0032] The method of automatically generating a path planning case by using a path planning algorithm comprises the following steps:

[0033] Convert the automatically generated path planning cases into a format that the large language model can understand. Use the few sample cases as learning materials to guide the large language model to perform path planning. Input the map information, starting point, and end point information as prompts to the large language model. The large language model generates an initial path planning solution based on the learned knowledge.

[0034] The rationality of the initial path planning scheme is evaluated, specifically: a series of judgment rules are defined to determine whether there are problems with the scheme, wherein the judgment rules include whether a collision occurs, whether a boundary is exceeded, and whether the path length is too long; the map information and the initial path planning scheme are analyzed by using the reasoning ability of the large language model to determine whether the scheme meets the predefined rules; if the self-reflection process of the large language model finds that there are problems with the path planning scheme, the identified problems are converted into natural language descriptions; feedback information is added to the prompt of the large language model, and the large language model is required to regenerate the path planning scheme;

[0035] During the iterative process of the Few-shot refine method, user instructions, conversation records, and good path planning cases are obtained to build a knowledge base for iterative optimization.

[0036] In some embodiments, converting the execution information of the robot according to the path planning scheme comprises the following steps:

[0037] Perform coordinate system conversion to convert the coordinate points in the path planning solution from the grid map coordinate system to the robot's own coordinate system, where the coordinate system conversion is calculated using the robot's kinematic model;

[0038] Smoothing the converted path, including using Bezier curve processing or cubic spline interpolation processing;

[0039] Select a local path planning algorithm for local path planning based on the actual application scenario, and set motion parameters based on the robot's physical constraints and environmental characteristics;

[0040] Obtaining obstacle information in the surrounding environment in real time, and integrating the obstacle information into the selected local path planning algorithm;

[0041] Selecting a motion control algorithm according to the kinematic model and dynamic model of the robot, and controlling the robot to move along the planned path according to the motion parameters;

[0042] Track the robot's motion trajectory in real time, compare the actual trajectory with the planned trajectory, obtain error information, and adjust the robot's control instructions based on the error information to obtain the final robot execution information.

[0043] In some embodiments, the step of acquiring target perception information and environmental information of the robot during movement, and generating a motion planning adjustment plan of the robot through the large language model according to the target perception information and the environmental information, comprises the following steps:

[0044] Use deep learning-based target detection algorithms to identify target objects in images;

[0045] The reasoning ability and knowledge base of the large language model are used to assist the robot in determining the best observation angle and distance. Specifically:

[0046] The SLAM algorithm is used to obtain the position information of the robot and the target object in the grid map; then, the position information of the robot and the target object is converted into natural language description, and the large language model is asked whether the current observation angle and distance are reasonable, and corresponding suggestions are given; the robot adjusts its own posture according to the suggestions of the large language model, plans the robot's motion trajectory using the path planning algorithm, and uses the motion control algorithm to control the robot to execute the motion planning plan, adjust its own posture, and obtain the best observation angle and distance;

[0047] Through the steps of target recognition, optimal observation angle and distance judgment, and posture adjustment, the robot can identify and understand the target objects in the environment, and adjust its own posture as needed to obtain the optimal observation angle and distance.

[0048] Another aspect of the embodiment of the present invention further provides a robot path planning perception system based on a large language model, comprising:

[0049] The first module is used to obtain an initial data set, wherein the initial data set includes robot sensor information and robot posture information;

[0050] The second module is used to select a SLAM algorithm according to an actual application scenario to process the initial data set to obtain a first map constructed by SLAM;

[0051] A third module is used to perform rasterization processing on the first map to obtain a second map;

[0052] A fourth module is used to perform semantic annotation processing on the second map to obtain a third map;

[0053] The fifth module is used to perform path planning on the third map after the semantic annotation processing according to the large language model, and generate a path planning solution;

[0054] A sixth module is used to convert the execution information of the robot according to the path planning scheme, and the execution information is used to control the robot to move along the target path using corresponding motion parameters;

[0055] The seventh module is used to obtain the target perception information and environmental information of the robot during movement, and generate a motion planning adjustment plan of the robot through the large language model according to the target perception information and the environmental information;

[0056] The eighth module is used to build a human-computer interaction module to realize information interaction between users and robots.

[0057] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0058] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0059] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0060] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a robot path planning perception method and system based on a large language model, the scheme acquires an initial data set, the initial data set includes robot sensor information and robot posture information; selects a SLAM algorithm according to the actual application scenario to process the initial data set to obtain a first map constructed by SLAM; performs rasterization processing on the first map to obtain a second map; performs semantic annotation processing on the second map to obtain a third map; performs path planning on the third map after the semantic annotation processing according to the large language model to generate a path planning scheme; obtains the execution information of the robot according to the conversion of the path planning scheme, and the execution information is used to control the robot to move along the target path using corresponding motion parameters; acquires the target perception information and environmental information of the robot during the movement, and generates the motion planning adjustment scheme of the robot through the large language model according to the target perception information and the environmental information; and constructs a human-computer interaction module to realize information interaction between the user and the robot. The embodiments of the present invention can generate a more reliable and smoother path, and improve the navigation and perception capabilities of the robot in a complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0062] Figure 2 is a flow chart of overall steps provided by an embodiment of the present invention;

[0063] Figure 3 It is a schematic diagram of the overall architecture provided by an embodiment of the present invention;

[0064] Figure 4 is a process diagram of grid mapping provided by an embodiment of the present invention;

[0065] Figure 5 is a process diagram of a semantically labeled grid map provided by an embodiment of the present invention;

[0066] Figure 6 is a flowchart of refine provided by an embodiment of the present invention;

[0067] Figure 7 is a schematic diagram of a visualized execution result and a motion result provided by an embodiment of the present invention;

[0068] Figure 8 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0070] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0071] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0072] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0073] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0074] 1. Large Language Model: It can be understood as a super "student" that has read a huge amount of text data and learned to understand and generate human language. You can imagine it as a huge library that stores a variety of knowledge. You can ask it questions and it will answer you based on the knowledge it has learned.

[0075] 2. Chain of Thought: Imagine that when you solve a math problem, instead of writing down the answer directly, you write down the steps to solve the problem step by step. This is the chain of thought. For large language models, the chain of thought means that the model can reason step by step like humans, instead of directly giving the final result. This can help the model better understand the problem and improve the accuracy of problem solving.

[0076] 3. In-context learning: It can be understood as a learning method for large language models. It does not require special training of the model, but provides some examples or prompts to the model, allowing the model to understand the current task and give an answer based on these examples or prompts. (In-context learning is more like a "learning by analogy" ability. The model understands the task pattern by observing examples and applies it to new problems. The thinking chain emphasizes the reasoning process. The model needs to think step by step and finally get the answer.)

[0077] 4. Robot Perception Task: The goal of the robot perception task is to enable the robot to obtain environmental information through sensors and understand this information, such as identifying objects, estimating distances, and sensing environmental changes. This is the basis for robots to complete various tasks. It is like the "eyes" and "ears" of the robot, helping the robot understand the surrounding environment.

[0078] 5. Robot Navigation Task: The goal of the robot navigation task is to enable the robot to autonomously plan a path in the environment, avoid obstacles, and eventually reach the designated destination. This requires the robot to have the ability to perceive the environment, locate itself, plan paths, and control movement. Just like a delivery robot, it needs to be able to identify roads, avoid pedestrians and vehicles, and finally deliver the package to the customer.

[0079] 6. Embodied intelligence refers to the ability of an intelligent agent to learn and develop intelligence through physical interaction with the environment. In other words, intelligence does not exist solely in the brain or computer, but is generated and developed through the interaction of the body with the environment.

[0080] The robot path planning perception method and system based on a large language model provided in an embodiment of the present invention relate to the field of computer technology. The robot path planning perception method based on a large language model provided in an embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a robot path planning perception method based on a large language model, etc., but is not limited to the above forms.

[0081] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0082] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0083] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0084] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0085] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0086] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a robot path planning perception method based on a large language model. The following is explained using the application of the robot path planning perception method based on a large language model in the server 101 as an example. It can be understood that the method can also be applied to the terminal 102.

[0087] Reference Figure 2 , Figure 2 The flowchart of the robot path planning perception method based on a large language model applied to a server provided in an embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method may include the following steps:

[0088] Acquire an initial data set, wherein the initial data set includes robot sensor information and robot posture information;

[0089] Selecting a SLAM algorithm according to an actual application scenario to process the initial data set to obtain a first map constructed by SLAM;

[0090] rasterizing the first map to obtain a second map;

[0091] Performing semantic annotation processing on the second map to obtain a third map;

[0092] According to the large language model, path planning is performed on the third map after the semantic annotation processing is completed to generate a path planning solution;

[0093] Obtaining execution information of the robot according to the path planning scheme, wherein the execution information is used to control the robot to move along the target path using corresponding motion parameters;

[0094] Acquire target perception information and environmental information of the robot during movement, and generate a motion planning adjustment plan of the robot through the large language model according to the target perception information and the environmental information;

[0095] Build a human-computer interaction module to realize information interaction between users and robots.

[0096] In some embodiments, the process of selecting a SLAM algorithm according to an actual application scenario to process the initial data set to obtain a first map constructed by SLAM includes the following steps:

[0097] Selecting a Karto SLAM algorithm, and adjusting parameters of the Karto SLAM algorithm to obtain an optimal parameter combination; the parameter adjustment includes but is not limited to adjusting a node adding strategy and adjusting a threshold of loop detection;

[0098] According to the adjusted optimal parameter combination, the initial data set is processed by the Karto SLAM algorithm to obtain a first map constructed by SLAM;

[0099] Wherein, the adjusting node adding strategy includes selecting whether to add a node according to the distance between nodes;

[0100] The adjusting the loop detection threshold includes: determining whether a loop occurs according to point cloud similarity.

[0101] In some embodiments, the rasterizing the first map to obtain the second map comprises the following steps:

[0102] Dividing the continuous map space on the first map into grid units of equal size, each grid unit representing an area on the map, and converting the first map into a grid map of a specified resolution;

[0103] The grid map is binarized, and the value of each grid cell is converted into 0 or 1 to obtain a second map.

[0104] In some embodiments, performing semantic annotation processing on the second map to obtain a third map includes the following steps:

[0105] For the second map, each grid cell is assigned an important semantic label; specifically:

[0106] The Yo lov8 target detection model is used as the robot perception module algorithm. Through the calibration relationship between the camera and the lidar, the positions of important objects in reality are projected onto the grid map. Finally, each grid unit is given a corresponding semantic label and a passable attribute.

[0107] In some embodiments, performing path planning on the third map after semantic annotation processing according to the large language model to generate a path planning solution includes the following steps:

[0108] converting the binary map of the second map into a natural language description;

[0109] Determine the robot's mission objectives based on the user's preference information or control instructions;

[0110] According to the task objective, a few-shot refine method is introduced into the large language model to iteratively correct the path planning scheme to obtain the final path planning scheme. The specific step is as follows: corresponding examples are added to the prompt words. When the planning result output by the large language model is judged to be wrong, the path planning is continued until the planning result is judged to be correct.

[0111] In some embodiments, the adding of corresponding examples to the prompt word includes:

[0112] Design multiple representative path planning cases, which include factors of different map types, obstacle distributions, and target locations;

[0113] Obtain path planning examples through expert design and annotation; or automatically generate path planning examples through path planning algorithms;

[0114] The method of automatically generating a path planning case by using a path planning algorithm comprises the following steps:

[0115] Convert the automatically generated path planning cases into a format that the large language model can understand. Use the few sample cases as learning materials to guide the large language model to perform path planning. Input the map information, starting point, and end point information as prompts to the large language model. The large language model generates an initial path planning solution based on the learned knowledge.

[0116] The rationality of the initial path planning scheme is evaluated, specifically: a series of judgment rules are defined to determine whether there are problems with the scheme, wherein the judgment rules include whether a collision occurs, whether a boundary is exceeded, and whether the path length is too long; the map information and the initial path planning scheme are analyzed by using the reasoning ability of the large language model to determine whether the scheme meets the predefined rules; if the self-reflection process of the large language model finds that there are problems with the path planning scheme, the identified problems are converted into natural language descriptions; feedback information is added to the prompt of the large language model, and the large language model is required to regenerate the path planning scheme;

[0117] During the iterative process of the Few-shot refine method, user instructions, conversation records, and good path planning cases are obtained to build a knowledge base for iterative optimization.

[0118] In some embodiments, converting the execution information of the robot according to the path planning scheme comprises the following steps:

[0119] Perform coordinate system conversion to convert the coordinate points in the path planning solution from the grid map coordinate system to the robot's own coordinate system, where the coordinate system conversion is calculated using the robot's kinematic model;

[0120] Smoothing the converted path, wherein the smoothing process includes using Bezier curve processing or cubic spline interpolation processing;

[0121] Select a local path planning algorithm for local path planning based on the actual application scenario, and set motion parameters based on the robot's physical constraints and environmental characteristics;

[0122] Obtaining obstacle information in the surrounding environment in real time, and integrating the obstacle information into the selected local path planning algorithm;

[0123] Selecting a motion control algorithm according to the kinematic model and dynamic model of the robot, and controlling the robot to move along the planned path according to the motion parameters;

[0124] Track the robot's motion trajectory in real time, compare the actual trajectory with the planned trajectory, obtain error information, and adjust the robot's control instructions based on the error information to obtain the final robot execution information.

[0125] In some embodiments, the step of acquiring target perception information and environmental information of the robot during movement, and generating a motion planning adjustment plan of the robot through the large language model according to the target perception information and the environmental information, comprises the following steps:

[0126] Use deep learning-based target detection algorithms to identify target objects in images;

[0127] The reasoning ability and knowledge base of the large language model are used to assist the robot in determining the best observation angle and distance. Specifically:

[0128] The SLAM algorithm is used to obtain the position information of the robot and the target object in the grid map; then, the position information of the robot and the target object is converted into natural language description, and the large language model is asked whether the current observation angle and distance are reasonable, and corresponding suggestions are given; the robot adjusts its own posture according to the suggestions of the large language model, plans the robot's motion trajectory using the path planning algorithm, and uses the motion control algorithm to control the robot to execute the motion planning plan, adjust its own posture, and obtain the best observation angle and distance;

[0129] Through the steps of target recognition, optimal observation angle and distance judgment, and posture adjustment, the robot can identify and understand the target objects in the environment, and adjust its own posture as needed to obtain the optimal observation angle and distance.

[0130] The implementation process of the method of the present invention in a specific application scenario is described in detail below:

[0131] In response to the shortcomings of LLMs' planning capabilities, the present invention proposes a method called "few-shot refine". By inputting hand-crafted examples and using in-context learning, coupled with an iterative self-reflection and correction process, LLMs can learn from their own mistakes and improve planning results, generating more reliable and smoother paths. Ultimately, an end-to-end robot planning and perception framework based on LLMs is constructed to improve the robot's navigation and perception capabilities in complex environments.

[0132] like Figure 3 As shown in the figure, this scheme aims to build an embodied intelligent robot planning and perception framework based on large language models (LLMs), so that the robot can autonomously complete navigation and perception tasks in complex environments and have good human-computer interaction capabilities.

[0133] The basic idea is as follows:

[0134] 1. Leverage LLMs’ powerful reasoning and generalization capabilities for path planning: Traditional robot path planning methods usually rely on precise environmental models and complex algorithms, while LLMs can transform complex path planning problems into simple text generation problems through natural language understanding and reasoning capabilities. By converting environmental information (such as maps, obstacles, target locations, etc.) into natural language descriptions and combining them with a small number of example cases, LLMs can quickly generate reasonable path planning solutions.

[0135] 2. Use the Few-shot refine method (innovative method) to improve the performance of LLMs path planning: In order to further improve the accuracy and reliability of LLMs path planning, the Few-shot refine method is introduced. This method guides LLMs to learn the basic rules and strategies of path planning by designing a small number of representative path planning cases. At the same time, the self-reflection ability of LLMs is used to identify potential problems in the path planning scheme, and iterative corrections are made to finally generate a path planning scheme that meets actual needs.

[0136] 3. Combine traditional robot technology to achieve navigation and control: The path planning solution generated by LLMs needs to be converted into instructions that the robot can understand and execute. To this end, this solution combines traditional robot navigation and control technology, including coordinate system conversion, path smoothing, local path planning, motion control, etc., to convert the planning results of LLMs into specific actions of the robot.

[0137] 4. Introduce the target perception module to enhance the robot's environmental understanding ability: In order to enable the robot to better understand the surrounding environment, this solution introduces a target perception module. This module uses a deep learning target detection algorithm to identify the target object and adjusts the robot's posture according to the LLM's recommendations to obtain the best observation angle and distance, thereby obtaining more comprehensive and accurate environmental information.

[0138] 5. Build a human-computer interaction interface to achieve user-friendly control: In order to facilitate the interaction between users and robots, this solution builds a human-computer interaction interface. Users can issue instructions to the robot through voice commands or other means, and the robot will feedback information to the user through visualization and other means. At the same time, the system can also learn the user's preferences and habits based on the user's feedback and instructions, and adjust its own action strategy to better meet the user's needs.

[0139] In summary, this scheme combines the powerful reasoning and generalization capabilities of LLMs with traditional robotics technology to build an embodied intelligent robot framework with environmental perception, path planning, navigation control, and human-computer interaction capabilities. This framework has strong flexibility and scalability, can adapt to different application scenarios, and provides new ideas and methods for the development of future intelligent robots.

[0140] The following is a detailed description of the technical details of the embodied intelligent robot planning and perception framework based on large language models (LLMs), which mainly includes two parts: robot mapping and large language model path planning:

[0141] (I) Robot Mapping (Mapping Module) (Real2Sim), the process includes the following parts:

[0142] 1. Data Collection:

[0143] First, select the appropriate sensor for data collection. This solution uses the YDLIDAR model single-line laser radar as the main sensor to obtain high-precision, large-scale two-dimensional point cloud data. Then, in order to improve the robustness and accuracy of mapping, a depth camera (such as Intel Real Sense D435i) is fused to obtain RGB images and depth information, and an IMU (inertial measurement unit) is used to obtain the robot's posture information. Finally, the time synchronization mechanism of ROS (Robot Operating System) or other hardware / software synchronization methods are used to ensure that the data timestamps of the laser radar, depth camera, and IMU are synchronized to prepare for the subsequent mapping algorithm.

[0144] 2.SLAM Mapping:

[0145] First, choose a suitable open source SLAM toolbox according to the actual application scenario. For example, if the target environment is large and needs to take real-time into account, you can choose RTAB-Map; if you need to build a large-scale, high-precision map, you can choose Cartographer; if you have high real-time requirements, you can choose ORB-SLAM2. This method uses KartoSLAM. (KartoSLAM is an open source SLAM algorithm based on graph optimization. It uses scan matching technology to efficiently detect loops and builds accurate two-dimensional environment maps by optimizing graph structures. Its advantages are that it can effectively eliminate cumulative errors, support multiple sensor data, and is open source and extensible, which is convenient for users to carry out secondary development. Its workflow mainly includes four steps: scan matching, adding nodes and edges, loop detection, and graph optimization. By iteratively executing these steps, a globally consistent map and robot trajectory are finally obtained.)

[0146] Then, adjust the parameters for the selected SLAM algorithm. For example, you can adjust the keyframe selection strategy, such as selecting keyframes based on the distance between keyframes or the change in viewing angle; you can adjust the loop detection threshold, such as judging whether a loop occurs based on the point cloud similarity or the number of feature point matches. Parameter adjustment needs to be carried out through experiments and debugging to find the best parameter combination to optimize the mapping effect.

[0147] 3. Rasterized Map

[0148] refer to Figure 4First, the map constructed by SLAM is rasterized. Rasterization is to divide the continuous map space into grid cells of equal size, each grid cell represents an area on the map. The resolution of the grid map determines the accuracy and storage space of the map, and needs to be selected according to actual needs. For example, you can choose a resolution of 0.05 meters, which means that each grid cell represents an area of ​​0.05 meters x 0.05 meters. You can use the resize function in the OpenCV library to convert the map constructed by SLAM into a grid map of a specified resolution.

[0149] refer to Figure 5 Then, the raster map is binarized. Binarization converts the value of each raster cell into 0 or 1, where 0 represents a passable area and 1 represents an obstacle. Binarization can simplify the map representation and facilitate the subsequent path planning algorithm processing. The threshold function in the OpenCV library can be used to binarize the raster map. For example, the threshold is set to 0.5, the raster cell value greater than 0.5 is set to 1, and the raster cell value less than 0.5 is set to 0.

[0150] 4. Semantic Annotation

[0151] In addition, the raster map needs to be semantically annotated, that is, each raster cell is given an important semantic label, such as "starting point", "end point", "wounded people in need of rescue", "fire location", etc. The starting point can be marked as "S" and the end point as "E". This text-based map representation supports flexible and diverse semantic labels and can be understood by large models.

[0152] Semantic labeling can help robots better understand the environment. For example, robots can use semantic labels to determine which areas can be placed and which areas need to be avoided. The Yolo series of object detection models can be used as robot perception module algorithms. The model covers a variety of common object categories in daily environments and is a widely used detection model. Then, through the calibration relationship between the camera and the lidar, the location of important objects in reality is projected onto the grid map, and finally each grid unit is assigned a corresponding semantic label and a passable attribute.

[0153] (II) Large language model path planning:

[0154] First, convert the binary map into a natural language description. If it is a simple path planning task, describe the positions of the starting point 'S' and the end point 'E' in the grid map in the figure above. For example, if the robot's starting position is (1,1) and the target position is (9,9), fill in the corresponding positions with the corresponding letters and add letter descriptions.

[0155] After that, the user gives preferences or instructions to clarify the robot's mission objectives, such as "Plan the shortest path from the start point to the end point, avoiding all obstacles." Finally, other information can be added as needed, such as the robot's size, motion constraints, etc., to make the prompts more complete.

[0156] In order to further improve the performance of the large language model (LLM) in path planning tasks, this scheme introduces the few-shot refine method, which guides LLM to learn the basic rules and strategies of path planning through few-shot learning, and uses the self-reflection ability of LLM to make iterative corrections, and finally generates a path planning scheme that meets actual needs.

[0157] 1. Manually add corresponding examples to the prompt words. Examples are collected manually in the initialization phase and imported into the database. In actual operation, corresponding examples are imported from the UI or manually input by humans.

[0158] 2. If the planning result output by the big model is considered wrong by humans or itself (the big model), it will continue to plan until the judge (or the big model) believes that its planning plan is correct (no path conflict). Figure 6 shown.

[0159] Regarding how to obtain examples in the prompt words, first, design 5-10 representative path planning cases, covering factors such as different map types, obstacle distribution and target location. Case data can be obtained through manual design and annotation, such as inviting experts or users to provide high-quality path planning solutions; traditional path planning algorithms (such as A* algorithm) can also be used to automatically generate paths in cases, and manually screen and optimize. Case data needs to be converted into a format that LLM can understand, such as JSON format, containing information such as map information, starting point, end point and optimal path. Use a few sample cases as learning materials to guide LLM to perform path planning. Select powerful LLM models such as GPT-3 and Codex, and input map information, starting point, end point and other information as prompts to LLM. LLM generates an initial path planning solution based on the learned knowledge.

[0160] In the method of the embodiment of the present invention, in order to evaluate the rationality of the initial path planning scheme, a series of rules are defined to determine whether there are problems with the scheme, such as whether a collision occurs, whether it exceeds the boundary, whether the path length is too long, etc. The reasoning ability of LLM is used to analyze the map information and the path planning scheme to determine whether the scheme meets the pre-defined rules. If the self-reflection process of LLM finds that there is a problem with the path planning scheme, the identified problem is converted into a natural language description, such as "the path planning scheme collides with an obstacle at position (5,6)". The feedback information is added to the prompt of LLM, and LLM is required to regenerate the path planning scheme. The iterative process of the few-shot refine method requires setting a termination condition, such as setting a maximum number of iterations (for example, 10 times), or terminating the iteration when the path planning scheme generated by LLM meets all rules. Finally, the path planning scheme generated by LLM is converted into a format that the robot can understand, such as a series of coordinate points or control instructions for the robot to execute.

[0161] During the iteration of the few-shot refine method and the interaction with the user, the following information is collected to build the knowledge base: user instructions, such as "please avoid the table" and "go to the bedroom"; conversation records, such as the user's evaluation of the path planning scheme and the description of the target object's location; good paths, such as the path planning scheme generated by LLM and recognized by the user; good observation cases, such as the best observation angle and distance obtained by the robot after adjusting its posture according to the LLM suggestion, as well as the corresponding target object and robot position information, etc. The context of these knowledge constructions can be used for subsequent LLM training and optimization to improve the performance and generalization ability of LLM in path planning tasks.

[0162] 3.Robot path execution module

[0163] The robot path execution module is responsible for converting the path planning scheme generated by the large language model (LLM) into instructions that the robot can understand and execute, and controlling the robot to move safely and efficiently along the planned path. The path planning scheme generated by the LLM is usually represented by discrete coordinate points in the grid map coordinate system. In order for the robot to execute the scheme, it is first necessary to perform a coordinate system conversion to convert the coordinate points in the path planning scheme from the grid map coordinate system to the robot's own coordinate system. The coordinate system conversion can be calculated using the robot kinematic model, such as rotation and translation transformations. Secondly, the converted path needs to be smoothed, such as using Bezier curves or cubic spline interpolation methods, to make the robot's motion trajectory more natural and smooth, and avoid unstable movements such as sharp turns or sudden stops of the robot.

[0164] Due to factors such as dynamic changes in the environment and sensor errors, the robot may encounter some unknown obstacles or situations when actually executing the path planning solution, so local path planning is required. The local path planning algorithm needs to be selected according to the actual application scenario. This solution uses the dynamic window method (DWB) and sets corresponding parameters according to the physical constraints of the robot and the characteristics of the environment, such as maximum speed, acceleration, steering angle, safe distance, etc. The robot's sensors (such as lidar and depth camera) are used to perceive obstacle information in the surrounding environment in real time, and the obstacle information is integrated into the local path planning algorithm to ensure that the robot can safely avoid obstacles.

[0165] The motion control module is responsible for controlling the robot's speed, acceleration, and steering angle so that it can move along the planned path. The motion control algorithm needs to be selected based on the robot's kinematic model and dynamic model. PID control is used in this solution. The robot's onboard sensors (such as odometers, IMUs) are used to track the robot's motion trajectory in real time, and the actual trajectory is compared with the planned trajectory. The control instructions are adjusted based on the error information to ensure that the robot can accurately move along the planned path.

[0166] Through steps such as coordinate system conversion, path smoothing, local path planning and motion control, the robot navigation and control module can convert the path planning scheme generated by LLM into instructions that the robot can execute, and control the robot to move to the target position safely and efficiently to complete the navigation task.

[0167] 4. Target perception module:

[0168] The target perception module is responsible for enabling the robot to identify and understand the target objects in the environment, and adjust its posture according to the needs to obtain the best observation angle and distance, so as to obtain more comprehensive and accurate environmental information. Target recognition refers to the use of computer vision technology to identify target objects in images or videos, such as tables, chairs, people, etc. This solution uses a target detection algorithm based on deep learning for target recognition. Based on the characteristics of the target objects and application scenarios, the YOLO algorithm is used to detect multiple target objects in real time.

[0169] In order to obtain more detailed information about the target object, such as shape, color, texture, etc., the robot needs to adjust its posture to obtain the best observation angle and distance. This solution uses the reasoning ability and knowledge base of LLM to assist the robot in judging the best observation angle and distance. First, the SLAM algorithm is used to obtain the position information of the robot and the target object in the grid map (sim). Then, the position information of the robot and the target object is converted into a natural language description, and the LLM is asked whether the current observation angle and distance are reasonable. In order to help LLM learn the best observation strategy, some text-based best observation cases can be provided. Based on the learned knowledge and input information, LLM judges whether the current observation angle and distance are reasonable, and gives corresponding suggestions.

[0170] According to LLM's suggestion, the robot needs to adjust its posture to obtain the best observation angle and distance. The path planning algorithm is used to plan the robot's motion trajectory so that it can reach the best observation position, and the motion control algorithm is used to control the robot to execute the motion planning plan and adjust its posture, such as moving the position and adjusting the camera angle.

[0171] Through steps such as target recognition, optimal observation angle and distance judgment, and posture adjustment, the target perception module enables the robot to identify and understand target objects in the environment, and adjust its own posture as needed to obtain the optimal observation angle and distance, thereby obtaining more comprehensive and accurate environmental information to provide support for subsequent decision-making and actions.

[0172] 5. Human-computer interaction module (Intercat ion Module):

[0173] The human-computer interaction module is responsible for realizing information exchange between users and robots, so that users can easily control robots and obtain the status and result information of robot tasks. The human-computer interaction module of this solution mainly includes three parts: text instructions, feedback and explanation, and visual execution process. Text instructions refer to instructions issued by users to robots through text, such as "go to the kitchen", "get a glass of water", etc. The robot needs to be able to understand the user's voice instructions and convert them into corresponding action instructions. This step requires the use of a natural language understanding (NLU) model to understand the user's intentions and convert text information into action instructions that the robot can understand. Commonly used NLU models include BERT, ERNI E, ChatGPT, etc.

[0174] The robot needs to provide feedback to the user on the status and results of the task execution, such as "heading to the kitchen", "already in the kitchen", etc. The robot can also explain to the user the reasons and basis for its actions, such as "because you said you were going to the kitchen, so I'm going to the kitchen". The robot's feedback information is converted into text output and the robot's action plan, execution status, and perception results are visualized through images or videos to facilitate user understanding. Figure 7 As shown, this method uses OpenCV to visualize the robot's scene, distribution, and task execution results.

[0175] In summary, compared with the prior art, the present invention has the following advantages:

[0176] The present invention proposes an embodied intelligent robot planning and perception framework based on a large language model, innovatively applies the large language model to the field of robotics, and introduces human interaction and knowledge base construction mechanisms, effectively solving the shortcomings of the prior art in terms of end-to-end framework, path planning quality, virtual-reality migration, human-machine collaborative learning, and target perception efficiency.

[0177] The present invention builds a complete end-to-end framework and uses the few-shot refine method to enhance the path planning capability of the large language model, generate a more reliable, smoother, and more human-desirable path, and combines the local path planning algorithm to solve the virtual-real migration problem. The reasoning ability of the large language model is used to determine the optimal observation angle and distance, and guide the robot to adjust its posture, thereby improving the efficiency and accuracy of target perception.

[0178] In addition, the present invention also realizes human-computer collaborative learning and knowledge creation through artificial interaction and knowledge base construction mechanism. Users can guide the robot to correct path planning plans through voice or images. The robot can learn user preferences and habits and use this knowledge for future decision-making. Ultimately, it significantly improves the robot's navigation, perception ability, learning adaptability and human-computer interaction experience in complex environments, and promotes the intelligent and automated development of robotics technology.

[0179] Another aspect of the embodiment of the present invention further provides a robot path planning perception system based on a large language model, comprising:

[0180] The first module is used to obtain an initial data set, wherein the initial data set includes robot sensor information and robot posture information;

[0181] The second module is used to select a SLAM algorithm according to an actual application scenario to process the initial data set to obtain a first map constructed by SLAM;

[0182] A third module is used to perform rasterization processing on the first map to obtain a second map;

[0183] A fourth module is used to perform semantic annotation processing on the second map to obtain a third map;

[0184] The fifth module is used to perform path planning on the third map after the semantic annotation processing according to the large language model, and generate a path planning solution;

[0185] A sixth module is used to convert the execution information of the robot according to the path planning scheme, and the execution information is used to control the robot to move along the target path using corresponding motion parameters;

[0186] The seventh module is used to obtain the target perception information and environmental information of the robot during movement, and generate a motion planning adjustment plan of the robot through the large language model according to the target perception information and the environmental information;

[0187] The eighth module is used to build a human-computer interaction module to realize information interaction between users and robots.

[0188] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0189] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned robot path planning perception method based on a large language model when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0190] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0191] See also Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0192] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0193] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the robot path planning perception method based on a large language model in the embodiment of the present invention;

[0194] Input / output interface 803, used to implement information input and output;

[0195] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0196] A bus 805 that transmits information between the various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0197] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0198] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned robot path planning perception method based on a large language model.

[0199] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0200] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0201] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0202] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0203] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0204] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0205] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0206] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0207] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0208] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0209] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0211] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0212] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A robot path planning perception method based on a large language model, characterized in that: The following steps are involved: Acquire an initial data set, wherein the initial data set includes robot sensor information and robot posture information; Selecting a SLAM algorithm according to an actual application scenario to process the initial data set to obtain a first map constructed by SLAM; rasterizing the first map to obtain a second map; Performing semantic annotation processing on the second map to obtain a third map; According to the large language model, path planning is performed on the third map after the semantic annotation processing is completed to generate a path planning solution; Obtaining execution information of the robot according to the path planning scheme, wherein the execution information is used to control the robot to move along the target path using corresponding motion parameters; Acquire target perception information and environmental information of the robot during movement, and generate a motion planning adjustment plan of the robot through the large language model according to the target perception information and the environmental information; Build a human-computer interaction module to realize information interaction between users and robots.

2. A robot path planning perception method based on a large language model according to claim 1, characterized in that: The step of selecting a SLAM algorithm according to an actual application scenario to process the initial data set to obtain a first map constructed by SLAM includes the following steps: Selecting a Karto SLAM algorithm, and adjusting parameters of the Karto SLAM algorithm to obtain an optimal parameter combination; the parameter adjustment includes but is not limited to adjusting a node adding strategy and adjusting a threshold of loop detection; According to the adjusted optimal parameter combination, the initial data set is processed by the Karto SLAM algorithm to obtain a first map constructed by SLAM; Wherein, the adjusting node adding strategy includes selecting whether to add a node according to the distance between nodes; The adjusting the loop detection threshold includes: determining whether a loop occurs according to point cloud similarity.

3. A robot path planning perception method based on a large language model according to claim 1, characterized in that: The step of rasterizing the first map to obtain the second map comprises the following steps: Dividing the continuous map space on the first map into grid units of equal size, each grid unit representing an area on the map, and converting the first map into a grid map of a specified resolution; The grid map is binarized, and the value of each grid cell is converted into 0 or 1 to obtain a second map.

4. A robot path planning perception method based on a large language model according to claim 1, characterized in that: The step of performing semantic annotation processing on the second map to obtain a third map comprises the following steps: For the second map, each grid cell is assigned an important semantic label; specifically: The Yolov8 target detection model is used as the robot perception module algorithm. Through the calibration relationship between the camera and the lidar, the positions of important objects in reality are projected onto the grid map. Finally, each grid unit is given a corresponding semantic label and a passable attribute.

5. The robot path planning perception method based on a large language model according to claim 1, characterized in that: The method of performing path planning on the third map after the semantic annotation processing according to the large language model to generate a path planning solution includes the following steps: converting the binary map of the second map into a natural language description; Determine the robot's mission objectives based on the user's preference information or control instructions; According to the task objective, a few-shot refine method is introduced into the large language model to iteratively correct the path planning scheme to obtain the final path planning scheme. The specific step is as follows: corresponding examples are added to the prompt words. When the planning result output by the large language model is judged to be wrong, the path planning is continued until the planning result is judged to be correct.

6. A robot path planning perception method based on a large language model according to claim 5, characterized in that: The corresponding examples added to the prompt words include: Design multiple representative path planning cases, which include factors of different map types, obstacle distributions, and target locations; Obtain path planning examples through expert design and annotation; or automatically generate path planning examples through path planning algorithms; The method of automatically generating a path planning case by using a path planning algorithm comprises the following steps: Convert the automatically generated path planning cases into a format that the large language model can understand. Use the few sample cases as learning materials to guide the large language model to perform path planning. Input the map information, starting point, and end point information as prompts to the large language model. The large language model generates an initial path planning solution based on the learned knowledge. The rationality of the initial path planning scheme is evaluated, specifically: a series of judgment rules are defined to determine whether there are problems with the scheme, wherein the judgment rules include whether a collision occurs, whether a boundary is exceeded, and whether the path length is too long; the map information and the initial path planning scheme are analyzed by using the reasoning ability of the large language model to determine whether the scheme meets the predefined rules; if the self-reflection process of the large language model finds that there are problems with the path planning scheme, the identified problems are converted into natural language descriptions; feedback information is added to the prompt of the large language model, and the large language model is required to regenerate the path planning scheme; During the iterative process of the Few-shot refine method, user instructions, conversation records, and good path planning cases are obtained to build a knowledge base for iterative optimization.

7. The robot path planning perception method based on a large language model according to claim 1, characterized in that: The step of converting the execution information of the robot according to the path planning scheme comprises the following steps: Perform coordinate system conversion to convert the coordinate points in the path planning solution from the grid map coordinate system to the robot's own coordinate system, where the coordinate system conversion is calculated using the robot's kinematic model; Smoothing the converted path, wherein the smoothing process includes using Bezier curve processing or cubic spline interpolation processing; Select a local path planning algorithm for local path planning based on the actual application scenario, and set motion parameters based on the robot's physical constraints and environmental characteristics; Obtaining obstacle information in the surrounding environment in real time, and integrating the obstacle information into the selected local path planning algorithm; Selecting a motion control algorithm according to the kinematic model and dynamic model of the robot, and controlling the robot to move along the planned path according to the motion parameters; Track the robot's motion trajectory in real time, compare the actual trajectory with the planned trajectory, obtain error information, and adjust the robot's control instructions based on the error information to obtain the final robot execution information.

8. The robot path planning perception method based on a large language model according to claim 1, characterized in that: The step of acquiring target perception information and environmental information of the robot during movement, and generating a motion planning adjustment plan of the robot through the large language model according to the target perception information and the environmental information, comprises the following steps: Use deep learning-based target detection algorithms to identify target objects in images; The reasoning ability and knowledge base of the large language model are used to assist the robot in determining the best observation angle and distance. Specifically: The SLAM algorithm is used to obtain the position information of the robot and the target object in the grid map; then, the position information of the robot and the target object is converted into natural language description, and the large language model is asked whether the current observation angle and distance are reasonable, and corresponding suggestions are given; the robot adjusts its own posture according to the suggestions of the large language model, plans the robot's motion trajectory using the path planning algorithm, and uses the motion control algorithm to control the robot to execute the motion planning plan, adjust its own posture, and obtain the best observation angle and distance; Through the steps of target recognition, optimal observation angle and distance judgment, and posture adjustment, the robot can identify and understand the target objects in the environment, and adjust its own posture as needed to obtain the optimal observation angle and distance.

9. A robot path planning perception system based on a large language model, characterized in that: include: The first module is used to obtain an initial data set, wherein the initial data set includes robot sensor information and robot posture information; The second module is used to select a SLAM algorithm according to an actual application scenario to process the initial data set to obtain a first map constructed by SLAM; A third module is used to perform rasterization processing on the first map to obtain a second map; A fourth module is used to perform semantic annotation processing on the second map to obtain a third map; The fifth module is used to perform path planning on the third map after the semantic annotation processing according to the large language model, and generate a path planning solution; A sixth module is used to convert the execution information of the robot according to the path planning scheme, and the execution information is used to control the robot to move along the target path using corresponding motion parameters; The seventh module is used to obtain the target perception information and environmental information of the robot during movement, and generate a motion planning adjustment plan of the robot through the large language model according to the target perception information and the environmental information; The eighth module is used to build a human-computer interaction module to realize information interaction between users and robots.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.

Images