A method and system for task planning of a robot

Abstract

The application relates to the technical field of robots, and discloses a task planning method and system for a robot, the method comprising: acquiring a user instruction; based on a large language model and a semantic coding model, retrieving environment perception information related to the user instruction in a memory bank; inputting a first prompt word determined based on the user instruction and the environment perception information into the large language model to generate an overall action sequence corresponding to the user instruction; for each sub-action in the overall action sequence, determining a second prompt word, inputting the second prompt word into the large language model to generate a detailed action plan corresponding to the current sub-action, and iterating through each sub-action in the overall action sequence. The application can receive natural language or unstructured instructions, understand user intentions, and generate an executable overall action sequence and a detailed action plan for each sub-action in the sequence, thereby completing a task issued by a user and ensuring that the robot can execute the task and improve the accuracy and efficiency of executing the task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a task planning method and system of a robot. BACKGROUND

[0002] With the rapid development of artificial intelligence and robot technology, intelligent robots are gradually applied to many fields such as home service, industrial automation, medical care, etc. However, the traditional robot control method usually relies on pre-programmed fixed processes, for example, in a manufacturing factory, each mechanical arm is a fixed process optimized in advance, and each mechanical arm processes different tasks. When encountering unexpected situations or custom tasks, it lacks flexibility and adaptability, and is difficult to cope with dynamic environments or diversified user needs.

[0003] In the home service scenario, users usually want the robot to complete tasks such as "tidying up the living room" or "delivering a cup of water", which are non-fixed processes, and these tasks involve the coordination of multiple sub-actions (such as navigation, grasping, obstacle avoidance, etc.), and the robot needs to have the ability of task decomposition, logical reasoning and real-time adjustment. However, the existing robot control method can only select the corresponding process from the fixed process to execute, and cannot flexibly change the process according to the issued task, resulting in the robot being unable to correct the process when encountering unexpected situations, and being unable to complete the user's demand for action, ultimately reducing the accuracy, precision and efficiency of the robot in executing the task. SUMMARY

[0004] In view of this, the present application provides a task planning method and system of a robot.

[0005] The present application discloses a task planning method of a robot, applicable to a server, comprising:

[0006] Step 1: obtaining a user instruction; the user instruction is used to instruct the robot body to execute a non-fixed process task;

[0007] Step 2: retrieving environment perception information related to the user instruction in a memory bank based on a large language model and a semantic encoding model; the large language model is used to analyze the user instruction to obtain a candidate object related to the user instruction, and the semantic encoding model is used to encode the candidate object into semantic features in the same format as the memory bank;

[0008] Step 3: inputting a first prompt word determined based on the user instruction and the environment perception information into a large language model to generate an overall action sequence corresponding to the user instruction; the first prompt word at least contains a plurality of sub-actions supported by the robot;

[0009] Step 4: determining a second prompt word for each sub-action in the overall action sequence, inputting the second prompt word into a large language model to generate a detailed action plan corresponding to the current sub-action until all sub-actions in the overall action sequence are traversed; the detailed action plan is composed of a plurality of basic actions that support the sub-action.

[0010] Further, the step 2 comprises:

[0011] obtaining a third prompt word, inputting the third prompt word into the large language model to obtain an object name related to the user instruction; the third prompt word is used to control the large language model to analyze the user instruction;

[0012] determining a semantic feature of the object name based on the semantic encoding model;

[0013] retrieving at least one item of environment perception information similar to the semantic feature in a memory bank based on the semantic feature; the semantic encoding model is a multi-modal semantic encoding model, which is used to encode text and images expressing the same meaning into the same semantic feature;

[0014] performing content matching on the at least one item of environment perception information and the user instruction to obtain first de-duplicated environment perception information; the content matching is used to determine, in the at least one item of environment perception information, environment perception information matched with the object name related to the user instruction;

[0015] performing target matching on the first de-duplicated environment perception information based on a fourth prompt word to obtain second de-duplicated environment perception information, and determining the second de-duplicated environment perception information as the environment perception information related to the user instruction;

[0016] The fourth prompt word is used to control the large language model to determine the matched environment perception information in a plurality of environment perception information representing the same object name.

[0017] Further, the step 3 comprises:

[0018] obtaining a list of sub-actions that the robot can support;

[0019] filling the sub-action list into a pre-configured first prompt word template to obtain a first prompt word; the replacement symbols in the first prompt word template include sub-action list, environment perception information, user instruction and data format of the overall action sequence respectively corresponding replacement symbols; each of the replacement symbols is used to generate the first prompt word matching the user instruction;

[0020] inputting the first prompt word into the large language model to obtain the overall action sequence corresponding to the user instruction.

[0021] Further, the step 4 comprises:

[0022] In chronological order, each sub-action in the overall action sequence is traversed, and the current sub-action, the robot's effector, and the plurality of basic actions that the effector can perform are obtained;

[0023] The current sub-action, the effector, and the plurality of basic actions that the effector can perform are filled into a preconfigured second prompt word template to obtain a second prompt word;

[0024] The second prompt word is input into a large language model to obtain a detailed action plan corresponding to the current sub-action.

[0025] Further, the step 3 further comprises:

[0026] The overall action sequence corresponding to the user instruction is verified by a first verifier. If the verification fails, the overall action sequence and the corresponding failure result are input into a large language model to regenerate the overall action sequence corresponding to the user instruction;

[0027] The first verifier is determined based on a preconfigured verification rule and based on the feasibility of the overall action sequence analyzed by the large language model;

[0028] The step 4 further comprises:

[0029] Each detailed action plan generated by the large language model is verified by a second verifier. If the verification fails, the detailed action plan and the corresponding failure result are input into the large language model to regenerate each detailed action plan corresponding to the overall action sequence until the verification succeeds.

[0030] Further, before the step 1, the method further comprises:

[0031] Receiving object information in each environment captured by the information acquisition device of the robot; the object information includes object names, image information corresponding to the object names, and position information; the objects include environmental objects, large object objects, and small object objects;

[0032] Based on a preconfigured data structure, the image information and the position information in the object information are processed to obtain environmental perception information corresponding to each object name; the environmental perception information at least includes RGB images, depth maps, object categories, detection boxes, and segmentation masks corresponding to each object name;

[0033] The environmental perception information is uploaded to the memory bank.

[0034] Further, the uploading of the environmental perception information to the memory bank comprises:

[0035] The plurality of environment perception information in the cache queue is de-duplicated in chronological order to obtain de-duplicated environment perception information; the de-duplication processing at least includes environment object de-duplication processing and object object de-duplication processing;

[0036] According to the detection frame in the environment perception information, the image information corresponding to each object is intercepted, and the image information is input into a semantic coding model to obtain the semantic features corresponding to the object;

[0037] According to a preset format, the semantic features and the environment perception information associated with the semantic features are uploaded to a memory bank; the memory bank is used to recall environment perception information with high similarity from similar semantic features.

[0038] Further, the step 4 further includes:

[0039] For each sub-action corresponding to the detailed action plan, the current environment perception information before the current sub-action is executed is obtained;

[0040] The execution command corresponding to the detailed action plan of the current sub-action is issued to each actuator of the robot; the actuator is used to instruct the hardware device of the robot to perform related control operations; the control operation at least includes navigation, grabbing, releasing, opening and closing;

[0041] After the actuator receives the corresponding execution command and executes it, the latest environment perception information after the current sub-action is executed is obtained;

[0042] The current environment perception information before the current sub-action is executed and the latest environment perception information after the execution are input into a large language model to obtain a sub-action verification result, until the detailed action plans of all sub-actions of the whole action sequence are verified.

[0043] Further, the current environment perception information before the current sub-action is executed and the latest environment perception information after the execution are input into a large language model to obtain a sub-action verification result, including:

[0044] Based on the current environment perception information before the current sub-action is executed and the latest environment perception information after the execution, a sub-action verification prompt word is generated;

[0045] The sub-action verification prompt word is input into the large language model to determine whether the first coordinate value of the target object in the current environment perception information and the second coordinate value of the target object in the latest environment perception information match the expected result of the current sub-action, if the first coordinate value and the second coordinate value match the expected result of the current sub-action, the sub-action verification result is determined to be verified.

[0046] The application also discloses a task planning system of a robot, which implements the task planning method of the robot.

[0047] A user instruction obtaining module is configured to obtain a user instruction, which is used to instruct the robot body to execute a task of a non-fixed process.

[0048] An environment perception information retrieving module is configured to retrieve environment perception information related to the user instruction from a memory bank based on a large language model and a semantic encoding model, wherein the large language model is used to analyze the user instruction to obtain candidate objects related to the user instruction, and the semantic encoding model is used to encode the candidate objects into semantic features in the same format as the memory bank.

[0049] A whole action sequence generating module is configured to input a first prompt word determined based on the user instruction and the environment perception information into the large language model to generate a whole action sequence corresponding to the user instruction, wherein the first prompt word at least contains a plurality of sub-actions supported by the robot.

[0050] A detailed action plan generating module is configured to determine a second prompt word for each sub-action in the whole action sequence, input the second prompt word into the large language model to generate a detailed action plan corresponding to the current sub-action, and repeat the above process until all sub-actions in the whole action sequence are traversed, wherein the detailed action plan is composed of a plurality of basic actions supporting the sub-action.

[0051] Thanks to the above technical solutions, the application has the following advantages: the application obtains a user instruction used to instruct the robot body to execute a task of a non-fixed process, retrieves environment perception information related to the user instruction from a memory bank based on a large language model and a semantic encoding model, determines candidate objects corresponding to the user instruction through the large language model, determines semantic features of the candidate objects through the semantic encoding model, and retrieves environment perception information similar to the semantic features from the memory bank. Then, a first prompt word containing a plurality of sub-actions supported by the robot is determined according to the user instruction and the environment perception information, and a whole action sequence corresponding to the user instruction is generated through the large language model and the first prompt word. Finally, a second prompt word is determined, and a detailed action plan corresponding to the current sub-action is generated through the large language model and the second prompt word, so that detailed action plans corresponding to each sub-action in the whole action sequence are obtained.

[0052] By the technical solution, the user intention can be understood from the received natural language or unstructured instruction, an executable overall action sequence and a detailed action plan of each sub-action in the sequence can be generated, and the task issued by the user can be completed according to the overall action sequence and the overall action plan, so that the robot can finally execute the task and the precision and efficiency of executing the task are improved. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to these drawings.

[0054] Figure 1 A flowchart of a task planning method of a robot according to an embodiment of the present application;

[0055] Figure 2 A flowchart of generating an action plan and executing according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] The present application is further illustrated in conjunction with the drawings and embodiments. The described embodiments are only some of the embodiments of the present application, not all the embodiments. All other embodiments obtained by those skilled in the art should belong to the scope of protection of the embodiments of the present application.

[0057] Reference Figure 1 The present application provides an embodiment of a task planning method of a robot, which is suitable for a server and includes:

[0058] S1: obtaining a user instruction.

[0059] The user instruction is used to instruct the robot body to execute a task of a non-fixed process. The user instruction can be that the user sends a voice to the robot body, the robot body converts the voice into a natural language text, the robot body encapsulates the natural language text into the user instruction, and sends the user instruction to the server. The server can be a micro server directly physically connected in the robot body, or a server in the cloud. The robot body communicates with the cloud server through communication connection, and the present application does not make specific limitation.

[0060] In the implementation of the present application, when the user issues an operation instruction to the robot body, the robot body receives the operation instruction and converts it into a user instruction that can be processed by a computer, and sends it to the server, so that the server can obtain the user instruction. The user can issue an operation instruction to the robot in various ways, such as inputting text, voice, interface interaction, etc. to the robot body. The operation instruction is used to instruct the robot to complete a specific task, and its core elements include but are not limited to execution action, operation object, position information, etc.

[0061] In an embodiment of the present application, the content appearing in the instruction issuing pipeline is monitored, for example, the user instruction is received through OCR text recognition, Chinese text input, and voice conversion text, etc. and finally converted into a text format instruction; after the user instruction is monitored, the user instructions are put into the instruction processing pipeline according to the order in which the user instructions are received.

[0062] S2: Based on the large language model and the semantic encoding model, the environment perception information related to the user instruction is retrieved in the memory bank.

[0063] The large language model is used to analyze the user instruction to obtain the candidate object related to the user instruction, and the semantic encoding model is used to encode the candidate object into semantic features in the same format as the memory bank. The memory bank is used to store the environment perception information corresponding to each object, and the environment perception information in the memory bank is constantly updated as the robot body moves and time passes. The environment perception information is the structured data composed of the environment information obtained by the robot body continuously scanning the environment during patrol, and the information of each object in the environment.

[0064] In one example, the environment perception information can include image information and position information. The image information includes at least RGB images, depth images, object categories, object detection boxes, and object corresponding segmentation masks, etc. The image information can be obtained by shooting with binocular cameras, laser radars, etc. and processed into corresponding RGB images and depth images. Through image recognition models and image segmentation models, the types of objects in the image can be identified, and for each object category, the detection box of the object and the segmentation mask of the object, etc. can be identified, and the above data can be packaged and stored as environment perception information. The position information can first obtain the positioning information of the robot itself, and take this information as the reference, combine the depth image containing the object, and determine the positional relationship between each object in the depth image and the positioning information of the robot itself through algorithm, to determine the actual position of each object in the image.

[0065] Specifically, a large language model is an artificial intelligence model built based on deep learning technology. When a user inputs a natural language instruction, the core task of the large language model is not simply keyword matching, but deep understanding of the deep intention and contextual meaning of the instruction. The large language model understands the semantics of the user's instruction. After being parsed by the large language model, the system obtains a "candidate object" description based on text. However, the environment perception information (such as visual features of objects, position point clouds, etc.) stored in the memory bank is non-textual multi-modal data. In order to perform effective matching retrieval, it is necessary to map the text description and the memory bank information into the same semantic feature space.

[0066] The embodiment of the present application adopts a pre-trained semantic encoding model (such as CLIP, BERT, etc. variants or multi-modal model). The model encodes the candidate object description in the form of text into a high-dimensional, fixed-length semantic feature vector. The semantic feature vector of the candidate object is calculated for similarity with all semantic feature vectors in the memory bank, for example, by calculating the similarity through cosine similarity or Euclidean distance. Each entry in the memory bank is associated with past perceived environment perception information (for example, RGB images of objects, depth information, 3D coordinates, etc.). According to the similarity score, the highest score or several environment perception information is retrieved, and these retrieved environment perception information is the most relevant environment perception information to the user's instruction. For example, the server will find a complete environment perception information of a book "on the living room table, blue cover, recorded yesterday" in the memory through the system.

[0067] S3: inputting the first prompt word determined based on the user instruction and the environment perception information into the large language model to generate an overall action sequence corresponding to the user instruction.

[0068] The first prompt word at least further contains a plurality of sub-actions supported by the robot. The first prompt word is used to let the large language model design a set of overall action sequences for the robot body according to the user instruction, the environment perception information corresponding to the user instruction, and the plurality of sub-actions supported by the robot, so that the robot body can complete the user instruction according to the overall action sequence.

[0069] In the embodiments of the present application, a prompt word template is preset, user instructions and environment perception information are filled into the prompt word template according to a certain format, and a plurality of sub-actions that can be supported by the robot body are filled into the prompt word template, so as to construct a first prompt word; the first prompt word is input into a large language model, the large language model decomposes a high-level task corresponding to the user instructions into a series of sequential or parallel sub-tasks, and designs a corresponding execution action for each sub-task to generate an overall action sequence. Therefore, the output of the large language model is a structured overall action sequence, which can be a JSON array, a step list or any format recognized by a robot control system.

[0070] S4: For each sub-action in the overall action sequence, a second prompt word is determined, the second prompt word is input into the large language model, and a detailed action plan corresponding to the current sub-action is generated until each sub-action in the overall action sequence is traversed.

[0071] The detailed action plan is composed of a plurality of basic actions that support the sub-action. The basic action is the most basic and simplest action that can be executed by each actuator in the robot body.

[0072] The overall action sequence (such as [navigation, moving the robot arm, grabbing]) generated by the foregoing steps is a high-level action plan. In order to ensure that the robot can safely and accurately execute each sub-action, the large language model is used for more fine-grained motion planning for each sub-action in the overall action sequence, so as to convert the high-level instruction into a bottom-level control instruction that can directly drive the actuator.

[0073] Taking a specific embodiment as an example:

[0074] Overall action sequence: ["navigate to point A", "pick up object B"].

[0075] Process the first sub-action "navigate to point A":

[0076] Second prompt word: "... current state: the robot is located at the origin. Next action: navigate to point A (coordinates x: 10, y: 5). Please generate navigation steps."

[0077] Detailed plan: ["plan global path", "move along path"].

[0078] Process the second sub-action "pick up object B":

[0079] Second prompt word: "... current state: the robot has arrived at point A, and the camera detects that object B is 30 cm to the left. Next action: pick up object B."

[0080] Detailed plan: ["Turn left 30 degrees", "Move the robot arm to the object coordinates", "Grab"].

[0081] The present application can receive natural language or unstructured instructions, understand user intent, and generate executable overall action sequences. At the same time, the method also has the ability of action monitoring and dynamic adjustment, ensuring that the robot can respond to environmental changes or unexpected situations during execution. For example, when executing the task of "get a cup of water", the method plans the movement path, identifies the water cup position, adjusts the robot arm grasping posture, and re-adjusts the action sequence when detecting that the water cup has been moved, thereby ensuring the successful completion of the task.

[0082] For the steps of S2, the specific implementation process of the method includes:

[0083] S201: Obtain a third prompt word, input the third prompt word into a large language model, and obtain object names related to the user instruction; the third prompt word is used to control the large language model to analyze the user instruction;

[0084] Among them, the third prompt word is a preset text template, which is usually constructed in a role-playing and instruction-specific manner. Through the third prompt word, the large language model can be used as a natural language processing expert, and then it can perform word segmentation and entity extraction on the subsequent input text to obtain multiple entity objects, and analyze the multiple entity objects to determine the object names of the candidate objects related to the user instruction. The object name can be the object name in the scene where the robot executes the task, or it can be the scene name directly.

[0085] Specifically, the user instruction is: "Hand over the red cup in the kitchen."

[0086] The third prompt word constructed is: "You are a natural language processing expert, please perform word segmentation and entity extraction on the user instruction, and determine the object names of all objects related to the user instruction."

[0087] The large language model outputs: "Scene name: kitchen; Object name: red cup; Object name: user."

[0088] S202: Determine the semantic features of the object names based on a semantic encoding model;

[0089] Among them, the semantic encoding model encodes the text semantics of the object name into a high-dimensional vector (i.e., semantic features), and the high-dimensional vector and the visual feature vector encoded by the same model in the memory bank are in the same semantic space, making it possible to calculate the semantic similarity between text and image.

[0090] Specifically, after parsing the object name in the text form (such as "red cup") from the user instruction, in order to be able to perform semantic-based similarity matching in the memory library, it is necessary to convert the name from discrete symbols (characters) to continuous, semantic-containing numerical representation. This step is to complete the conversion of text to vector through the semantic encoding model, which lays the foundation for subsequent cross-modal retrieval.

[0091] S203: retrieving at least one environment perception information similar to the semantic feature in the memory library based on the semantic feature; the semantic encoding model is a multi-modal semantic encoding model, used for encoding text and image expressing the same meaning into the same semantic feature;

[0092] Among them, the environment perception information at least includes the RGB image, the depth map, the object category, the detection frame and the segmentation mask corresponding to each object name.

[0093] Specifically, the semantic feature is calculated with the environment perception information in the memory library, the maximum of the first N is selected from all the similarity values, and the environment perception information corresponding to the maximum of the first N is selected as the environment perception information similar to the semantic feature.

[0094] S204: content matching is performed on the at least one environment perception information and the user instruction to obtain the environment perception information after the first deduplication; the content matching is used to determine the environment perception information matched with the object name related to the user instruction in the at least one environment perception information;

[0095] Among them, the matching can be realized by similarity or distance. The core purpose of this content matching and deduplication step is to filter out the information most directly related to the user instruction and with the highest uniqueness from these candidate information, and to eliminate redundant data, so as to provide refined and non-redundant input for subsequent action planning.

[0096] Specifically, after obtaining a group of candidate environment perception information from the memory library through semantic retrieval, there may be multiple groups of environment perception information that are completely irrelevant, but the semantic features corresponding to them may be similar. At this time, content matching needs to be performed on the object names corresponding to the environment perception information, and the multiple environment perception information corresponding to the similar semantic features are matched with the user instruction to determine the environment perception information corresponding to the object name required by the user instruction. For example, there are semantic feature 1 and semantic feature 2, and the object names corresponding to the two are respectively: water cup; cat. According to the user instruction in the above example, it is obvious that the cat has nothing to do with the user instruction, and based on this, the cat-related environment perception information needs to be deleted.

[0097] S205: based on the fourth prompt word, target matching is performed on the environment perception information after the first deduplication, secondly deduplicated environment perception information is obtained, and it is determined as the environment perception information related to the user instruction; wherein the fourth prompt word is used to control the large language model, and in the multiple environment perception information representing the same object name, the matched environment perception information is determined.

[0098] Wherein, target matching refers to matching the target object in the prompt word with the object in the environment perception information, and selecting the target object that is more consistent with the user instruction.

[0099] Specifically, after the first deduplication of the environment perception information representing different entities based on simple rules, there may still be multiple candidate information representing the same entity, which represents different physical entities but conforms to the basic object type and attribute (for example, there are "multiple blue cups" on the table). In order to accurately identify the one that the user really means from these highly similar candidate objects in terms of semantics and vision, the powerful context reasoning ability of the large language model is used to simulate the process of human judgment based on common sense and dialogue context, and the final target locking is realized.

[0100] The present application can efficiently and accurately determine the matched environment perception information from multiple environment perception information representing the same object name based on the prompt word and the large language model.

[0101] For the steps of S3, the specific implementation process of the method includes:

[0102] S301: obtaining a list of sub-actions supported by the robot;

[0103] Wherein, the sub-action list is used to store multiple sub-actions.

[0104] S302: filling the sub-action list and the environment perception information related to the user instruction into the pre-configured first prompt word template to obtain the first prompt word;

[0105] Wherein, the replacement symbols in the first prompt word template include the data formats of the sub-action list, the environment perception information, the user instruction and the overall action sequence respectively corresponding to the replacement symbols; each replacement symbol is used to generate the first prompt word matching the user instruction.

[0106] Specifically, in order to guide the large language model to generate a high-quality action sequence that conforms to the robot's capability constraints and fits the actual environment, this step integrates the robot's internal capabilities (sub-action list) and external environment state (environment perception information) through a first prompt word template. The template structures all the context information required for planning and fills it into the template to form the first prompt word finally submitted to the large language model.

[0107] In one example, the first prompt word can be an action sequence prompt word, the structure of which can be: "You are an action generation expert, the actions that the current robot can perform include ' ', the current environment information includes ' ', the user wants the robot to complete the instruction ' ', please generate an action sequence that can complete the user's instruction based on the executable action, the actual control range of the environment information, the format of the action sequence is ' ', and the final output is the overall action plan."

[0108] S303: input the first prompt word into the large language model to obtain an overall action sequence corresponding to the user instruction.

[0109] Specifically, the large language model decomposes the user instruction (such as "pick up an apple") into multiple sub-tasks according to the first prompt word. Match one or more specific atomic actions from the sub-action list for each sub-task; determine the order between actions.

[0110] The present application can accurately obtain an overall action sequence corresponding to the user instruction according to the prompt word and the large language model.

[0111] For the steps of S4, the specific implementation process of the method includes:

[0112] S401: traverse each sub-action in the overall action sequence in chronological order, and obtain the current sub-action, the actuator of the robot, and the multiple basic actions that the actuator can execute;

[0113] Among them, the multiple basic actions that the actuator can execute refer to the most basic action instructions directly supported by the robot bottom hardware (such as motors, servos, hydraulic cylinders, etc.) and driving software, which cannot be further divided or are preset by the system. These basic actions constitute the capability boundary of all complex tasks of the robot, and are the atomic operation units of the action sequence generated by the task planning layer to be finally converted into physical motion.

[0114] S402: fill the current sub-action, the actuator, and the multiple basic actions that the actuator can execute into the pre-configured second prompt word template to obtain a second prompt word;

[0115] Among them, the second prompt word is composed of the current sub-action, the actuator, and the multiple basic actions that the actuator can execute.

[0116] Specifically, the second prompt word, i.e. the action generation prompt word, can be: "You are an action generation expert, and the current sub-action that needs to be generated in detail is 'the current sub-action in the overall action plan ', the hardware device (actuator) of the executable sub-action robot" ”, the basic actions include: ’ based on the basic actions of each hardware device, generate operation information corresponding to the sub-actions, and determine the positioning information changed before and after the execution of each basic action.

[0117] S403: input the second prompt word into the large language model to obtain the detailed action plan corresponding to the current sub-action.

[0118] Specifically, the sub-actions in the overall action sequence indicate the target, but do not specify the specific movement path, posture and obstacle avoidance strategy. Direct execution of such actions may cause collision or failure. The purpose of this step is to solve “how to safely and efficiently complete this sub-action”. The second prompt word provides the specific context required to complete the sub-action (such as real-time target coordinates, obstacle information), guiding the large language model to conduct microscopic movement planning.

[0119] The present application can obtain the detailed action plan corresponding to each sub-action in the overall action sequence to guide the robot to flexibly complete the user instruction.

[0120] For the steps of S3, S3 further includes:

[0121] The overall action sequence corresponding to the user instruction is verified by the first verifier, and if the verification fails, the overall action sequence and the corresponding failure result are input into the large language model to regenerate the overall action sequence corresponding to the user instruction; wherein the first verifier is determined based on the pre-configured verification rule and the feasibility of the overall action sequence analyzed by the large language model.

[0122] Specifically, the first verifier is used to check whether the action sequence in the overall action sequence conforms to the basic logic of the physical world. For example, the grasping action must be after moving to the position, and the releasing action must be after grasping.

[0123] For the steps of S4, S4 further includes:

[0124] The detailed action plans generated by the large language model are verified by the second verifier, and if the verification fails, the detailed action plans and the corresponding failure results are input into the large language model to regenerate the detailed action plans corresponding to the overall action sequence until the verification is successful.

[0125] Specifically, whenever the large language model generates a detailed action plan corresponding to a sub-action, the plan is verified by the second verifier. The second verifier has a different division of labor from the first verifier, and its core task is to evaluate the safety, accuracy and executability of each detailed action from the microscopic physical level, to ensure that each movement of the robot is safe, collision-free and conforms to the physical constraints.

[0126] The application verifies the detailed action plan generated by the large language model through the verifier, thereby improving the correctness of the action generated by the large language model.

[0127] In one exemplary embodiment, before S1, the method further comprises:

[0128] Receiving object information in each environment captured by the information acquisition device of the robot; the object information includes object names, image information corresponding to the object names, and position information; the objects include environmental objects, large object, and small object; based on a pre-configured data structure, the image information and the position information in the object information are processed to obtain environmental perception information corresponding to each object name; and the environmental perception information is uploaded to a memory library.

[0129] Specifically, through the camera, laser radar and other information acquisition devices of the robot, the image information and position information of each object in the scene where the robot performs the task are obtained and updated, and the specific implementation process is as follows:

[0130] A standard format for synchronizing environmental perception information is defined, which includes image information and position information. The image information at least includes RGB image, depth image, object category, object detection frame and object corresponding segmentation mask, etc. The image information can be obtained by binocular camera, laser radar and other devices, and processed into corresponding RGB image and depth image. Through the image recognition model and the image segmentation model, the types of each object in the image can be recognized, and for each object category, the detection frame of the object and the segmentation mask of the object are identified, and the above data is packaged and stored as environmental perception information. The positioning information of the robot itself can be obtained first, and the information is taken as the reference, combined with the depth image containing the object, and the position relationship between each object in the depth image and the positioning information of the robot itself is determined through algorithm, so as to determine the actual position of each object in the image.

[0131] Through the camera and image processing side of the robot, the information obtained from each frame of image is extracted according to the defined format to obtain the environmental perception information; each group of images will send the environmental perception information to the server of the robot after being processed; after receiving the environmental perception information sent each time, the server puts the environmental perception information into the cache queue and waits for processing.

[0132] In one embodiment of the application, the specific implementation process of the step "uploading the environmental perception information to the memory library" includes:

[0133] The plurality of environment perception information in the cache queue is de-duplicated in chronological order to obtain de-duplicated environment perception information; the de-duplication processing at least includes environment object de-duplication processing and object object de-duplication processing; according to the detection frame in the environment perception information, the image information corresponding to each object is intercepted, and the image information is input into a semantic coding model to code to obtain the semantic features corresponding to the object; the semantic features and the environment perception information associated with the semantic features are uploaded to the memory bank in a preset format; the memory bank is used to recall environment perception information with high similarity from similar semantic features.

[0134] Specifically, the environment perception information is obtained from the cache queue in sequence; the environment perception information is de-duplicated twice, once for environment de-duplication and once for object de-duplication. First, each piece of environment perception information in the cache queue is input into a scene de-duplication module for processing, and the environment perception information under repeated environments is deleted, and high-quality environment perception information is retained. Secondly, according to the data with detection and segmentation in the environment perception information (including RGB image, depth image, object category, detection frame, segmentation mask, etc.), the point cloud volume corresponding to each object is calculated. According to the point cloud volume corresponding to the object and the segmentation mask area, the object is divided into large objects and small objects, and it is judged whether the large objects and the objects in the constructed memory bank are repeated, and whether the small objects and the objects in the constructed memory bank are repeated; if repeated, the large object or small object is not added to the memory bank, if not repeated, the large object or small object is considered as a new object, and the new object is stored in the memory bank.

[0135] According to the detection frame in the de-redundant environment perception information, the image of the corresponding object is intercepted and input into the trained multi-modal feature coding model to code to obtain the semantic features corresponding to the object; the semantic features and the environment perception information associated with the semantic features are uploaded to the memory bank in a preset format. In addition, the semantic features can also be used as an index to retrieve the environment perception information containing the object. That is, as long as the user has the semantic features, the environment perception information corresponding to the object can be retrieved in the memory bank.

[0136] The present application improves the efficiency and accuracy of information recognition and matching by de-duplicating the environment perception information multiple times.

[0137] In one exemplary embodiment, after S4, the method further comprises:

[0138] For the detailed action plan corresponding to each sub-action, current environment perception information before execution of the current sub-action is obtained; the execution command corresponding to the detailed action plan of the current sub-action is issued to each actuator of the robot; the actuator is used to instruct the hardware device of the robot to perform a related control operation; the control operation at least includes navigation, grabbing, releasing, opening and closing; after the actuator receives the corresponding execution command and executes, the latest environment perception information after execution of the current sub-action is obtained; the current environment perception information before execution of the current sub-action and the latest environment perception information after execution are input into the large language model to obtain a sub-action verification result, until the detailed action plans of all sub-actions of the overall action sequence are all verified.

[0139] Specifically, the environment perception information corresponding to the latest received image frame is saved, and the execution command corresponding to the single action plan is issued to the corresponding actuator; the actuator executes the corresponding task according to the execution command received by it, parses the execution command and executes the specific action plan; the environment perception information after action execution is obtained; the environment perception information corresponding to the latest received image frame is saved; whether the detailed planned action plan is executed successfully is verified.

[0140] In the embodiment of the application, the large language model is used to verify the sub-action, which can discover possible problems in the execution process of the sub-action as soon as possible and correct them in time, thereby improving the correctness of the sub-action execution.

[0141] In an embodiment of the application, the specific implementation process of the step "inputting the current environment perception information before execution of the current sub-action and the latest environment perception information after execution into the large language model to obtain a sub-action verification result" includes:

[0142] Based on the current environment perception information before execution of the current sub-action and the latest environment perception information after execution, a sub-action verification prompt word is generated; the sub-action verification prompt word is input into the large language model, so that the large language model determines whether the first coordinate value of the target object in the current environment perception information and the second coordinate value of the target object in the latest environment perception information match the expected result of the current sub-action, and if the first coordinate value and the second coordinate value match the expected result of the current sub-action, it is determined that the sub-action verification result is verified.

[0143] Specifically, the sub-action verification prompt word is determined to inform the large model how to complete the action verification task according to the instruction of the current to-be-completed action and the two environment perception information before and after action execution; the current instruction and the environment perception information before and after action execution are organized into input data according to the format defined in the prompt word; the input data is input into the large language model to obtain a verification result.

[0144] Exemplarily, the action of obtaining obtains environment perception information related to the object in the user instruction before the action is executed, for example, coordinates 1 of the object A; the action of obtaining obtains environment perception information related to the object in the user instruction after the action is executed, for example, coordinates 2 of the object A; and the action of determining determines whether the environment perception information of the two frames before and after the action is executed matches the coordinate information of the object in the detailed action plan. If the two match, it is determined that the detailed action plan is completed. For example, it is determined whether the coordinates 2 of the object A are close to the position where the object A is intended to move in the detailed action plan. If the two are close, it is determined that the two match.

[0145] The embodiment of the application further provides a task planning system of a robot, which implements the task planning method of the robot of the above embodiment, and comprises:

[0146] A user instruction obtaining module is configured to obtain a user instruction, wherein the user instruction is used to instruct the robot body to execute a task of a non-fixed process;

[0147] An environment perception information retrieving module is configured to retrieve environment perception information related to the user instruction from a memory bank based on a large language model and a semantic encoding model, wherein the large language model is used to analyze the user instruction to obtain a candidate object related to the user instruction, and the semantic encoding model is used to encode the candidate object into semantic features in the same format as the memory bank;

[0148] A whole action sequence generating module is configured to input a first prompt word determined based on the user instruction and the environment perception information into the large language model to generate a whole action sequence corresponding to the user instruction, wherein the first prompt word at least further comprises a plurality of sub-actions supported by the robot;

[0149] A detailed action plan generating module is configured to determine a second prompt word for each sub-action in the whole action sequence, input the second prompt word into the large language model, and generate a detailed action plan corresponding to the current sub-action until all the sub-actions in the whole action sequence are traversed, wherein the detailed action plan is composed of a plurality of basic actions supporting the sub-action.

[0150] It should be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working process of the task planning system of the robot described in the above embodiment can refer to the corresponding process in the task planning method of the robot described in the foregoing embodiment, which will not be described herein.

[0151] For the convenience of understanding, the application provides the following more specific embodiments:

[0152] As shown in Figure 2 the application scenario of the application is a home service robot, and the overall concept is:

[0153] 1. Preparation: Construct a memory base for the home scene to obtain feature data and location data of each item in the scene, so that the robot can remember the items and their locations in the scene.

[0154] 2. After receiving the user's instruction, based on the large language model, recall the information related to the user's instruction in the memory base.

[0155] 3. Through the large language model, let the robot think, and based on the user's instruction and related information, generate a complete action sequence that can complete the user's instruction, as well as the action execution plan for each action.

[0156] 4. Execute the plan, verify whether the action is completed through the large language model, complete the execution and traverse the next action, if not, adjust the action execution plan in time, re-execute the action, repeat the above steps until all actions in the overall action sequence are completed.

[0157] Step 1: Receive environmental perception information of the home scene.

[0158] Function: Through the robot's camera, laser radar and other information acquisition devices, obtain and update the image information and location information of each item in the home.

[0159] Implementation:

[0160] 1. Define a standard format for synchronizing environmental perception information, which contains image information and location information. Image information includes at least RGB image, depth image, object category, object detection box and object corresponding segmentation mask, etc.

[0161] Image information can be obtained by binocular camera, laser radar and other devices, and processed into corresponding RGB image and depth image. Through image recognition model and image segmentation model, the types of objects in the image can be identified, and for each object category, the detection box of the object and the segmentation mask of the object can be identified, and the above data can be packaged and stored as environmental perception information.

[0162] Location information can first obtain the positioning information of the robot itself, and based on this information, combined with the depth image containing the object, determine the location relationship between each object in the depth image and the robot's own positioning information, and thus determine the actual location of each object in the image.

[0163] Data structure example:

[0164] Each image processed will be organized into the following data format, representing all the perception information corresponding to the current image.

[0165] {

[0166] "agent_room": "bobs_room", / / Custom scene, which can be determined based on the coordinate range of the electronic fence.

[0167] "image_RGB_url": "url_RGB.jpg", / / RGB image

[0168] "image_depth_url": "url_depth.png", / / Depth map

[0169] "world2camera": [0.866, -0.5, 0, 2, 0.5, 0.866, 0, 3, 0, 0, 1, 5, 0,0, 0, 1], / / The pose information corresponding to the camera mounted on the robot body, 16 / 4 A 4-dimensional matrix vector, pose information includes position and orientation information, such as coordinates and angles.

[0170] "camera_intrinsics":[606.0801391601562,0.0, 318.43902587890625,0.0,606.3120727539062, 246.55946350097656, 0.0, 0.0, 1.0], / / Camera intrinsics, hardware parameters, generally not changed;

[0171] "instance": [ / / example]

[0172] {

[0173] "object_id": "12", / / ID

[0174] "name": "coffee_mug1",

[0175] "room": "bobs_room", / / Environment

[0176] "position":{"x":34.53,"y":23.82, "z":10.31}, / / Item position information

[0177] "detection_2d":{"left_top":[45, 560],"right_bottom": [78, 780]}, / / The range of the detection box for the item in the RGB image.

[0178] "segmentation": [[23,345], [234, 345]], / / The segmentation mask corresponding to this item;

[0179] },

[0180] {

[0181] "object_id": "15", / / The object's ID is the string "15"

[0182] "name": "coffee_mug2", / / The object is a coffee cup named "coffee_mug2".

[0183] "room": "bobs_room", / / This object is located in a room named "bobs_room"

[0184] "position":{"x":34.53,"y":23.82,"z":10.31}, / / The precise position of the object in the room, x=34.53, y=23.82, z=10.31

[0185] "detection_2d":{"left_top":[45, 60],"right_bottom": [78, 780]}, / / The bounding box of the object in the image, with the top-left pixel coordinates [45,60] and the bottom-right pixel coordinates [78,780]

[0186] "segmentation": [[27,145], [24, 325]], / / Coordinates of the vertices of the polygon outline of the object in the image, for example, point 1 [27,145], point 2 [24,325]

[0187] } ]

[0189] 2. Using the robot's camera and image processing side, the information acquired in each frame is perceived according to the defined format to obtain environmental perception information; environmental perception information is sent once after each set of images is processed.

[0190] 3. After receiving each environmental perception information sent, the robot's server puts the data into a buffer queue for processing.

[0191] Step 2: Build the memory.

[0192] Function: The server performs redundancy removal processing on the received environmental perception information and builds a perception memory library.

[0193] Implementation method:

[0194] 1. Obtain the environment perception information from the cache queue in sequence;

[0195] 2. Perform two deduplication processes on the environment perception information, one being scene deduplication and the other being object deduplication. First, input each piece of environment perception information in the cache queue into the scene deduplication module for processing, delete the environment perception information under repeated scenes, and retain high-quality environment perception information. Second, calculate the point cloud volume corresponding to each object based on the data with detection and segmentation in the environment perception information (including RGB images, depth images, object categories, detection boxes, segmentation masks, etc.).

[0196] When the point cloud volume exceeds the set value, the data is sent to the large object deduplication module for processing, and when the point cloud volume is less than the set value, the data is sent to the small object deduplication module for processing. The large object deduplication module and the small object deduplication module are based on two deduplication logics and perform deduplication operations on environment perception information containing repeated objects.

[0197] In one example, the logic for small object deduplication is as follows:

[0198] 1. Each new object added to the memory library will first be deduplicated based on the object's ID number. Each object in the environment perception data (information) will have a unique ID number. If the ID numbers are the same, it is considered that the objects are the same.

[0199] 2. If the ID of the newly added object is different from each number in the memory library, use a multi-modal encoding model to extract the image feature information of the object, and then use the feature information to search for possible candidate matching objects in the vector library with a similarity higher than a set threshold.

[0200] 3. If there are no candidate objects, it is considered that the object is a new object that is not recorded in the memory library. The object is directly added to the memory library as a new object.

[0201] 4. If there are candidate objects, calculate the distance to all candidate objects, then sort the distance values, and select the object with the smallest distance value as the possible repeated object. If the minimum distance value is less than the set threshold, it is considered that the newly added object and the object corresponding to the minimum distance are the same object. If the minimum distance value is greater than the set threshold, it is considered that the newly added object is an object that is not in the memory library.

[0202] The logic for large object deduplication is as follows:

[0203] 1. For each new large object added, extract the detection box, segmentation mask, and image pixel coordinate to world coordinate conversion relationship of each object in the environment perception data.

[0204] 2. The image of the corresponding object is intercepted according to the detection frame, and is sent into a multi-modal coding model to obtain the corresponding semantic vector information;

[0205] 3. The point cloud of the corresponding object in the world coordinate system is calculated by using the segmentation mask of the object and the conversion relationship of the image pixel coordinates to the world coordinates;

[0206] 4. The similarity between the semantic features of the object and the existing large objects in the existing memory bank is calculated, and the large object with a similarity greater than a set threshold is selected as a candidate, and the intersection over union of the point cloud of the candidate is calculated, and the large object with the maximum intersection over union and exceeding the set threshold is selected as the large object matched with the newly added large object, and if there is no match, the newly added large object is considered as a new large object;

[0207] 5. For the matched large object (each large object has multiple images with different perspectives), the intersection over union of the point cloud between the newly added large object and all images inside the matched large object is calculated, and the image of the newly added large object is replaced with all images with an intersection over union greater than a set threshold, and the new image information representing the large object is obtained.

[0208] 3. Each sub-module processes the corresponding data to remove data with repeated content and low information quantity.

[0209] 4. The image of the corresponding object is intercepted according to the detection frame in the environmental perception information, and is sent into a trained multi-modal feature coding model to obtain the semantic features corresponding to the object.

[0210] 5. The semantic feature is used as an index of the vector library to construct a semantic understanding vector library for subsequent query. In addition, the semantic feature is used as an index, so the environmental perception information containing the object can also be retrieved. That is, as long as the user has the semantic feature, the environmental perception information corresponding to the object can be retrieved in the vector library.

[0211] In one example, the semantic feature-RGB image-depth map-object category-detection frame-segmentation mask-position information is used as the data structure of an object stored in the vector library; for example, the stored structure is the same as the above perception data structure, and the data in the vector library is in the form of a string. The information stored at present is in the format of perception information, which is a json format string. The json format is mainly used to reflect the hierarchical relationship structure of the data.

[0212] Step three: receiving user instructions.

[0213] Function: listening to the instruction information sent by the user, such as receiving voice information and converting it into natural language.

[0214] Implementation:

[0215] 1. Listen to the content appearing in the instruction issuing pipeline, for example, receive user instructions through OCR text recognition, Chinese text input, and voice conversion text, and finally convert them into text format instructions;

[0216] 2. When the user instruction is issued, according to the order of receiving the user instruction, the command is put into the instruction processing pipeline.

[0217] Step four: memory bank recalls related information.

[0218] Function: understand user instructions and find highly relevant environmental perception information in the memory bank;

[0219] Implementation:

[0220] 1. Analyze the highly relevant candidate object names of the user instruction by using the pre-configured entity extraction prompt word and the large language model;

[0221] 2. Encode the candidate object names using a multi-modal semantic encoding model to obtain the semantic features of the object; the multi-modal semantic encoding model can encode text and image data, and encode text and image data expressing the same meaning into similar semantic features. Therefore, users can find matching images through semantic features, that is, retrieve corresponding images through text, or retrieve corresponding text through images;

[0222] 3. Use the encoded semantic features to recall related environmental perception information in the memory bank one by one, and perform content matching and deduplication on the multiple environmental perception information to obtain the most suitable environmental perception information;

[0223] 4. Use the large language model to make a second judgment on the matched information to remove irrelevant information and ensure that the remaining information is highly relevant to the user instruction.

[0224] For example, the recall content of the vector library has no logical judgment and will only recall related items, such as helping you get a bottle of water on the kitchen table. The vector library may recall several bottles of water in different positions, and the multi-modal large model needs to make a second judgment on whether this bottle of water is the water on the kitchen table, so as to remove irrelevant environmental perception information.

[0225] Specifically, the content of the entity extraction prompt word can be "You are a natural language processing expert, you need to identify the operable entities contained in the above sentence and give the object name to the entity; The name given can be the name in the sentence, or a more general alias after the name in the sentence is sorted." After inputting the entity extraction prompt word into the large language model, multiple candidate object names can be obtained.

[0226] Optionally, the prompt word of the judgment process in step 4 can be "Please determine the environment perception information related to the entity and the environment perception information unrelated to the entity according to the environment perception information of the following objects and the entities in the 'user command', and output the environment perception information unrelated to the entity." The server can delete the environment perception information unrelated to the entity according to the output of the large language model.

[0227] Step five: generating an overall action sequence.

[0228] Function: generating an overall action sequence based on the recalled environment perception information in the memory bank.

[0229] Implementation:

[0230] 1. Write an action sequence prompt word, which explicitly informs the large language model of the sub-actions currently supported by the robot to complete, and commands the relevant environment perception information, which needs to be analyzed by the large language model, and performs the task of generating an overall action sequence. At this time, the large language model can think about how to generate an overall plan and how to generate an overall task plan based on memory information and user instructions according to the prompt word (the output result can be configured: the task plan is a list, which contains multiple robot-executable sub-actions such as grasping, releasing, opening, closing, and navigation in order);

[0231] 2. Organize the user instructions and memory-recalled perception information into the format defined in the prompt word and send it to the large model to generate the corresponding overall action plan;

[0232] 3. Use the verifier to check the generated overall plan, and if problems are found, feed them back to the large model to regenerate the plan, or return the generated overall plan if there are no problems.

[0233] For example, the verifier can be verified by the large language model, or some existing verification methods can be configured for verification. Therefore, the verification method can be based on the large model or program logic.

[0234] Specifically, the structure of the action sequence prompt word is: "You are an action generation expert, the actions that the current robot can perform include 'action list', the current environment information includes 'environment information list', the user wants the robot to complete the instruction is 'user instruction', please generate an action sequence that can complete the user instruction based on executable actions, environment information as the actual control range, the format of the action sequence is 'action sequence format', and the final output is the overall action plan."

[0235] Step six: generating a single action detailed plan. ​​​​

[0236] Function: Refine each sub-action in the execution of the global plan according to the field situation, and generate a task plan that can be actually issued and executed.

[0237] Implementation:

[0238] 1. Take out a sub-action in the whole action sequence (for example: navigate to the destination point, grab the target object, release the object to the target position);

[0239] 2. Write action generation prompt words, in which it is informed that the current task of the large model is to generate detailed action plans, and each sub-action detailed action plan mainly includes multiple basic actions, each basic action can include operation information and positioning information. Positioning can be divided into two forms, which are frame and point. If it is a robot navigation operation and a mechanical arm release operation, the detailed action plan is to draw a point on the image, indicating the position to be approached or the position of the object to be released. If it is a mechanical arm for grabbing, opening and closing operation, the detailed action plan is to draw a frame, which mainly frames the target object to be operated;

[0240] 3. Verify the single action plan generated by the model, and if the verification fails, feedback the problem to the model to regenerate the plan until the verification passes and return the detailed action plan after the verification.

[0241] Specifically, the action generation prompt words can be: "You are an action generation expert, and you need to generate a detailed sub-action in the whole action plan The hardware devices of the robot that can execute the sub-action ” include: ‘ ’, please generate the operation information of the corresponding sub-action based on the basic actions of each hardware device, and determine the positioning information changed before and after the execution of each basic action.”

[0242] Step seven: Obtain the environmental perception information before action execution.

[0243] Function: Obtain the perception information before action execution, which is used for verifying whether the action is executed successfully.

[0244] Implementation:

[0245] 1. Save the latest received frame of environmental perception information.

[0246] Step eight: Execute the action.

[0247] Function: Specifically execute the detailed planned action plan.

[0248] Implementation:

[0249] 1. Issue the execution command corresponding to a single action plan to the corresponding executor (existing executors: navigation, grab, release, open, close);

[0250] 2. After the executor accepts the task, it parses and executes the command and the specific action plan.

[0251] In one example, the specific data format is defined as follows; the format may differ slightly for different actions:

[0252] {

[0253] "cmd_type": "pickup | open | close | turn_on | turn_off", / / Current operation (pickup / open / close / turn device on / off)

[0254] "obj_type": "apple", / / The object being operated on is "apple".

[0255] "image_RGB_url": "url_RGB.jpg", / / URL to store the color image of the object

[0256] "image_depth_url": "url_depth.png", / / URL of the image that stores the object's depth information

[0257] "detection_2d": {"left_top":[45, 560],"right_bottom": [78, 780]}, / / The position of the object in the image, with the top left corner coordinates [45,560] and the bottom right corner coordinates [78,780]

[0258] }

[0259] {

[0260] "cmd_type": "release", / / Release the object

[0261] "obj_type": "apple", / / The object being released is an "apple".

[0262] "image_RGB_url": "url_RGB.jpg", / / Color image of the scene at the time of release

[0263] "image_depth_url": "url_depth.png", / / Scene depth information at the time of release

[0264] "point_2d": [45, 560], / / Target position of the release operation (image pixel coordinates [x=45, y=560])

[0265] }

[0266] {

[0267] "cmd_type": "access", / / Access or locate an object

[0268] "obj_type": "apple", / / The object being located is an "apple".

[0269] "position": {"x":34.53, "y":23.82, "z":10.31}, / / Position information of the large object: x=34.53, y=23.82, z=10.31

[0270] }

[0271] Step 9: Obtain environmental awareness information after the action is executed;

[0272] Function: Acquire sensory information after an action is performed, and then use it to verify whether the action was successfully executed.

[0273] Implementation method:

[0274] 1. Save the latest received frame of environmental awareness information.

[0275] Step 10: Verify the execution results.

[0276] Function: Verify whether the detailed action plan has been executed successfully.

[0277] Implementation method:

[0278] 1. Write verification prompts to inform the large model how to complete the action verification task based on the instruction to be performed and the two environmental perception information before and after the action execution;

[0279] 2. Organize the current instruction and the environmental awareness information before and after execution into an input format according to the format defined in the prompt words;

[0280] 3. Feed the input data to the large model, obtain and return the validation results.

[0281] Specifically, such as Figure 1 The process shown involves two sets of loops: one for generating the overall sequence of actions and the other for generating detailed plans for individual actions.

[0282] In the previous iteration of generating the overall action sequence, the large language model can be prompted to verify whether the robot has completed the user command content based on the user command content and the current environmental perception information. An example of the specific verification logic is as follows:

[0283] 1. Determine the predicted perception information of each object after completing the user command;

[0284] 2. Compare the actual perception information in the memory library with the predicted perception information to determine whether they are consistent;

[0285] 3. If the information is consistent, it is determined that the robot has successfully completed the user command content.

[0286] In the latter iteration of generating a single action detailed plan, the large language model can be prompted to verify whether each action detailed plan is successfully completed based on each action detailed plan and the image data of the two frames before and after executing the detailed plan. If it is successfully completed, the iteration is completed. If it is not completed, the action detailed plan is repeated. If it cannot be completed after repeated execution, the large language model can be used to regenerate the overall action plan and update the image data of the two frames before and after execution in order to verify again whether the execution is successfully completed.

[0287] An example of the specific verification logic is as follows:

[0288] 1. Obtain the environmental perception information related to the items in the user command before action execution, such as the coordinates 1 of item A;

[0289] 2. Obtain the environmental perception information related to the items in the user command after action execution, such as the coordinates 2 of item A;

[0290] 3. Determine whether the environmental perception information of the two frames before and after action execution matches the coordinate information of the items in the action detailed plan. If they match, it is determined that the action detailed plan is completed. For example, it is determined whether the coordinates 2 of item A are close to the position where item A is intended to move in the action detailed plan. If they are close, it is determined that they match.

[0291] The technical effects achieved by the above specific embodiments of the present application are as follows:

[0292] 1. The present application can receive natural language or unstructured instructions, understand user intent, and generate executable overall action sequences. At the same time, the present application also has action monitoring and dynamic adjustment capabilities to ensure that the robot can respond to environmental changes or unexpected situations during execution. For example, when executing the task of "getting a cup of water", the method plans the movement path, identifies the water cup position, adjusts the robotic arm grasping posture, and re-adjusts the action sequence when detecting that the water cup has been moved, thereby ensuring the successful completion of the task.

[0293] 2. When facing a cross-scene random task issued by a user, the robot can configure a set of task execution processes according to scene information and command content, and can decompose the task execution process into multiple actions, and can complete multiple actions through the iteration of execution-verification, thereby completing the non-fixed process issued by the user, and improving the accuracy, precision and efficiency of the robot in executing the user's command.

[0294] 3. The environment perception information related to the user instruction is retrieved from the memory bank to support the generation of the overall action sequence, thereby solving the cross-scene task generation problem.

[0295] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, but not to limit it, although the present application has been described in detail with reference to the above examples, those skilled in the art should understand that: the specific embodiments of the present application can still be modified or replaced equivalently without departing from the spirit and scope of the present application, any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered within the protection scope of the claims of the present application.

Claims

1. A task planning method for robots, applicable to servers, characterized in that, include: Step 1: Obtain user instructions; the user instructions are used to instruct the robot to perform tasks with non-fixed processes. Step 2: Based on the large language model and semantic coding model, retrieve the environmental awareness information related to the user command from the memory bank; The large language model is used to parse the user instruction and obtain candidate objects related to the user instruction. The semantic encoding model is used to encode the candidate objects into semantic features in the same format as the memory. Step 3: Input the first prompt word determined based on the user instruction and the environmental perception information into the large language model to generate the overall action sequence corresponding to the user instruction; the first prompt word also includes at least several sub-actions supported by the robot; Step 4: For each sub-action in the overall action sequence, determine the second prompt word, input the second prompt word into the large language model, and generate the detailed action plan corresponding to the current sub-action, until all sub-actions in the overall action sequence have been traversed; The detailed action plan is based on multiple basic actions that support the sub-actions; Step 2 includes: A third prompt word is obtained and input into a large language model to obtain the object name related to the user command; the third prompt word is used to control the large language model to parse the user command. Based on the semantic encoding model, the semantic features of the object name are determined; Based on the semantic features, at least one piece of environmental awareness information similar to the semantic features is retrieved from the memory bank; the semantic encoding model is a multimodal semantic encoding model, used to encode text and images expressing the same meaning into the same semantic features; Content matching is performed on the at least one piece of environmental perception information and the user instruction to obtain the environmental perception information after the first deduplication; the content matching is used to determine the environmental perception information that matches the object name related to the user instruction in the at least one piece of environmental perception information. Based on the fourth prompt word, target matching is performed on the environmental perception information after the first deduplication to obtain the environmental perception information after the second deduplication, and it is determined as the environmental perception information related to the user command. The fourth prompt word is used to control the large language model to determine the matching environmental perception information among multiple environmental perception information representing the same object name.

2. The robot task planning method according to claim 1, characterized in that, Step 3 includes: Obtain a list of sub-actions that the robot can support; The sub-action list is filled into a pre-configured first prompt word template to obtain a first prompt word; the replacement characters in the first prompt word template include replacement characters corresponding to the data formats of the sub-action list, environmental awareness information, user instructions, and the overall action sequence; each of the replacement characters is used to generate the first prompt word that matches the user instructions; The first prompt word is input into the large language model to obtain the overall action sequence corresponding to the user command.

3. The robot task planning method according to claim 1, characterized in that, Step 4 includes: According to the time sequence, traverse each sub-action in the overall action sequence and obtain the current sub-action, the robot's actuator, and the multiple basic actions that the actuator can perform. The current sub-action, the executor, and the multiple basic actions that the executor can perform are filled into a pre-configured second prompt word template to obtain the second prompt word; The second prompt word is input into the large language model to obtain the detailed action plan corresponding to the current sub-action.

4. The robot task planning method according to claim 1, characterized in that, Step 3 also includes: The first verifier verifies the overall action sequence corresponding to the user instruction. If the verification fails, the overall action sequence and the corresponding failure result are input into the large language model to regenerate the overall action sequence corresponding to the user instruction. The first verifier is determined based on pre-configured verification rules and the feasibility of analyzing the overall action sequence based on a large language model. Step 4 also includes: The second validator verifies each detailed action plan generated by the large language model. If the verification fails, the detailed action plan and the corresponding failure result are input into the large language model to regenerate each detailed action plan corresponding to the overall action sequence until the verification is successful.

5. The robot task planning method according to claim 1, characterized in that, Before step 1, the following are also included: The robot receives information from the acquisition device, which captures object information in various environments. The object information includes the object name, as well as the image information and location information corresponding to the object name. The objects include environmental objects, large object objects, and small object objects. Based on a pre-configured data structure, the image information and location information in the object information are processed to obtain the environmental perception information corresponding to each object name; the environmental perception information includes at least the RGB image, depth map, object category, detection box, and segmentation mask corresponding to each object name; The environmental perception information is uploaded to the memory bank.

6. The robot task planning method according to claim 5, characterized in that, Uploading the environmental perception information to the memory bank includes: Multiple environmental awareness information entries in the cache queue are deduplicated according to time order to obtain deduplicated environmental awareness information; the deduplication process includes at least environmental object deduplication and object object deduplication. Based on the detection boxes in the environmental perception information, image information corresponding to each object is extracted, and the image information is input into the semantic coding model to encode the semantic features corresponding to the object. According to a preset format, the semantic features and the associated environmental perception information are uploaded to a memory bank; the memory bank is used to recall environmental perception information with high similarity through similar semantic features.

7. The robot task planning method according to claim 1, characterized in that, After step 4, the following is also included: For each sub-action, obtain the current environment awareness information before the current sub-action is executed, based on the detailed action plan corresponding to each sub-action; The robot issues execution commands corresponding to the detailed motion plan of the current sub-action to each actuator; the actuators are used to instruct the robot's hardware to perform related control operations; the control operations include at least navigation, grasping, releasing, opening, and closing; After the executor receives and executes the corresponding execution command, it obtains the latest environmental awareness information after executing the current sub-action; The current environmental perception information before the execution of the current sub-action and the latest environmental perception information after the execution are input into the large language model to obtain the sub-action verification result, until the detailed action plans of all sub-actions in the overall action sequence have been verified.

8. The robot task planning method according to claim 7, characterized in that, The step of inputting the current environmental perception information before the execution of the current sub-action and the latest environmental perception information after the execution into the large language model to obtain the sub-action verification result includes: Based on the current environmental awareness information before the execution of the current sub-action and the latest environmental awareness information after the execution, generate sub-action verification prompt words; The sub-action verification prompt is input into the large language model so that the large language model determines whether the first coordinate value of the target object in the current environmental perception information matches the second coordinate value of the target object in the latest environmental perception information. If the first coordinate value and the second coordinate value match the expected result of the current sub-action, the sub-action verification result is determined to be verified as passed.

9. A robot task planning system, implementing the robot task planning method according to any one of claims 1-8, characterized in that, include: The user instruction acquisition module is used to acquire user instructions; the user instructions are used to instruct the robot body to perform tasks with non-fixed processes. The environmental awareness information retrieval module is used to retrieve environmental awareness information related to the user command from the memory bank based on a large language model and a semantic coding model. The large language model is used to parse the user instruction and obtain candidate objects related to the user instruction. The semantic encoding model is used to encode the candidate objects into semantic features in the same format as the memory. The overall action sequence generation module is used to input a first prompt word determined based on the user instruction and the environmental perception information into a large language model to generate an overall action sequence corresponding to the user instruction; the first prompt word also includes at least several sub-actions supported by the robot; The detailed action plan generation module is used to determine a second prompt word for each sub-action in the overall action sequence, input the second prompt word into the large language model, and generate a detailed action plan corresponding to the current sub-action, until all sub-actions in the overall action sequence have been traversed; the detailed action plan is composed of multiple basic actions that support the sub-action.

Images