Task planning method and system for robot

Abstract

The invention relates to the technical field of robots, and discloses a task planning method and system for a robot, and the method comprises the steps: obtaining a user instruction; based on the large language model and the semantic coding model, retrieving in a memory bank to obtain environment perception information related to the user instruction; inputting a first cue word determined based on the user instruction and the environment perception information into the large language model, and generating an overall action sequence corresponding to the user instruction; and for each sub-action in the whole action sequence, determining a second cue word, inputting the second cue word into the large language model, and generating a detailed action plan corresponding to the current sub-action until each sub-action in the whole action sequence is traversed. According to the method, the natural language or unstructured instruction can be received, the intention of the user can be understood, and the executable overall action sequence and the detailed action plan of each sub-action in the sequence are generated, so that the task issued by the user is completed, it is ensured that the robot can execute the task, and the accuracy and efficiency of task execution are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a task planning method and system of a robot. BACKGROUND

[0002] With the rapid development of artificial intelligence and robot technology, intelligent robots are gradually applied to many fields such as home service, industrial automation, medical care, etc. However, the traditional robot control method usually relies on pre-programmed fixed processes, for example, in a manufacturing factory, each mechanical arm is a fixed process optimized in advance, and each mechanical arm processes different tasks. When encountering unexpected situations or custom tasks, it lacks flexibility and adaptability, and is difficult to cope with dynamic environments or diversified user needs.

[0003] In the home service scenario, users usually want the robot to complete tasks such as "tidying up the living room" or "delivering a cup of water", which are non-fixed processes, and these tasks involve the coordination of multiple sub-actions (such as navigation, grasping, obstacle avoidance, etc.), and the robot needs to have the ability of task decomposition, logical reasoning and real-time adjustment. However, the existing robot control method can only select the corresponding process from the fixed process to execute, and cannot flexibly change the process according to the issued task, resulting in the robot being unable to correct the process when encountering unexpected situations, and being unable to complete the user's demand for action, ultimately reducing the accuracy, precision and efficiency of the robot in executing tasks. SUMMARY

[0004] In view of this, the present application provides a task planning method and system of a robot.

[0005] The present application discloses a task planning method of a robot, applicable to a server, comprising: Step 1: obtaining a user instruction; the user instruction is used to instruct the robot body to execute a non-fixed process task; Step 2: retrieving environment perception information related to the user instruction in a memory bank based on a large language model and a semantic encoding model; the large language model is used to analyze the user instruction to obtain a candidate object related to the user instruction, and the semantic encoding model is used to encode the candidate object into semantic features in the same format as the memory bank; Step 3: inputting a first prompt word determined based on the user instruction and the environment perception information into a large language model to generate an overall action sequence corresponding to the user instruction; the first prompt word at least contains a plurality of sub-actions supported by the robot; Step 4: determining a second prompt word for each sub-action in the overall action sequence, inputting the second prompt word into a large language model to generate a detailed action plan corresponding to the current sub-action until all sub-actions in the overall action sequence are traversed; the detailed action plan is composed of a plurality of basic actions that support the sub-action.

[0006] Further, the step 2 comprises: obtaining a third prompt word, inputting the third prompt word into the large language model to obtain an object name related to the user instruction; the third prompt word is used to control the large language model to analyze the user instruction; determining a semantic feature of the object name based on the semantic encoding model; retrieving at least one item of environment perception information similar to the semantic feature in the memory bank based on the semantic feature; the semantic encoding model is a multi-modal semantic encoding model, which is used to encode text and image expressing the same meaning into the same semantic feature; performing content matching on the at least one item of environment perception information and the user instruction to obtain first de-duplicated environment perception information; the content matching is used to determine the environment perception information matched with the object name related to the user instruction in the at least one item of environment perception information; performing target matching on the first de-duplicated environment perception information based on a fourth prompt word to obtain second de-duplicated environment perception information, and determining the second de-duplicated environment perception information as the environment perception information related to the user instruction; wherein, the fourth prompt word is used to control the large language model to determine the matched environment perception information in a plurality of environment perception information representing the same object name.

[0007] Further, the step 3 comprises: obtaining a list of sub-actions that the robot can support; filling the sub-action list into a pre-configured first prompt word template to obtain a first prompt word; the replacement symbols in the first prompt word template include sub-action list, environment perception information, user instruction and data format of overall action sequence respectively corresponding replacement symbols; each of the replacement symbols is used to generate the first prompt word matching the user instruction; inputting the first prompt word into the large language model to obtain the overall action sequence corresponding to the user instruction.

[0008] Further, the step 4 comprises: traversing each sub-action in the overall action sequence in chronological order, and obtaining a current sub-action, an actuator of the robot, and a plurality of basic actions that the actuator can perform; Fill the current sub-action, the executor, and a plurality of basic actions that the executor can perform to the pre-configured second prompt word template to obtain a second prompt word; Input the second prompt word into a large language model to obtain a detailed action plan corresponding to the current sub-action.

[0009] Further, the step 3 further includes: Check the overall action sequence corresponding to the user instruction by a first verifier. If the check fails, input the overall action sequence and the corresponding failure result into a large language model to regenerate the overall action sequence corresponding to the user instruction; The first verifier is determined based on a pre-configured verification rule and based on the feasibility of the overall action sequence analyzed by the large language model; The step 4 further includes: Check each detailed action plan generated by the large language model by a second verifier. If the check fails, input the detailed action plan and the corresponding failure result into the large language model to regenerate each detailed action plan corresponding to the overall action sequence until the verification is successful.

[0010] Further, before the step 1, it further includes: Receive object information in each environment captured by the information acquisition device of the robot; the object information includes object name, image information and position information corresponding to the object name; the object includes environmental object, large object and small object; Process the image information and position information in the object information based on a pre-configured data structure to obtain environmental perception information corresponding to each object name; the environmental perception information at least includes RGB image, depth map, object category, detection frame, and segmentation mask corresponding to each object name; Upload the environmental perception information to the memory bank.

[0011] Further, the uploading of the environmental perception information to the memory bank includes: Perform de-duplication processing on a plurality of environmental perception information in the cache queue in chronological order to obtain de-duplicated environmental perception information; the de-duplication processing at least includes environmental object de-duplication processing and object object de-duplication processing; According to the detection frame in the environmental perception information, intercept image information corresponding to each object object, and input the image information into a semantic coding model to encode to obtain semantic features corresponding to the object object; upload the semantic feature and the environment perception information associated with the semantic feature to a memory bank in a preset format; the memory bank is configured to recall environment perception information with high similarity from the memory bank through similar semantic features.

[0012] Further, after step 4, the method further comprises: For each sub-action corresponding to a detailed action plan, obtaining current environment perception information before the execution of the current sub-action; issuing an execution command corresponding to the detailed action plan of the current sub-action to each actuator of the robot; the actuator is configured to instruct the hardware device of the robot to perform a related control operation; the control operation at least includes navigation, grabbing, releasing, opening and closing; After the actuator receives the corresponding execution command and executes it, obtaining the latest environment perception information after the execution of the current sub-action; inputting the current environment perception information before the execution of the current sub-action and the latest environment perception information after the execution into a large language model to obtain a sub-action verification result, until the detailed action plans of all sub-actions of the overall action sequence are verified.

[0013] Further, the method further comprises: generating a sub-action verification prompt word based on the current environment perception information before the execution of the current sub-action and the latest environment perception information after the execution; inputting the sub-action verification prompt word into the large language model to determine whether the first coordinate value of the target object in the current environment perception information and the second coordinate value of the target object in the latest environment perception information match the expected result of the current sub-action, if the first coordinate value and the second coordinate value match the expected result of the current sub-action, the sub-action verification result is determined to be verified.

[0014] The application also discloses a task planning system of a robot, which implements the task planning method of the robot. a user instruction acquisition module configured to acquire a user instruction; the user instruction is configured to instruct the robot body to execute a task of a non-fixed process; an environment perception information retrieval module configured to retrieve environment perception information related to the user instruction from a memory bank based on a large language model and a semantic encoding model; the large language model is configured to analyze the user instruction to obtain a candidate object related to the user instruction, and the semantic encoding model is configured to encode the candidate object into a semantic feature in the same format as the memory bank; The whole action sequence generation module is configured to input a first prompt word determined based on the user instruction and the environment perception information into a large language model to generate a whole action sequence corresponding to the user instruction, and the first prompt word further comprises a plurality of sub-actions supported by the robot. The detailed action plan generation module is configured to determine a second prompt word for each sub-action in the whole action sequence, input the second prompt word into the large language model to generate a detailed action plan corresponding to the current sub-action, and repeat the above process until all sub-actions in the whole action sequence are traversed, and the detailed action plan is composed of a plurality of basic actions supporting the sub-action.

[0015] According to the above technical solution, the present application has the following advantages: the present application obtains a user instruction for indicating a task of a robot body executing a non-fixed process, retrieves environment perception information related to the user instruction in a memory bank based on a large language model and a semantic encoding model, determines a candidate object corresponding to the user instruction through the large language model, determines semantic features of the candidate object through the semantic encoding model, and retrieves environment perception information similar to the semantic features in the memory bank. Then, a first prompt word containing a plurality of sub-actions supported by the robot is determined according to the user instruction and the environment perception information, and then a whole action sequence corresponding to the user instruction is generated through the large language model and the first prompt word. Finally, a second prompt word is determined, and a detailed action plan corresponding to the current sub-action is generated through the large language model and the second prompt word, so as to obtain a detailed action plan corresponding to each sub-action in the whole action sequence.

[0016] Through the above technical solution, the user's intention can be understood from the received natural language or unstructured instruction, and an executable whole action sequence and a detailed action plan of each sub-action in the sequence can be generated, so that the task issued by the user can be completed according to the whole action sequence and the detailed action plan, and finally the robot can execute the task, improving the accuracy and efficiency of executing the task. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0018] Figure 1 A flowchart of a task planning method of a robot according to an embodiment of the present application; Figure 2 A flowchart of generating an action plan and executing according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] The application will be further described with reference to the drawings and embodiments, and the described embodiments are only part of the embodiments of the application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art shall fall within the scope of protection of the embodiments of the application.

[0020] Referring to Figure 1 Embodiments of a task planning method of a robot are provided, which are suitable for a server, and the server comprises: S1: obtaining a user instruction.

[0021] The user instruction is used to instruct the robot body to perform a task of a non-fixed process. The user instruction can be obtained by the robot body converting a voice sent by a user into natural language text and then encapsulating the natural language text into the user instruction and sending the user instruction to the server. The server can be a micro server directly physically connected in the robot body, or a server in the cloud, and the robot body communicates with the server in the cloud through communication connection, which is not specifically limited in the application.

[0022] In the specific implementation of the application, when a user issues an operation instruction to the robot body, the robot body receives the operation instruction and converts it into a user instruction that can be processed by a computer and sends it to the server, so that the server can obtain the user instruction. The user can issue the operation instruction to the robot in various ways, such as inputting text, voice, interface interaction, etc. to the robot body. The operation instruction is used to instruct the robot to complete a specific task, and its core elements include but are not limited to execution action, operation object, position information, etc.

[0023] In an embodiment of the application, the content appearing in the instruction issuing pipeline is monitored, for example, the user instruction is received through OCR text recognition, Chinese text input, and voice conversion text, etc. and finally converted into a text format instruction; after the user instruction is issued, the user instructions are put into the instruction processing pipeline according to the order of receiving the user instructions.

[0024] S2: retrieving environment perception information related to the user instruction in the memory bank based on a large language model and a semantic encoding model.

[0025] The large language model is used to analyze a user instruction to obtain a candidate object related to the user instruction, and the semantic encoding model is used to encode the candidate object into semantic features in the same format as the memory library. The memory library is used to store environment perception information corresponding to each object. The environment perception information in the memory library is constantly updated as the robot body moves and time elapses. The environment perception information is structured data obtained by the robot body continuously scanning the environment during patrol, and information of each object in the environment.

[0026] In one example, the environment perception information can include image information and position information. The image information at least includes an RGB image, a depth image, an object category, an object detection box, and an object corresponding segmentation mask, etc. The image information can be obtained by shooting with devices such as binocular cameras and laser radars, and processed into corresponding RGB images and depth images. Through an image recognition model and an image segmentation model, the types of objects in the image can be recognized, and for each object category, the detection box of the object and the segmentation mask of the object can be identified, and the above data can be packaged and stored as environment perception information. The position information can first obtain the positioning information of the robot itself, and take this information as the reference, combine the depth image containing the object, and determine the positional relationship between each object in the depth image and the positioning information of the robot itself through an algorithm, to determine the actual position of each object in the image.

[0027] Specifically, the large language model is an artificial intelligence model constructed based on deep learning technology. When a user inputs a natural language instruction, the core task of the large language model is not simply keyword matching, but deep understanding of the deep intention and context meaning of the instruction. The large language model understands the semantics of the user instruction. After being analyzed by the large language model, the system obtains a “candidate object” description based on text. However, the environment perception information (such as visual features of objects, position point clouds, etc.) stored in the memory library is non-textual multi-modal data. In order to perform effective matching retrieval, it is necessary to map the text description and the memory library information into the same semantic feature space.

[0028] The embodiment of the present application adopts a pre-trained semantic encoding model (such as CLIP, BERT, or a multi-modal model). The model encodes the candidate object description in the form of text into a high-dimensional, fixed-length semantic feature vector. The semantic feature vector of the candidate object is calculated for similarity with all semantic feature vectors in the memory bank, for example, by calculating the similarity through cosine similarity or Euclidean distance. Each entry in the memory bank is associated with past perceived environmental perception information (for example, RGB images of objects, depth information, 3D coordinates, etc.). According to the similarity score, the highest score or several environmental perception information are retrieved, which are the most relevant environmental perception information to the user instruction. For example, the server will find a complete environmental perception information of a book "on the living room table, blue cover, recorded yesterday" in the memory through the system.

[0029] S3: inputting the first prompt word determined based on the user instruction and the environmental perception information into the large language model to generate an overall action sequence corresponding to the user instruction.

[0030] The first prompt word at least contains a plurality of sub-actions supported by the robot. The first prompt word is used to let the large language model design an overall action sequence for the robot body according to the user instruction, the environmental perception information corresponding to the user instruction, and the plurality of sub-actions supported by the robot, so that the robot body can complete the user instruction according to the overall action sequence.

[0031] In the embodiment of the present application, a prompt word template is preset, the user instruction and the environmental perception information are filled into the prompt word template according to a certain format, and a plurality of sub-actions supported by the robot body are filled into the prompt word template, so as to construct the first prompt word; the first prompt word is input into the large language model, the large language model decomposes the high-level task corresponding to the user instruction into a series of sequential or parallel sub-tasks, and designs a corresponding execution action for each sub-task to generate an overall action sequence. Therefore, the output of the large language model is a structured overall action sequence, which can be a JSON array, a step list, or any format recognized by a robot control system.

[0032] S4: determining a second prompt word for each sub-action in the overall action sequence, inputting the second prompt word into the large language model to generate a detailed action plan corresponding to the current sub-action, until all sub-actions in the overall action sequence are traversed.

[0033] The detailed action plan is composed of a plurality of basic actions supporting the sub-action. The basic action is the most basic and simplest action that can be executed by each actuator in the robot body.

[0034] The overall action sequence generated by the foregoing steps (such as [navigation, moving the robot arm, grabbing]) is a high-level action plan. In order to ensure that the robot can safely and accurately perform each sub-action, for each sub-action in the overall action sequence, a large language model is used to perform more fine-grained motion planning, converting high-level instructions into low-level control instructions that can directly drive the actuators.

[0035] Taking one specific embodiment as an example: Overall action sequence: ["navigate to point A", "pick up object B"]. Processing the first sub-action "navigate to point A": Second prompt word: "... current state: the robot is located at the origin. Next action: navigate to point A (coordinates x: 10, y: 5). Please generate navigation steps." Detailed plan: ["plan global path", "move along path"].

[0036] Processing the second sub-action "pick up object B": Second prompt word: "... current state: the robot has arrived at point A, and the camera detects that object B is 30 cm to the left. Next action: pick up object B." Detailed plan: ["turn left 30 degrees", "move robot arm to object coordinates", "grab"].

[0037] The present application can receive natural language or unstructured instructions, understand user intent, and generate executable overall action sequences. At the same time, the method also has the ability of action monitoring and dynamic adjustment, ensuring that the robot can respond to environmental changes or unexpected situations during execution. For example, when executing the task of "get a cup of water", the method plans the movement path, identifies the water cup position, adjusts the robot arm grabbing posture, and re-adjusts the action sequence when detecting that the water cup has been moved, thereby ensuring the successful completion of the task.

[0038] For the steps of S2, the specific implementation process of the method includes: S201: Obtain a third prompt word, input the third prompt word into a large language model, and obtain object names related to the user instruction; the third prompt word is used to control the large language model to analyze the user instruction; Wherein, the third prompt word is a preset text template, usually constructed in a role-playing and instruction-specific manner, through which the large language model can be used as a natural language processing expert, then it can perform word segmentation and entity extraction on the input text, obtain multiple entity objects, and analyze the multiple entity objects to determine the object names of the candidate objects related to the user instruction. The object name can be the name of an object in the scene where the robot performs the task, or it can be the name of the scene directly.

[0039] Specifically, the user instruction is: "Pass over the red cup in the kitchen." The third prompt word constructed is: "You are a natural language processing expert. Please perform word segmentation and entity extraction on the user instruction, and determine the object names of all objects related to the user instruction." The large language model output is: "Scene name: kitchen; object name: red cup; object name: user."

[0040] S202: Determine the semantic features of the object name based on a semantic encoding model; The semantic encoding model encodes the text semantics of the object name into a high-dimensional vector (i.e., semantic features), which is in the same semantic space as the visual feature vector encoded by the same model in the memory bank, making it possible to calculate the semantic similarity between text and image.

[0041] Specifically, after parsing the object name in text form (such as "red cup") from the user instruction, in order to be able to perform semantic-based similarity matching in the memory bank, it is necessary to convert the name from discrete symbols (characters) to continuous, semantic-containing numerical representations. This step is completed by the semantic encoding model to convert text to vector, laying the foundation for subsequent cross-modal retrieval.

[0042] S203: Retrieve at least one item of environment perception information similar to the semantic features in the memory bank based on the semantic features; the semantic encoding model is a multi-modal semantic encoding model for encoding text and images expressing the same meaning into the same semantic features; The environment perception information at least includes RGB images, depth maps, object categories, detection boxes, and segmentation masks corresponding to each object name.

[0043] Specifically, the semantic features are compared with the environment perception information in the memory bank for similarity, and the top N largest similarities are selected from all the similarity values obtained, and the environment perception information corresponding to the top N largest similarities is selected as the environment perception information similar to the semantic features.

[0044] S204: Perform content matching on the at least one item of environment perception information and the user instruction to obtain the first de-duplicated environment perception information; the content matching is used to determine the environment perception information matched by the object name related to the user instruction in the at least one item of environment perception information; The matching can be achieved by similarity or distance. The core purpose of this content matching and de-duplication step is to filter out the information that is most directly related to the user instruction intent and has the highest uniqueness from the candidate information, and to eliminate redundant data, providing refined and non-redundant input for subsequent action planning.

[0045] Specifically, after a set of candidate environmental perception information is obtained from the memory bank through semantic retrieval, there may be multiple sets of environmental perception information that are completely irrelevant, but their corresponding semantic features may be similar. At this time, content matching needs to be performed on the object names corresponding to the environmental perception information to match the multiple environmental perception information corresponding to the similar semantic features with the user instruction, and determine the environmental perception information corresponding to the object name that truly meets the user instruction. For example, there are semantic feature 1 and semantic feature 2, and the object names corresponding to the two are respectively: cup; cat. According to the user instruction in the above example, it is obvious that the cat is completely irrelevant to the user instruction, and based on this, the environmental perception information related to the cat needs to be deleted.

[0046] S205: performing target matching on the environmental perception information after the first deduplication based on a fourth prompt word to obtain environmental perception information after the second deduplication, and determining the environmental perception information after the second deduplication as the environmental perception information related to the user instruction; wherein the fourth prompt word is used to control the large language model to determine the matched environmental perception information in multiple environmental perception information representing the same object name.

[0047] Wherein, the target matching refers to matching the target object in the prompt word with the object in the environmental perception information to select the target object that is more consistent with the user instruction.

[0048] Specifically, after the environmental perception information representing different entities is deduplicated based on the simple rules for the first time, there may still be multiple candidate information representing the same entity, which represent different physical entities but all conform to the basic object type and attribute (for example, there are “multiple blue cups” on the table). In order to accurately identify the one that the user really means from these highly similar candidate objects in terms of semantics and vision, the powerful context reasoning ability of the large language model is used to simulate the process of human judgment based on common sense and dialogue context, and the final target locking is realized.

[0049] The present application can efficiently and accurately determine the matched environmental perception information from multiple environmental perception information representing the same object name based on the prompt word and the large language model.

[0050] For the steps of S3, the specific implementation process of the method includes: S301: obtaining a list of sub-actions that the robot can support; Wherein, the sub-action list is used to store multiple sub-actions.

[0051] S302: filling the sub-action list and the environmental perception information related to the user instruction into a pre-configured first prompt word template to obtain a first prompt word; The replacement symbol in the first prompt word template includes a sub-action list, environment perception information, user instructions, and data formats of the overall action sequence, each corresponding to a replacement symbol; each replacement symbol is used to generate a first prompt word matching the user instructions.

[0052] Specifically, in order to guide the large language model to generate a high-quality action sequence that conforms to the robot's capability constraints and fits the actual environment, this step integrates the robot's internal capabilities (sub-action list) and external environment state (environment perception information) through a first prompt word template. The template structurally fills all the context information required for planning, forming a final first prompt word submitted to the large language model.

[0053] In one example, the first prompt word can be an action sequence prompt word, and the structure of the prompt word can be: "You are an action generation expert, the actions that the current robot can perform include'sub-action list', the current environment information includes 'environment perception information', the user wants the robot to complete the instruction 'user instructions', please generate an action sequence that can complete the user instruction based on executable actions, environment information as the actual control range, the format of the action sequence is 'action sequence', and the final output is the overall action plan."

[0054] S303: input the first prompt word into the large language model to obtain an overall action sequence corresponding to the user instruction.

[0055] Specifically, the large language model decomposes the user instruction (such as "pick up an apple") into multiple sub-tasks according to the first prompt word. For each sub-task, one or more specific atomic actions from the sub-action list are matched; the order between actions is determined.

[0056] The present application can accurately obtain an overall action sequence corresponding to the user instruction according to the prompt word and the large language model.

[0057] For the steps of S4, the specific implementation process of the method includes: S401: traverse each sub-action in the overall action sequence in chronological order, and obtain the current sub-action, the robot's actuator, and the multiple basic actions that the actuator can execute; The multiple basic actions that the actuator can execute refer to the most basic action instructions that are directly supported by the robot's bottom hardware (such as motors, servos, hydraulic cylinders, etc.) and driving software, cannot be further divided, or are preset by the system. These basic actions constitute the capability boundary of all complex tasks of the robot, and are the atomic operation units for the action sequence generated by the task planning layer to be finally converted into physical motion.

[0058] ​​​​S402: Fill the current sub-action, the executor, and the multiple basic actions that the executor can perform into the pre-configured second prompt word template to obtain a second prompt word; The second prompt word is composed of the current sub-action, the executor, and the multiple basic actions that the executor can perform.

[0059] Specifically, the second prompt word, i.e., the action generation prompt word, can be: "You are an action generation expert, and the current need is to generate the detailed action plan of the current sub-action in the overall action sequence in detail. The hardware device (executor) of the robot that can perform the sub-action ", has the basic action of: Please generate the operation information of the corresponding sub-action based on the basic action of each hardware device, and determine the positioning information changed before and after the execution of each basic action." S403: Input the second prompt word into the large language model to obtain a detailed action plan corresponding to the current sub-action.

[0060] Specifically, the sub-action in the overall action sequence indicates the target, but does not specify the specific motion path, posture, and obstacle avoidance strategy. Direct execution of such an action may cause collision or failure. The purpose of this step is to solve "how to safely and efficiently complete this sub-action". The second prompt word provides the specific context required to complete the sub-action (such as real-time target coordinates, obstacle information), guiding the large language model to conduct microscopic motion planning.

[0061] The present application can obtain a detailed action plan corresponding to each sub-action according to each sub-action in the overall action sequence to guide the robot to flexibly complete the user instruction.

[0062] For the steps of S3, S3 further includes: The first verifier checks the overall action sequence corresponding to the user instruction, and if the checking fails, inputs the overall action sequence and the corresponding failure result into the large language model to regenerate the overall action sequence corresponding to the user instruction; wherein the first verifier is determined based on the pre-configured verification rule and the feasibility of the overall action sequence analyzed by the large language model.

[0063] Specifically, the first verifier is used to check whether the action sequence in the overall action sequence conforms to the basic logic of the physical world. For example, the grabbing action must be after moving to the position, and the releasing action must be after grabbing.

[0064] For the steps of S4, S4 further includes: The detailed action plans generated by the large language model are verified by the second verifier. If the verification fails, the detailed action plan and the corresponding failure result are input to the large language model to regenerate each detailed action plan corresponding to the overall action sequence until the verification succeeds.

[0065] Specifically, each time the large language model generates a detailed action plan corresponding to a sub-action, the plan is verified by the second verifier. The second verifier has a different division of labor from the first verifier, and its core task is to evaluate the safety, accuracy, and executability of each detailed action from a microscopic physical level, ensuring that each movement of the robot is safe, collision-free, and consistent with physical constraints.

[0066] The present application verifies the detailed action plans generated by the large language model multiple times through the verifier, thereby improving the correctness of the actions generated by the large language model.

[0067] In an exemplary embodiment, before S1, the method further comprises: Receiving object information in each environment captured by the information acquisition device of the robot; the object information includes object names, image information corresponding to the object names, and position information; the objects include environmental objects, large object, and small object; processing the image information and the position information in the object information based on a pre-configured data structure to obtain environmental perception information corresponding to each object name; and uploading the environmental perception information to the memory bank.

[0068] Specifically, the image information and position information of each object in the scene where the robot performs the task are obtained and updated through the information acquisition device such as the camera and laser radar of the robot, and the specific implementation process is as follows: A standard format for synchronizing environmental perception information is defined, which includes image information and position information. The image information at least includes RGB image, depth image, object category, object detection box, and object corresponding segmentation mask, etc. The image information can be obtained by binocular camera, laser radar and other devices, and processed into corresponding RGB image and depth image. Through image recognition model and image segmentation model, the types of objects in the image can be identified, and for each object category, the detection box of the object and the segmentation mask of the object are identified, and the above data is packaged and stored as environmental perception information. The positioning information of the robot itself can be obtained first, and the position relationship between each object in the depth image and the positioning information of the robot itself is determined through algorithm based on the information, so as to determine the actual position of each object in the image.

[0069] The information obtained from each frame of image is extracted according to a defined format by the camera of the robot and image processing, to obtain environment perception information; each group of images is sent to the server of the robot once after being processed; after receiving the environment perception information sent each time, the server puts the environment perception information into a cache queue and waits for processing.

[0070] In an embodiment of the present application, the specific implementation process of the step "uploading the environment perception information to the memory library" includes: The multiple environment perception information in the cache queue is processed in time sequence to obtain de-duplicated environment perception information; the de-duplication processing at least includes environment object de-duplication processing and object object de-duplication processing; according to the detection frame in the environment perception information, the image information corresponding to each object is intercepted, and the image information is input into a semantic coding model to code to obtain the semantic features corresponding to the object; the semantic features and the environment perception information associated with the semantic features are uploaded to the memory library according to a preset format; the memory library is used to recall environment perception information with high similarity from the memory library through similar semantic features.

[0071] Specifically, the environment perception information is obtained from the cache queue in sequence; the environment perception information is de-duplicated twice, once for environment de-duplication and once for object de-duplication. First, each piece of environment perception information in the cache queue is input into a scene de-duplication module for processing, to delete the environment perception information under repeated environments and retain high-quality environment perception information. Second, according to the data with detection and segmentation in the environment perception information (including RGB image, depth image, object category, detection frame, segmentation mask, etc.), the point cloud volume corresponding to each object is calculated. According to the point cloud volume and the segmentation mask area corresponding to the object, the object is divided into a large object and a small object, to determine whether the large object is repeated with the object in the constructed memory library and whether the small object is repeated with the object in the constructed memory library; if repeated, the large object or the small object is not added to the memory library; if not repeated, the large object or the small object is considered as a new object, and the new object is stored in the memory library.

[0072] According to the detection frame in the de-redundant environment perception information, the image of the corresponding object is intercepted and input into a trained multi-modal feature coding model to code to obtain the semantic features corresponding to the object; the semantic features and the environment perception information associated with the semantic features are uploaded to the memory library according to a preset format. In addition, the semantic features can also be used as an index to retrieve the environment perception information containing the object. That is, as long as the user has the semantic features, the environment perception information corresponding to the object can be retrieved in the memory library.

[0073] The present application improves the efficiency and accuracy of information recognition and matching through multiple de-duplication processing of environment perception information.

[0074] In an exemplary embodiment, after S4, the method further comprises: For each sub-action corresponding to a detailed action plan, obtaining current environment perception information before execution of the current sub-action; issuing an execution command corresponding to the detailed action plan of the current sub-action to each actuator of the robot; the actuator is used to instruct the hardware device of the robot to perform a related control operation; the control operation at least includes navigation, grasping, releasing, opening and closing; after the actuator receives the corresponding execution command and executes, obtaining the latest environment perception information after executing the current sub-action; inputting the current environment perception information before execution of the current sub-action and the latest environment perception information after execution into the large language model to obtain a sub-action verification result, until all sub-action detailed action plans of the overall action sequence are verified.

[0075] Specifically, save the environment perception information corresponding to the latest received image frame, issue the execution command corresponding to the single action plan to the corresponding actuator; the actuator executes the corresponding task according to the execution command received by it, parses the execution command and executes the specific action plan; obtain the environment perception information after action execution; save the environment perception information corresponding to the latest received image frame; verify whether the detailed planned action plan is executed successfully.

[0076] The present application verifies the sub-action through the large language model, which can discover possible problems in the sub-action execution process as soon as possible and correct them in time, thereby improving the correctness of the sub-action execution.

[0077] In an embodiment of the present application, the specific implementation process of the step "inputting the current environment perception information before execution of the current sub-action and the latest environment perception information after execution into the large language model to obtain a sub-action verification result" comprises: Based on the current environment perception information before execution of the current sub-action and the latest environment perception information after execution, generate a sub-action verification prompt word; input the sub-action verification prompt word into the large language model, so that the large language model determines whether the first coordinate value of the target object in the current environment perception information and the second coordinate value of the target object in the latest environment perception information match the expected result of the current sub-action, if the first coordinate value and the second coordinate value match the expected result of the current sub-action, it is determined that the sub-action verification result is verified.

[0078] Specifically, determine the sub-action verification prompt word to tell the large model how to complete the action verification task according to the current instruction to be completed and the two environment perception information before and after action execution; organize the current instruction and the environment perception information before and after action execution into input data according to the format defined in the prompt word; input the input data into the large language model to obtain the verification result.

[0079] Exemplarily, the action of obtaining obtains environmental perception information related to the object in the user instruction before the action is executed, such as coordinate 1 of object A; the action of obtaining obtains environmental perception information related to the object in the user instruction after the action is executed, such as coordinate 2 of object A; and it is determined whether the environmental perception information of the two frames before and after the action is executed matches the coordinate information of the object in the detailed action plan. If the two are matched, it is determined that the detailed action plan is completed. For example, it is determined whether coordinate 2 of object A is close to the position where object A is intended to move in the detailed action plan. If the two are close, it is determined that the two are matched.

[0080] The embodiment of the application further provides a task planning system of a robot, which implements the task planning method of the robot in the above embodiment, and comprises the following modules: a user instruction obtaining module, configured to obtain a user instruction; the user instruction is used to instruct the robot to execute a task of a non-fixed process; an environmental perception information retrieving module, configured to retrieve environmental perception information related to the user instruction from a memory bank based on a large language model and a semantic encoding model; the large language model is used to analyze the user instruction to obtain a candidate object related to the user instruction, and the semantic encoding model is used to encode the candidate object into semantic features in the same format as the memory bank; a whole action sequence generating module, configured to input a first prompt word determined based on the user instruction and the environmental perception information into the large language model to generate a whole action sequence corresponding to the user instruction; the first prompt word at least further comprises a plurality of sub-actions supported by the robot; a detailed action plan generating module, configured to determine a second prompt word for each sub-action in the whole action sequence, input the second prompt word into the large language model to generate a detailed action plan corresponding to the current sub-action, and iterate through each sub-action in the whole action sequence; the detailed action plan is composed of a plurality of basic actions supporting the sub-action.

[0081] Those skilled in the art should clearly understand that, for the convenience and brevity of description, the specific working process of the task planning system of the robot described in the above embodiment can refer to the corresponding process in the foregoing task planning method of the robot, which will not be described here.

[0082] For the convenience of understanding, the application provides the following more specific embodiments: As shown in Figure 2 the application scenario of the application is a home service robot, and the overall concept is as follows: 1. Preparation: constructing a memory bank for a home scene to obtain feature data and position data of each object in the scene, so that the robot can remember the objects and the positions of the objects in the scene; 2. After receiving the user instruction, recalling information related to the user instruction from the memory bank based on a large language model; 3. Through the large language model, the robot thinks, and according to the user's instructions and related information, the overall action sequence that can complete the user's instructions is generated, as well as the action execution plan of each action; 4. The execution plan verifies whether the action is completed through the large language model, and if the action is completed, the next action is traversed, and if the action is not completed, the action execution plan is adjusted in time, the action is re-executed, and the above steps are repeated until all actions in the overall action sequence are completed.

[0083] Step 1: Receive environmental perception information of the home scene.

[0084] Function: Through the camera, laser radar and other information acquisition devices of the robot, the image information and position information of each object in the home are obtained and updated.

[0085] Implementation: 1. Define a standard format for synchronizing environmental perception information, which contains image information and position information. The image information at least includes RGB image, depth image, object category, object detection box and object corresponding segmentation mask, etc.

[0086] Image information can be obtained by binocular camera, laser radar and other devices, and processed into corresponding RGB image and depth image. Through image recognition model and image segmentation model, the types of objects in the image can be recognized, and for each object category, the detection box of the object and the segmentation mask of the object can be identified, and the above data can be packaged and stored as environmental perception information.

[0087] The position information can first obtain the positioning information of the robot itself, and take this information as the reference, combined with the depth image containing the object, determine the position relationship between each object in the depth image and the positioning information of the robot itself through algorithm, and determine the actual position of each object in the image.

[0088] Data structure example: Each image processed will be organized into the following data format, indicating all the perception information corresponding to the current image.

[0089] { "agent_room": "bobs_room", / / Custom scene, can be determined according to the coordinate range of the electronic fence "image_RGB_url": "url_RGB.jpg", / / RGB image "image_depth_url": "url_depth.png", / / Depth image "world2camera": [0.866, -0.5, 0, 2, 0.5, 0.866, 0, 3, 0, 0, 1, 5, 0,0, 0, 1], / / The pose information corresponding to the camera mounted on the robot body, 16 / 4 A 4-dimensional matrix vector, pose information includes position and orientation information, such as coordinates and angles.

[0090] "camera_intrinsics":[606.0801391601562,0.0, 318.43902587890625,0.0,606.3120727539062, 246.55946350097656, 0.0, 0.0, 1.0], / / Camera intrinsics, hardware parameters, generally not changed; "instance": [ / / example] { "object_id": "12", / / ID "name": "coffee_mug1", "room": "bobs_room", / / Environment "position":{"x":34.53,"y":23.82, "z":10.31}, / / Item position information "detection_2d":{"left_top":[45, 560],"right_bottom": [78, 780]}, / / The range of the detection box for the item in the RGB image. "segmentation": [[23,345], [234, 345]], / / The segmentation mask corresponding to this item; }, { "object_id": "15", / / The object's ID is the string "15" "name": "coffee_mug2", / / The object is a coffee cup named "coffee_mug2". "room": "bobs_room", / / This object is located in a room named "bobs_room" "position":{"x":34.53,"y":23.82,"z":10.31}, / / The precise position of the object in the room, x=34.53, y=23.82, z=10.31 "detection_2d": {"left_top": [45, 60], "right_bottom": [78, 780]}, / / Object's bounding box in the image, left-top pixel coordinate [45, 60], right-bottom pixel coordinate [78, 780] "segmentation": [[27, 145], [24, 325]], / / Object's contour polygon vertex coordinates in the image, e.g. point 1 [27, 145], point 2 [24, 325] 2. Through the camera of the robot, the image processing side, the information obtained by each frame is perceived according to the defined format, and the environment perception information is obtained; each group of images is processed and the environment perception information is sent once.

[0091] 3. After the robot's server receives the environment perception information sent each time, the data is put into the buffer queue for processing.

[0092] Step 2: Build a memory bank.

[0093] Function: The server performs de-duplication processing on the received environment perception information to build a perception memory bank.

[0094] Implementation: 1. Obtain the environment perception information from the cache queue in turn; 2. Perform two de-duplication processes on the environment perception information, one for scene de-duplication and one for object de-duplication. First, input each piece of environment perception information in the cache queue into the scene de-duplication module for processing, delete the environment perception information under the repeated scene, and keep the high-quality environment perception information. Secondly, according to the data with detection and segmentation in the environment perception information (including RGB image, depth image, object category, detection box, segmentation mask, etc.), the point cloud volume corresponding to each object is calculated.

[0095] When the point cloud volume exceeds the set value, the data is sent to the large object de-duplication module for processing, and when the point cloud volume is less than the set value, the data is sent to the small object de-duplication module for processing. The large object de-duplication module and the small object de-duplication module are based on two de-duplication logics and perform de-duplication operations on environment perception information containing repeated objects.

[0096] In one example, the logic of small object de-duplication is: 1. Each new object added to the memory bank will first be de-duplicated according to the object's ID number. Each object has a unique ID number in the environment perception data (information), if the ID number is the same, it is directly considered that the object is the same; ​​2. If the ID of the newly added object is not the same as each number in the memory library, use the multi-modal encoding model to extract the image feature information of the object, and then use the feature information to search the vector library for possible candidate matching objects with a similarity higher than a set threshold; 3. If there is no candidate object, consider the object to be a new object that is not recorded in the memory library, and directly add the object to the memory library as a new added object; 4. If there is a candidate object, calculate the distance from all candidate objects, then sort the distance values, and select the object with the smallest distance value as a possible duplicate object. If the minimum distance value is less than a set threshold, consider the newly added object to be the same object as the object with the minimum distance. If the minimum distance value is greater than the set threshold, consider the newly added object to be an object that is not in the memory library.

[0097] The logic for large object deduplication is as follows: 1. For each newly added large object, extract the detection box, segmentation mask, and image pixel coordinate to world coordinate conversion relationship of each object in the environmental perception data; 2. According to the detection box, the image of the corresponding object is intercepted and sent to the multi-modal encoding model for encoding to obtain the corresponding semantic vector information; 3. Using the segmentation mask of the object and the conversion relationship of the image pixel coordinates to the world coordinates, the point cloud of the corresponding object in the world coordinate system is calculated; 4. Using the semantic features of the object, calculate the similarity with the existing large objects in the memory library. If the similarity is greater than a set threshold, it is considered to be a candidate. Then calculate the intersection over union of the point cloud, and select the large object with the maximum intersection over union and greater than a set threshold as the matching large object with the newly added large object. If there is no match, consider the newly added large object to be a new large object; 5. For the matched large object (each large object will have multiple images from different perspectives), calculate the intersection over union of the point cloud between the new large object and all images inside the matched large object. Replace all images with the new large object image whose intersection over union is greater than a set threshold, and update to obtain the new image information that can represent the large object.

[0098] 3. Each sub-module processes the corresponding data to remove redundant content and low information content data.

[0099] 4. For the de-duplicated data, according to the detection box in the environmental perception information, the image of the corresponding object is intercepted and sent to the trained multi-modal feature encoding model to encode the semantic features of the object; 5. Using this semantic feature as an index, a semantic understanding vector library is constructed for subsequent queries. Furthermore, using this semantic feature as an index, environmental awareness information containing the item can also be retrieved. That is, as long as a user possesses the semantic feature, they can retrieve the environmental awareness information corresponding to the item from the vector library.

[0100] In one example, semantic features - RGB image - depth image - object category - detection box - segmentation mask - location information are stored as a data structure in the vector library as an item; for example, the stored structure is the same as the perceptual data structure above. The data in the vector library is all in string form. The information currently stored is in the format of perceptual information, which is a JSON string. The use of JSON format is mainly to reflect the hierarchical relationship structure of the data.

[0101] Step 3: Receive user instructions.

[0102] Function: Listen for user-sent instructions, such as receiving voice messages and converting them into natural language.

[0103] Implementation method: 1. Listen to the content appearing in the command delivery pipeline, such as receiving user commands through OCR text recognition, Chinese text input, and speech-to-text conversion, and finally converting them into text format commands; 2. After a user command is received, the commands are placed into the command processing pipeline according to the order in which they were received.

[0104] Step 4: Retrieve relevant information from the memory bank.

[0105] Function: Understand user commands and locate environmental awareness information highly relevant to the user commands in the memory bank; Implementation method: 1. By using pre-configured entity extraction prompts and parsing user commands using a large language model, candidate object names that are highly relevant to user commands are identified. 2. Encode the candidate object names using a multimodal semantic coding model to obtain the semantic features of the object. The multimodal semantic coding model can encode text and image data, encoding text and image data expressing the same meaning into similar semantic features. Therefore, users can find images that match the text through semantic features, i.e., retrieve the corresponding image through text, or retrieve the corresponding text through an image. 3. Use the encoded semantic features to retrieve multiple relevant environmental perception information from the memory in turn, and perform content matching and deduplication on the multiple environmental perception information to obtain the most suitable environmental perception information. 4. Use the large language model to make a second judgment on the matched information, remove irrelevant information, and ensure that the remaining information is highly relevant to the user's instructions.

[0106] For example, the recall content of the vector library has no logical judgment and only recalls related items. For example, if you say "help me get a bottle of water on the kitchen table," the vector library may recall several bottles of water in different locations. The multimodal large model needs to make a second judgment on whether this bottle of water is on the kitchen table, thereby removing irrelevant environmental perception information.

[0107] Specifically, the content of the entity extraction prompt word can be "You are a natural language processing expert, and you need to identify the actionable entities contained in the above sentence and give the entity a name. The name given can be the name in the sentence or a more general alias after refining the name in the sentence." After inputting the entity extraction prompt word into the large language model, multiple candidate object names can be obtained.

[0108] Optionally, the prompt word of the judgment process in step 4 can be "Please determine the environmental perception information related to the entity and the environmental perception information unrelated to the entity based on the environmental perception information of the following objects and the entities in the 'user command', and output the environmental perception information unrelated to the entity." The server can delete the environmental perception information unrelated to the entity based on the output of the large language model.

[0109] Step five: generate the overall action sequence.

[0110] Function: Based on the environmental perception information recalled from the memory library, generate the overall action sequence.

[0111] Implementation: 1. Write the action sequence prompt word, which clearly informs the large language model of the sub-actions currently supported by the robot and the command-related environmental perception information that needs to be analyzed by the large language model and executed to generate the overall action sequence. At this time, the large language model can think about how to generate the overall plan and how to generate the overall task plan based on the memory information and user instructions (the output result can be configured: the task plan is a list that contains multiple robot-executable sub-actions such as grasping, releasing, opening, closing, and navigation in order). 2. Organize the user instructions and the perception information recalled from the memory into the format defined in the prompt word and send it to the large model to generate the corresponding overall action plan. 3. Use the verifier to check the generated overall plan, and if problems are found, feed them back to the large model to regenerate the plan. If there are no problems, return the generated overall plan.

[0112] For example, the verifier can be verified by a large language model, or some existing verification methods can be configured to verify. Therefore, the verification method can be based on a large model or program logic.

[0113] Specifically, the structure of the action sequence prompt word is: "You are an action generation expert, the actions that the current robot can perform include ' ', the current environment information includes ' ', the user wants the robot to complete the instruction ' ', please base on the executable action, the actual control range of the environment information, generate an action sequence that can complete the user's instruction, the format of the action sequence is ' ', and the final output is the overall action plan."

[0114] Step six: generate a single action detailed plan.

[0115] Function: refine each sub-action of the global plan execution according to the on-site situation, and generate a task plan that can be actually executed.

[0116] Implementation: 1. Take out a sub-action (for example: navigate to the destination point, grab the target object, release the object to the target position) in the overall action sequence; 2. Write an action generation prompt word, which tells the large model that the task to be completed is to generate a detailed action plan, and each sub-action detailed action plan mainly includes multiple basic actions, and each basic action can include operation information and positioning information. Positioning can be divided into two forms, namely frame and point. If it is a robot navigation operation and a mechanical arm release operation, the detailed action plan is to draw a point on the image to represent the position to be approached or the position of the object to be released, and if it is a mechanical arm to grab, open, and close operation, the detailed action plan is to draw a frame, which mainly frames the target object to be operated; 3. Verify the single action plan generated by the model, and if the verification fails, feed back the problem to the model to re-generate the plan until the verification passes and return the detailed action plan after the verification passes.

[0117] Specifically, the action generation prompt word can be: "You are an action generation expert, and you need to generate a detailed action plan for the sub-action in the overall action plan , the hardware devices of the executable sub-action robot include: ' ', the basic actions of which include: ' ', please generate the operation information of the corresponding sub-action based on the basic actions of each hardware device, and determine the positioning information changed before and after the execution of each basic action." Step seven: obtain the environment perception information before action execution.

[0118] Function: Get the perception information before the action execution, which is used to verify whether the action is executed successfully.

[0119] Implementation: 1. Save the latest received frame of environment perception information.

[0120] Step eight: Execute the action.

[0121] Function: Specifically execute the action plan outlined in the detailed plan.

[0122] Implementation: 1. Issue the execution command corresponding to the single action plan to the corresponding executor (existing executors: navigation, grabbing, releasing, opening, closing); 2. After the executor claims the task, parse the execution command and execute the specific action plan; In one example, the specific data format is defined as follows, and the formats of different actions have some differences: { "cmd_type": "pickup | open | close | turn_on | turn_off", / / Current operation (grabbing / opening / closing / turning on / turning off the device) "obj_type": "apple", / / The operation object is "apple" "image_RGB_url": "url_RGB.jpg", / / URL of the color image of the object "image_depth_url": "url_depth.png", / / Image URL of the object depth information "detection_2d": {"left_top":[45, 560],"right_bottom": [78, 780]}, / / The position of the object in the image, left top coordinate [45,560], right bottom coordinate [78,780] } { "cmd_type": "release", / / Release the object "obj_type": "apple", / / The object to be released is "apple" "image_RGB_url": "url_RGB.jpg", / / Scene color image when releasing "image_depth_url": "url_depth.png", / / Scene depth information when releasing "point_2d": [45, 560], / / Target position of the release operation (image pixel coordinates [x=45, y=560]) } { "cmd_type": "access", / / Access or locate an object "obj_type": "apple", / / The object being located is an "apple". "position": {"x":34.53, "y":23.82, "z":10.31}, / / Position information of the large object: x=34.53, y=23.82, z=10.31 } Step 9: Obtain environmental awareness information after the action is executed; Function: Acquire sensory information after an action is performed, and then use it to verify whether the action was successfully executed. Implementation method: 1. Save the latest received frame of environmental awareness information.

[0123] Step 10: Verify the execution results.

[0124] Function: Verify whether the detailed action plan has been executed successfully.

[0125] Implementation method: 1. Write verification prompts to inform the large model how to complete the action verification task based on the instruction to be performed and the two environmental perception information before and after the action execution; 2. Organize the current instruction and the environmental awareness information before and after execution into an input format according to the format defined in the prompt words; 3. Feed the input data to the large model, obtain and return the validation results.

[0126] Specifically, such as Figure 1 The process shown involves two sets of loops: one for generating the overall sequence of actions and the other for generating detailed plans for individual actions.

[0127] In the previous iteration that generates the overall action sequence, the large language model can determine whether the robot has completed the user's command based on the user's command content and the current environmental perception information by verifying the cue words. An example of the specific verification logic is as follows: 1. Determine the predicted perception information of each object after the user command is completed; 2. Compare the actual perceived information in the memory bank with the predicted perceived information to determine whether they are consistent; 3. If the information matches, then the robot has successfully completed the user's command.

[0128] In the latter iteration of generating a single action detailed plan, the large language model can be prompted to verify the keywords to determine whether each action detailed plan is successfully completed according to the image data of the two frames before and after the execution of the detailed plan, and if so, the iteration is completed. If not, the action detailed plan is repeated, and if it cannot be completed after repeated execution, the large language model can also be used to regenerate the overall action plan, and the image data of the two frames before and after the execution is updated to verify whether the execution is successfully completed again.

[0129] Specific verification logic examples are as follows: 1. Obtain the environmental perception information related to the item in the user command before the action is executed, such as the coordinates 1 of item A. 2. Obtain the environmental perception information related to the item in the user command after the action is executed, such as the coordinates 2 of item A. 3. Determine whether the environmental perception information of the two frames before and after the action is executed matches the coordinate information of the item in the action detailed plan. If so, it is determined that the action detailed plan is completed. For example, it is determined whether the coordinates 2 of item A are close to the position where item A is intended to move in the action detailed plan, and if so, it is determined to be matched.

[0130] The technical effects achieved by the above specific embodiments of the present application are as follows: 1. The present application can receive natural language or unstructured instructions, understand user intent, and generate executable overall action sequences. At the same time, the present application also has the ability of action monitoring and dynamic adjustment to ensure that the robot can respond to environmental changes or unexpected situations during execution. For example, when executing the task of "taking a cup of water", the method plans the movement path, identifies the water cup position, adjusts the mechanical arm grasping posture, and re-adjusts the action sequence when detecting that the water cup has been moved, thereby ensuring the successful completion of the task.

[0131] 2. When facing a random task issued by the user across scenes, the robot can configure a task execution process according to the scene information and command content, and can decompose the task execution process into multiple actions, and can complete multiple actions through the execution-verification iteration method, thereby completing the non-fixed process issued by the user and improving the accuracy, precision and efficiency of the robot executing the user command.

[0132] 3. By retrieving a number of environmental perception information related to the user instruction from the memory bank to support the generation of the overall action sequence, the cross-scene task generation problem is solved.

[0133] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it. Although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the specific embodiments of the present application can be modified or equivalently replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application.

Claims

1. A task planning method of a robot, applicable to a server, characterized in that, The method comprises the following steps: Step 1: obtaining a user instruction; the user instruction is used to instruct a robot body to perform a task of a non-fixed process; Step 2: based on a large language model and a semantic encoding model, retrieving environment perception information related to the user instruction in a memory bank; The large language model is used to analyze the user instruction to obtain a candidate object related to the user instruction, and the semantic encoding model is used to encode the candidate object into semantic features in the same format as the memory bank; Step 3: inputting a first prompt word determined based on the user instruction and the environment perception information into the large language model to generate an overall action sequence corresponding to the user instruction; the first prompt word at least contains a plurality of sub-actions supported by the robot; Step 4: for each sub-action in the overall action sequence, determining a second prompt word, inputting the second prompt word into the large language model to generate a detailed action plan corresponding to the current sub-action, until each sub-action in the overall action sequence is traversed; the detailed action plan is composed of a plurality of basic actions supporting the sub-action.

2. The method of task planning for a robot according to claim 1, wherein, The step 2 comprises: obtaining a third prompt word, inputting the third prompt word into the large language model to obtain an object name related to the user instruction; the third prompt word is used to control the large language model to analyze the user instruction; determining the semantic features of the object name based on the semantic encoding model; based on the semantic features, retrieving at least one item of environment perception information similar to the semantic features in the memory bank; the semantic encoding model is a multi-modal semantic encoding model, which is used to encode texts and images expressing the same meaning into the same semantic features; performing content matching on the at least one item of environment perception information and the user instruction to obtain first de-duplicated environment perception information; the content matching is used to determine, in the at least one item of environment perception information, environment perception information matched with the object name related to the user instruction; based on a fourth prompt word, performing target matching on the first de-duplicated environment perception information to obtain second de-duplicated environment perception information, and determining the second de-duplicated environment perception information as the environment perception information related to the user instruction; wherein the fourth prompt word is used to control the large language model to determine the matched environment perception information among a plurality of environment perception information representing the same object name.

3. The method of claim 1, wherein, The step 3 comprises: obtaining a list of sub-actions that the robot can support; filling the sub-action list into a pre-configured first prompt word template to obtain a first prompt word; the replacement symbols in the first prompt word template include sub-action list, environment perception information, user instruction and data format of each of the overall action sequence; each of the replacement symbols is used to generate the first prompt word matched with the user instruction; inputting the first prompt word into the large language model to obtain the overall action sequence corresponding to the user instruction.

4. The method of claim 1, wherein, The step 4 comprises: traversing each sub-action in the overall action sequence in chronological order, and obtaining a current sub-action, an executor of the robot, and a plurality of basic actions that the executor can perform; Fill the current sub-action, the executor, and the plurality of basic actions that the executor can perform corresponding to the executor into a pre-configured second prompt word template to obtain a second prompt word; Input the second prompt word into a large language model to obtain a detailed action plan corresponding to the current sub-action.

5. The method of claim 1, wherein, The step 3 further comprises: Check the overall action sequence corresponding to the user instruction by a first verifier, and if the check fails, input the overall action sequence and the corresponding failure result into the large language model to re-generate the overall action sequence corresponding to the user instruction; The first verifier is determined based on a pre-configured verification rule and the feasibility of the overall action sequence analyzed by the large language model; The step 4 further comprises: Check each detailed action plan generated by the large language model by a second verifier, and if the check fails, input the detailed action plan and the corresponding failure result into the large language model to re-generate each detailed action plan corresponding to the overall action sequence until the verification is successful.

6. The method of claim 1, wherein, Before the step 1, it further comprises: Receive object information in each environment captured by the information acquisition device of the robot; the object information includes object name, image information and position information corresponding to the object name; the object includes environmental object, large object and small object; Process the image information and position information in the object information based on a pre-configured data structure to obtain environmental perception information corresponding to each object name; the environmental perception information at least includes RGB image, depth map, object category, detection frame and segmentation mask corresponding to each object name; Upload the environmental perception information to the memory bank.

7. The method according to claim 6, wherein The environmental perception information uploaded to the memory bank comprises: Perform de-duplication processing on the plurality of environmental perception information in the cache queue in chronological order to obtain de-duplicated environmental perception information; the de-duplication processing at least includes environmental object de-duplication processing and object object de-duplication processing; According to the detection frame in the environmental perception information, intercept the image information corresponding to each object object, and input the image information into a semantic coding model to encode to obtain the semantic features corresponding to the object object; Upload the semantic features and the environmental perception information associated with the semantic features to the memory bank in a pre-set format; the memory bank is used to recall environmental perception information with high similarity from similar semantic features. 8.The task planning method of the robot according to claim 1, wherein, After the step 4, it further comprises: For each detailed action plan corresponding to each sub-action, obtain current environmental perception information before the current sub-action is executed; Issue an execution command corresponding to the detailed action plan of the current sub-action to each executor of the robot; the executor is used to instruct the hardware device of the robot to perform a related control operation; the control operation at least includes navigation, grabbing, releasing, opening and closing; After the executor receives the corresponding execution command and executes it, obtain the latest environmental perception information after the current sub-action is executed; The current environment perception information before the current sub-action is executed and the latest environment perception information after the execution are input into the large language model to obtain a sub-action verification result, until the detailed action plan of all sub-actions of the overall action sequence is verified.

9. The method according to claim 8, wherein The current environment perception information before the current sub-action is executed and the latest environment perception information after the execution are input into the large language model to obtain a sub-action verification result, until the detailed action plan of all sub-actions of the overall action sequence is verified. Based on the current environment perception information before the current sub-action is executed and the latest environment perception information after the execution, a sub-action verification prompt word is generated. The sub-action verification prompt word is input into the large language model to determine whether the first coordinate value of the target object in the current environment perception information and the second coordinate value of the target object in the latest environment perception information match the expected result of the current sub-action, and if the first coordinate value and the second coordinate value match the expected result of the current sub-action, it is determined that the sub-action verification result is verified.

10. A task planning system of a robot, implementing the task planning method of the robot according to any one of claims 1 to 9, characterized in that, It includes: A user instruction acquisition module is configured to acquire a user instruction, wherein the user instruction is used to instruct a robot body to execute a task of a non-fixed process; An environment perception information retrieval module is configured to retrieve environment perception information related to the user instruction from a memory bank based on a large language model and a semantic encoding model; The large language model is used to analyze the user instruction to obtain a candidate object related to the user instruction, and the semantic encoding model is used to encode the candidate object into a semantic feature in the same format as the memory bank; An overall action sequence generation module is configured to input a first prompt word determined based on the user instruction and the environment perception information into the large language model to generate an overall action sequence corresponding to the user instruction, wherein the first prompt word at least includes a plurality of sub-actions supported by the robot; A detailed action plan generation module is configured to determine a second prompt word for each sub-action in the overall action sequence, input the second prompt word into the large language model, and generate a detailed action plan corresponding to the current sub-action until all sub-actions in the overall action sequence are traversed, wherein the detailed action plan is composed of a plurality of basic actions supporting the sub-action.

Images