A robot task planning method and device under implicit instruction driving

Abstract

The application discloses a kind of robot task planning methods and devices under implicit instruction driving, it is related to natural language processing technical field.The method includes: constructing auxiliary training module, obtaining implicit instruction image data and input module.In intention perception understanding module, implicit instruction intention is parsed through adapter, the region of interest is obtained by using target detection model to crop image and input large language model, the similarity of intention and attribute value is calculated, the perceived intention is determined, in visual marker demonstration module, the object of region of interest is converted into 2D segmentation mask, generates subtask 3D object and adds to image, and labels order to generate visual marker image;Image data, perceived intention and subtask image are input into large language model, and subtask is generated for robot task planning.The application can effectively improve the task planning precision of data containing implicit instruction.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a robot task planning method and apparatus driven by implicit instructions. Background Technology

[0002] As the computational demands of Large Language Models (LLMs) decrease, LLM-assisted robots are increasingly entering various fields such as homes, education, and industry, helping to complete long-term tasks. Currently, the main role of LLMs in long-term robotic tasks is task planning based on language instructions. Due to differences in culture and background, people's expectations for robot control are no longer limited to direct instructions or standardized templates, but rather to more personalized expressions. Just like communicating with people, instructions convey true intentions through emotional expression, causal cues, or other indirect means.

[0003] Existing technologies primarily focus on explicit instructions and their synonymous rewrites, as well as robot task planning under ambiguous or incomplete instructions. Models trained with large-scale data can better understand explicit instructions and their rewrites; while ambiguous and incomplete instructions can be clarified through dialogue, knowledge bases, or LLMs. However, these methods cannot handle robot task planning guided by implicit instructions (indirectly conveying true intentions through emotional expression, causal cues, etc.), resulting in robot task planning accuracy failing to meet actual planning requirements.

[0004] Therefore, there is an urgent need for a robot task planning method to improve the accuracy of task planning for data containing implicit instructions. Summary of the Invention

[0005] Therefore, it is necessary to provide a robot task planning method and apparatus driven by implicit instructions to address the above-mentioned technical problems.

[0006] The present invention adopts the following technical solution:

[0007] This invention provides a robot task planning method driven by implicit instructions, comprising:

[0008] An auxiliary training module for acquiring historical image data based on implicit instructions and inputting it into a large language model is included; the auxiliary training module includes an intent-aware understanding module and a visual model demonstration module.

[0009] In the intent-aware understanding module, the implicit instructions in historical image data are parsed using the question adapter and answer adapter of the large language model to generate intent parsing triples; the historical image data based on implicit instructions is identified and cropped using the object detection model to obtain the region of interest and input it into the pre-trained large language model to generate formal object attribute triples; the similarity between the parsed intent in the intent parsing triples and the attribute values ​​in the object attribute triples is calculated, and the one with the highest similarity is taken as the perceived intent of the implicit instruction;

[0010] In the visual labeling demonstration module, objects in the region of interest are converted into 2D segmentation masks and 3D object generation models are used to generate sub-task 3D objects. The sub-task 3D objects are added to historical image data based on implicit instructions according to specified positions and poses, and the order of sub-task 3D objects in the image is marked to obtain visually labeled sub-task images.

[0011] The large language model is trained by using historical image data based on implicit instructions, the perceived intent of the implicit instructions, and the sub-task images to obtain a trained large language model.

[0012] The image data based on implicit instructions to be planned is input into the trained large language model to generate the current subtask, and the current subtask is scored and corrected to determine the robot's task planning result.

[0013] Preferably, the specific form of the intent parsing triple is [implicit instruction, intent type, parsed intent].

[0014] Preferably, the intent types include: physiological, emotional, health, safety, hobby, entertainment, and cognition; the parsed intent is the intent inferred from the implicit instructions by the question adapter and answer adapter of the large language model.

[0015] Preferably, the specific form of the object attribute triple is [object name, attribute type, attribute value].

[0016] Preferably, the attribute types include physical attributes and functional attributes; the attribute values ​​are obtained by extracting from the knowledge base of the large language model.

[0017] Preferably, the current subtask is scored and corrected to determine the robot's task planning result, specifically including:

[0018] The image data based on implicit instructions to be planned, the preset rule-based thought chain, and the current subtask are input into GPT-4o. The rule-based thought chain guides GPT-4o and outputs the planning result of the current subtask.

[0019] Input the planning results of the current subtask into the evaluation model and score the planning results of the current subtask;

[0020] If the score of the planning result of the current subtask is less than the preset score threshold, the planning result of the current subtask will be corrected.

[0021] Based on the corrected sub-tasks, determine the robot's task planning results.

[0022] Preferably, the planning results of the current subtask are corrected, specifically including:

[0023] Retrieve the subtask most similar to the current subtask from the experience pool;

[0024] The consultant model is fed with the subtask most similar to the current subtask, the image data based on implicit instructions corresponding to the current subtask, the planning result of the current subtask, the inference result of the evaluation model, and the suggestions to obtain the correction suggestions for the planning result of the current subtask.

[0025] The correction suggestions are then fed back to the trained large language model to regenerate a subtask based on historical image data with implicit instructions, i.e., the corrected subtask.

[0026] Preferably, the method further includes:

[0027] If the score of the planning result of the current subtask is greater than or equal to the preset score threshold, it is considered a correct planning, and the image data based on implicit instructions to be planned and the generated subtask are saved to the reserved experience pool, and the planning result of the current subtask is used as the planning result of the robot.

[0028] The present invention also provides a robot task planning device driven by implicit instructions, comprising:

[0029] The intent-aware understanding module is used to parse the implicit instructions in historical image data through the question adapter and answer adapter of the large language model, and generate intent parsing triples; it uses the object detection model to identify and crop the historical image data based on implicit instructions, obtain the region of interest, and input it into the pre-trained large language model to generate formal object attribute triples; it calculates the similarity between the parsed intent in the intent parsing triples and the attribute values ​​in the object attribute triples, and takes the one with the highest similarity as the perceived intent of the implicit instruction;

[0030] The visual labeling demonstration module is used to convert objects in the region of interest into 2D segmentation masks and generate sub-task 3D objects using a 3D object generation model; add the sub-task 3D objects to historical image data based on implicit instructions according to specified positions and poses, and mark the order of sub-task 3D objects in the image to obtain visually labeled sub-task images.

[0031] The automatic error correction module is used to score and correct the current subtask, and determine the robot's task planning results.

[0032] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned implicit instruction-driven robot task planning method.

[0033] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described implicit instruction-driven robot task planning method.

[0034] The above-mentioned at least one technical solution adopted in this invention can achieve the following beneficial effects:

[0035] In the implicit instruction-driven robot task planning method provided by this invention, an auxiliary training module for a large language model is constructed. Its intent perception and understanding module performs intent parsing and attribute reasoning on images containing implicit instructions. By calculating the similarity between intent and attributes, implicit expressions are made explicit into clear intents. The visual labeling demonstration module decomposes explicit intents into simple sub-tasks in the large language model, effectively improving the reasoning ability of the large language model and effectively guiding robot task planning. Attached Figure Description

[0036] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0037] Figure 1 A flowchart illustrating a robot task planning method driven by implicit instructions provided by the present invention;

[0038] Figure 2 A schematic diagram of an auxiliary training module for a robot task planning method driven by implicit instructions provided by the present invention;

[0039] Figure 3 A schematic diagram of an automatic correction module for a robot task planning method driven by implicit instructions provided by the present invention;

[0040] Figure 4 A schematic diagram of a robot task planning device driven by implicit instructions provided by the present invention;

[0041] Figure 5 A diagram of a computer device for implementing a robot task planning method driven by implicit instructions, provided by the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in the specification without creative effort are within the scope of protection of this application.

[0043] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0044] Figure 1 This is a flowchart illustrating a robot task planning method driven by implicit instructions according to the present invention, which specifically includes the following steps:

[0045] S101: Acquire historical image data based on implicit instructions and input it into the auxiliary training module of the large language model; the auxiliary training module includes an intent perception and understanding module and a visual model demonstration module.

[0046] Specifically, the auxiliary training module, including intent-aware understanding and visually labeled demonstrations, further enhances the VLM's ability to reason from implicit instructions and images to simple sub-tasks for the robot. Intent-aware understanding resolves the intent of implicit instructions under environmental constraints, deriving it from the highest similarity score between the implicit instruction intent and object attributes in the image. Therefore, intent-aware understanding helps the VLM resolve implicit expressions into explicit intentions. Visually labeled demonstrations provide a fine-grained representation of sub-task sequence dependencies. Therefore, visually labeled demonstrations facilitate mapping from explicit intentions to simple sub-tasks. Specifically, the original implicit instruction-based data is copied into new data, intent-aware understanding is added to the new data, and images are replaced with visually labeled demonstrations. The original and new data are then input into the VLM for fine-tuning training to obtain a trained large language model. This trained large language model is then used for task planning on the images to be planned.

[0047] S102: In the intent-aware understanding module, the implicit instructions in historical image data are parsed using the question adapter and answer adapter of the large language model to generate intent parsing triples; the historical image data based on implicit instructions is identified and cropped using the object detection model to obtain the region of interest and input it into the pre-trained large language model to generate formal object attribute triples; the similarity between the parsed intent in the intent parsing triples and the attribute values ​​in the object attribute triples is calculated, and the one with the highest similarity is taken as the perceived intent of the implicit instruction.

[0048] Optionally, the specific form of the intent resolution triple is [implicit instruction, intent type, resolved intent].

[0049] Optionally, the types of intent include: physiological, emotional, health, safety, hobby, entertainment, and cognitive.

[0050] Optionally, the parsed intent is the intent inferred from the implicit instructions by the question adapter and answer adapter of the large language model.

[0051] Specifically, firstly, when the input implicit instruction is "throat on fire" and the corresponding image, the output is the parsed semantic triple [throat on fire, physiological, drinking water]. Here, "throat on fire" is the implicit instruction, "physiological" represents the intent type, and "drinking water" is the parsed possible intent. The image knowledge triple is extracted as [faucet, use, take water], [cup, use, hold liquid], etc., where "faucet" and "cup" are object names, "use" is the functional attribute type, and "take water" and "hold liquid" are the corresponding attribute values. Then, the similarity score between the parsed intent in the semantic triple and the attribute values ​​in the image knowledge triple is calculated, such as 0.87, 0.62, etc. The highest similarity occurs between [throat on fire, physiological, drinking water] and [faucet, use, take water], at 0.87, because "drinking water" and "take water" are semantically closest. "[throat on fire, physiological, drinking water]-[faucet, use, take water]-similarity:0.87" is the intention perception understanding.

[0052] Specifically, Sentence-BERT is used to calculate the similarity score between the parsed intent in the language semantic triple and the attribute value in the image knowledge triple, where the highest score is defined as intent-aware understanding.

[0053] Optionally, the specific form of the object attribute triple is [object name, attribute type, attribute value].

[0054] Optionally, attribute types include: physical attributes and functional attributes.

[0055] Optionally, attribute values ​​are obtained by extracting them from the knowledge base of a large language model.

[0056] Specifically, GDino-1.6pro is used to identify and crop objects of interest; then, text prompts and cropped objects are input into VLM to generate the objects' physical properties (such as color, shape, and material). Functional attribute values ​​are obtained by querying the Wikidata, ConceptNet, and WordNet knowledge bases.

[0057] S103: In the visual labeling demonstration module, the objects in the region of interest are converted into 2D segmentation masks and 3D object generation models are used to generate sub-task 3D objects; the sub-task 3D objects are added to the historical image data based on implicit instructions according to the specified position and pose, and the order of the sub-task 3D objects in the image is marked to obtain the visually labeled sub-task image.

[0058] S104: Train the large language model using historical image data based on implicit instructions, the perceived intent of the implicit instructions, and the sub-task images to obtain a trained large language model.

[0059] S105: Input the image data based on implicit instructions to be planned into the trained large language model, generate the current sub-task, score and correct the current sub-task, and determine the robot's task planning result.

[0060] Optionally, the current subtask is scored and corrected to determine the robot's task planning result. Specifically, this includes: inputting the image data based on implicit instructions to be planned, the preset rule-based thought chain, and the current subtask into GPT-4o; the rule-based thought chain guides GPT-4o to output the planning result of the current subtask; inputting the planning result of the current subtask into the evaluation model to score the planning result of the current subtask; if the score of the planning result of the current subtask is less than a preset scoring threshold, the planning result of the current subtask is corrected; and based on the corrected subtask, the robot's task planning result is determined.

[0061] Specifically, the subtask includes the robot's action execution process, which is the robot's planning result.

[0062] Optionally, the planning result of the current subtask is corrected, specifically including: retrieving the subtask most similar to the current subtask from the experience pool; inputting the subtask most similar to the current subtask, the image data based on implicit instructions corresponding to the current subtask, the planning result of the current subtask, the inference result of the evaluation model, and the suggestion prompts into the advisor model to obtain a correction suggestion for the planning result of the current subtask; and feeding the correction suggestion back to the trained large language model to regenerate the subtask based on the historical image data of implicit instructions, i.e., the corrected subtask.

[0063] Optionally, the method further includes:

[0064] If the score of the planning result of the current subtask is greater than or equal to the preset score threshold, it is considered a correct planning. The image data based on implicit instructions to be planned and the generated subtask are saved to the reserved experience pool, and the planning result of the current subtask is used as the planning result of the robot.

[0065] Optionally, the planning result of the current subtask is corrected, specifically including: retrieving the subtask most similar to the current subtask from the experience pool; inputting the subtask most similar to the current subtask, the historical image data based on implicit instructions corresponding to the current subtask, the planning result of the current subtask, the inference result of the evaluation model, and the suggestion prompts into the advisor model to obtain a correction suggestion for the planning result of the current subtask; and feeding the correction suggestion back to the trained large language model to regenerate the subtask based on the historical image data of implicit instructions.

[0066] Specifically, such as Figure 3 As shown, in the evaluation phase, we first use the GPT-4o evaluation model to determine whether the generated subtask is correct. Its inputs include implicit instructions, images, generated subtasks, and rule-based thought chain (CoT) hints. The output includes the reasoning process and a score. The CoT hints guide GPT-4o to progressively analyze the intent of the implicit instructions in the image and score the planning result according to pre-defined rules and standards. If the score exceeds a threshold, it is considered a correct plan, and the implicit instructions, images, and generated subtasks are saved to the experience pool as the final output.

[0067] Specifically, when the score falls below a threshold, the system enters an error correction phase. This involves first using Sentence-BERT to retrieve the most similar case from the experience pool to the current task and inputting it into the advisor model (GPT-4o). At this point, the retrieved similar tasks, implicit instructions, images, current planning results, the inference results of the evaluation model, and suggested hints are all fed into GPT-4o to generate corrective suggestions for the planning. These suggestions are then fed back to the deep inference VLM used for task planning to further improve the accuracy of sub-task planning.

[0068] Specifically, on 400 test samples of robot task planning guided by implicit instructions, we achieved an accuracy of 94%, significantly outperforming existing baseline methods. The baselines include closed-source models (Claude 3.5Sonnet, Gemini-1.5-Flash, Qwen-VL-Max, GPT-4o) and open-source models fine-tuned on our dataset (Qwen2-VL-7B, InternVL-2-4B). Ablation experiments show that both the auxiliary training modules (intent-aware understanding and visual labeling demonstration) and the automatic error correction module contribute to improving task planning accuracy. Without any added modules, the task planning accuracy was 69.3%; with only intent-aware understanding, it was 89.8%; with only visual labeling demonstration, it was 85.5%; with the auxiliary training module, it was 92.0%; and with both the auxiliary training module and the automatic error correction module, it was 94.0%.

[0069] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

[0070] The above describes a robot task planning method driven by implicit instructions, provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding robot task planning device driven by implicit instructions, such as... Figure 4 As shown.

[0071] Figure 4 This is a schematic diagram of a robot task planning device driven by implicit instructions provided by the present invention. The device 400 includes:

[0072] The intent perception and understanding module 401 is used to parse the implicit instructions in historical image data based on implicit instructions through the question adapter and answer adapter of the large language model, and generate intent parsing triples; to identify and crop the historical image data based on implicit instructions through the object detection model, obtain the region of interest and input it into the large language model to generate formal object attribute triples; to calculate the similarity between the parsed intent in the intent parsing triples and the attribute values ​​in the object attribute triples, and take the one with the highest similarity as the perceived intent of the implicit instruction;

[0073] The visual labeling demonstration module 402 is used to convert objects in the region of interest into 2D segmentation masks and generate sub-task 3D objects using a 3D object generation model; add the sub-task 3D objects to historical image data based on implicit instructions according to specified positions and poses, and mark the order of sub-task 3D objects in the image to obtain visually labeled sub-task images.

[0074] The automatic error correction module 403 is used to guide the rule-based thinking chain to infer the intent of implicit instructions in historical image data based on implicit instructions, and obtain the planning result of the current subtask. According to the preset standard, the planning result of the current subtask is scored. If the score of the planning result of the current subtask is greater than or equal to the preset scoring threshold, it is considered as correct planning, and the historical image data based on implicit instructions and the generated subtask are saved to the experience pool as the final planning result. If the score of the planning result of the current subtask is less than the preset scoring threshold, the planning result is corrected.

[0075] For specific limitations regarding an implicit instruction-driven robot task planning device, please refer to the limitations of an implicit instruction-driven robot task planning method described above, which will not be repeated here. Each module in the aforementioned implicit instruction-driven robot task planning device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0076] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This paper presents a robot task planning method driven by implicit instructions.

[0077] The present invention also provides Figure 5 The schematic diagram of the computer device shown is as follows: Figure 5 As shown, at the hardware level, this computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then executes it to achieve the above. Figure 1 This paper presents a robot task planning method driven by implicit instructions.

[0078] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

Claims

1. A method of robot task planning under implicit instruction driving, characterized by, The method comprises the following steps: acquiring historical image data based on implicit instructions and inputting the historical image data into an auxiliary training module of a large language model; the auxiliary training module comprises an intention perception understanding module and a visual marker demonstration module; in the intention perception understanding module, the intention of the historical image data based on implicit instructions is parsed through a question adapter and an answer adapter of the large language model, and an intention parsing triple is generated; the historical image data based on implicit instructions is recognized and cropped through a target detection model, a region of interest is obtained, and the region of interest is input into a pre-trained large language model to generate an object attribute triple in a form; the similarity between the parsed intention in the intention parsing triple and the attribute value in the object attribute triple is calculated, and the highest similarity is taken as the perceived intention of the implicit instruction; in the visual marker demonstration module, the objects in the region of interest are converted into 2D segmentation masks, and a subtask 3D object is generated by using a 3D object generation model; the subtask 3D object is added to the historical image data based on implicit instructions according to a specified position and attitude, and the order of the subtask 3D object in the image is marked to obtain a visual marked subtask image; the large language model is trained by using the historical image data based on implicit instructions, the perceived intention of the implicit instruction, and the subtask image, and a trained large language model is obtained; the image data based on implicit instructions to be planned is input into the trained large language model, a current subtask is generated, the current subtask is scored and corrected, and the task planning result of the robot is determined.

2. A method of robot task planning under implicit instruction as claimed in claim 1, characterized in that, The specific form of the intention parsing triple is [implicit instruction, intention type, parsed intention].

3. The robot task planning method driven by implicit instructions according to claim 2, wherein the intention type includes: physiological, emotional, health, safety, hobby, entertainment, and cognitive; the parsed intention is an intention inferred by the question adapter and the answer adapter of the large language model from the implicit instruction.

4. The method of claim 1, wherein, The specific form of the object attribute triple is [object name, attribute type, attribute value].

5. The robot task planning method driven by implicit instructions according to claim 4, wherein the attribute type includes: physical attribute and functional attribute; the attribute value is obtained by extracting the knowledge base of the large language model.

6. The method of claim 1, wherein, The scoring and correction of the current subtask to determine the task planning result of the robot specifically include: inputting the image data based on implicit instructions to be planned, a pre-set rule-based thinking chain, and the current subtask into GPT-4o, guiding GPT-4o based on the rule-based thinking chain, and outputting the planning result of the current subtask; inputting the planning result of the current subtask into an evaluation model to score the planning result of the current subtask; if the score of the planning result of the current subtask is less than a pre-set score threshold, the planning result of the current subtask is corrected; determining the task planning result of the robot according to the corrected subtask.

7. A method of robot task planning under implicit instruction as claimed in claim 6, characterized in that, The correction of the planning result of the current subtask specifically includes: retrieving a subtask most similar to the current subtask from an experience pool; The most similar subtask to the current subtask, the image data based on the implicit instruction corresponding to the current subtask, the planning result of the current subtask, the inference result of the evaluation model, and the suggestion prompt are input into the consultant model to obtain a correction suggestion for the planning result of the current subtask; And the correction suggestion is fed back to the trained large language model to regenerate the subtask based on the historical image data based on the implicit instruction, that is, the corrected subtask.

8. The method of claim 6, wherein the robot task planning under implicit instruction is characterized by, The method further comprises: If the score of the planning result of the current subtask is greater than or equal to the preset score threshold, it is considered to be correct planning, and the image data based on the implicit instruction to be planned and the generated subtask are saved to the reserved experience pool, and the planning result of the current subtask is taken as the planning result of the robot.

9. A robot task planning apparatus under implicit instruction driving, characterized by, Comprise: The data input module is used for acquiring the historical image data based on the implicit instruction and inputting the same into the auxiliary training module of the large language model; The auxiliary training module comprises an intention perception understanding module and a visual marker demonstration module; The intention perception understanding module is used for performing intention analysis on the implicit instruction of the historical image data by the question adapter and the answer adapter of the large language model to generate an intention analysis triple; The historical image data based on the implicit instruction is recognized and cropped by a target detection model to obtain a region of interest and input the same into a pre-trained large language model to generate an object attribute triple; the similarity between the analyzed intention in the intention analysis triple and the attribute value in the object attribute triple is calculated, and the one with the highest similarity is taken as the perception intention of the implicit instruction; The visual marker demonstration module is used for converting the object in the region of interest into a 2D segmentation mask and generating a subtask 3D object by using a 3D object generation model; the subtask 3D object is added to the historical image data based on the implicit instruction according to a specified position and posture, and the order of the subtask 3D object in the image is marked to obtain a visually marked subtask image; The model training module trains the large language model by using the historical image data based on the implicit instruction, the perception intention of the implicit instruction, and the subtask image to obtain a trained large language model; The automatic error correction module is used for inputting the image data based on the implicit instruction to be planned into the trained large language model to generate a current subtask, scoring and correcting the current subtask, and determining the task planning result of the robot.

Images