Tool usage command planning method based on large language model and fine-tuning optimization

Abstract

This application relates to a command planning method for tools based on a large language model and fine-tuning optimization. The method includes the basic workflow of command planning, such as prior knowledge acquisition, environmental target detection, and execution step generation. It incorporates a human-computer interaction component and combines two interactive visible and invisible exploration strategies for online command planning. When facing unknown scenarios and tasks, key information from human-recognized command plans is stored. When a sufficient number of command plans are stored, for the same human instruction, an offline command planning framework based on a retrieval-enhanced generation strategy is used to retrieve the results from the valid information of the stored correct command plans at each stage. This solves the problem in existing technologies of effectively balancing the universality of scenario instructions with the executability of generated execution steps.

Description

Technical Field

[0001] This application relates to the field of command planning and design technology, and in particular to a tool-use command planning method based on a large language model and fine-tuning optimization. Background Technology

[0002] The development of Large Language Models (LLMs) has garnered significant attention in academia. Leveraging massive datasets from sources like the internet and advanced hardware training facilities, LLMs demonstrate exceptional inductive, comprehension, and reasoning capabilities across multiple domains. Consequently, research on LLMs has surged both domestically and internationally in recent years. One particularly noteworthy area is the application of LLMs to embodied intelligent robots. This involves enabling LLMs to utilize their rich knowledge to provide robots with action steps that can be executed in response to human natural language commands, allowing robots to physically fulfill human instructions.

[0003] To achieve the above goals, the mainstream approach in academia is to first give the robot natural language instructions that do not express information about the surrounding environment; then, a large language model, taking into account its rich knowledge and information about the surrounding environment, will give the corresponding action steps based on the human natural language instructions.

[0004] Currently, the SayCan model can utilize the semantic understanding capabilities of a large language model to process natural language instructions given by humans. It then uses the latter to decompose the received instructions into several small executable task units. Through the large language model, it evaluates the contribution rate of a specific action step to the completion of the final task, and through reinforcement learning, it evaluates the probability that the action step can be completed in reality. The contribution rate and the completion probability are multiplied to obtain the final value function. The most suitable action step is selected by choosing the value function with the largest value.

[0005] Secondly, the PaLM-E model has significant end-to-end multimodal characteristics. It can encode natural language instructions given by humans and information from the external environment into vectors of the same format in a certain way, and input them into the model for training. While fulfilling human instructions, it can also complete other multimodal tasks (such as scene description based on input images), and performs well on OK-VQA.

[0006] Furthermore, the LM-Nav model does not require labeled data or fine-tuning of a large model, and can effectively follow human natural language instructions to complete navigation tasks. It emphasizes the collaborative operation of multiple dedicated models, including the Visual-Navigation Module (VNM), the Visual-Language Module (VLM), and the Large Language Model Module (LLM). By utilizing the VNM and VLM modules, the LM-Nav model can extract navigation landmarks from human natural language instructions, thereby navigating to a designated location.

[0007] Furthermore, the goal of the TaPA (TAsk Planning Agent) model is to provide execution steps in natural language that conform to human habits for general tasks requiring natural human commands. The approach involves using the CLIP method to identify object labels in the environment surrounding the embodied intelligent robot; then, using natural language as prompts, combining these object labels into a pre-designed overall prompt; and finally, using GPT-3.5 under this prompt to generate a human command-execution step dataset. This generated dataset is then used to fine-tune the LLaMA pre-trained large model, resulting in a large model that achieves the desired performance.

[0008] However, current academic research on embodied intelligence command planning tasks rarely considers both the universality of scenario commands and the executableness of generated execution steps. Several influential works, while focusing on the accuracy and executableness of command planning, often impose strict constraints on human commands and task scenarios. For example, the SayCan model imposes strict limitations on the task scenarios and actions of embodied intelligent robots, and performs poorly in untrained scenarios and with complex commands; the PaLM-E model also has strict limitations on task scenarios. Furthermore, efforts to enhance the universality of command planning under complex human commands and task scenarios are likely to encounter problems such as vague command planning lacking actual execution details due to the difficulty in extracting sufficient detailed information from the surrounding environment. For instance, the TaPA model's action steps generated for human commands lack execution details; they merely conform to human cognitive habits but cannot be truly executed by the robot.

[0009] In summary, existing technologies struggle to effectively balance the universality of scenario instructions with the executableness of generated execution steps, a problem that urgently needs to be addressed. Summary of the Invention

[0010] This application provides a tool usage command planning method based on a large language model and fine-tuning optimization to solve the problem that existing technologies cannot effectively balance the universality of scenario instructions with the executableness of generated execution steps.

[0011] The first aspect of this application provides a tool usage command planning method based on a large language model and fine-tuning optimization, applied in the online command planning stage, including the following steps: acquiring target human instructions, generating an image of the target's surrounding environment based on a preset online data acquisition strategy, and performing a command planning operation according to a preset large language model, the target human instructions, and the image of the target's surrounding environment to generate target candidate tools and a first command plan in the image of the target's surrounding environment, and determining whether the first command plan meets preset reasonable requirements; if the first command plan meets the preset reasonable requirements, then the first command plan is used as the final command plan; if the first command plan does not meet the preset reasonable requirements, then a first-level visible exploration operation is performed on the target candidate tools to generate a first local exploration region and a second command plan corresponding to the local exploration region, and determining whether the second command plan meets the preset reasonable requirements; if the second command plan does not meet the preset reasonable requirements, then a first-level visible exploration operation is performed on the target candidate tools to generate a first local exploration region and a second command plan corresponding to the local exploration region, and determining whether the second command plan meets the preset reasonable requirements. If the preset reasonable requirements are met, the second command plan is taken as the final command plan; if the second command plan does not meet the preset reasonable requirements, a second-level visible exploration operation is performed on the first local exploration area to obtain a second local exploration area, and a command planning operation is performed on the second local exploration area to generate a third command plan corresponding to the second local exploration area. It is then determined whether the third command plan meets the preset reasonable requirements. If the third command plan does not meet the preset reasonable requirements, the target embodied intelligent robot is controlled to perform an invisible exploration operation to generate a target invisible area, and the target embodied intelligent robot explores the target invisible area. If the third command plan meets the preset reasonable requirements, the third command plan is taken as the final command plan, and the planning information corresponding to the final command plan is acquired and stored, so as to use the planning information to perform an offline command plan generation operation in the offline command planning stage.

[0012] Optionally, in one embodiment of this application, the step of acquiring the target human instruction, generating a target surrounding environment image based on a preset online data acquisition strategy, and executing a command planning operation according to a preset large language model, the target human instruction, and the target surrounding environment image to generate target candidate tools and a first command plan in the target surrounding environment image includes: generating target tool labels and target tool appearance properties corresponding to the target human instruction based on the target human instruction and the large language model; randomly selecting an initial image and multiple interference images from a preset dataset according to the target tool labels; randomly splicing the initial image and multiple interference images to obtain the target surrounding environment image; inputting the target tool labels into a preset visual target detection model to generate different candidate tool schematic diagrams; and executing a command planning operation according to the target surrounding environment image, the different candidate tool schematic diagrams, and the target surrounding environment image. A visual target detection model acquires the confidence scores of different candidate tools; the confidence scores of the candidate tools are sorted in descending order to obtain a ranking result, and multiple target candidate tool confidence scores that meet preset confidence requirements are selected based on the ranking result. The ranking index of the multiple target candidate tool confidence scores in the ranking result and the target candidate tools corresponding to the multiple target candidate tool confidence scores are also acquired. Background filtering and tool recombination operations are performed on the target candidate tools to obtain a recombined candidate tool map. The recombined candidate tool map, the target tool label, and the target tool appearance properties are input into the large language model to filter out the target ranking index corresponding to the target human command. Based on a preset segmentation model, the target candidate tools corresponding to the target ranking index are segmented and filtered to obtain a corresponding segmentation map. The segmentation map is input into the large language model to generate the first command plan.

[0013] Optionally, in one embodiment of this application, the step of performing a first-level visible exploration operation on the target candidate tool to generate a first local exploration region and a second command plan corresponding to the local exploration region includes: filtering the confidence scores of the top N target candidate tools from the multiple target candidate tool confidence scores according to the ranking result, and obtaining the target candidate tools corresponding to the confidence scores of the top N target candidate tools, where N is a positive integer; adding the target candidate tool corresponding to the target ranking index to the target candidate tools corresponding to the confidence scores of the top N target candidate tools to obtain multiple target candidate tools to be filtered; performing splicing and background filtering operations on the multiple target candidate tools to be filtered to generate the first local exploration region, and inputting the first local exploration region into the large language model to generate the second command plan.

[0014] Optionally, in one embodiment of this application, the step of performing a second-level visible exploration operation on the first local exploration region to obtain a second local exploration region, and performing a command planning operation on the second local exploration region to generate a third command plan corresponding to the second local exploration region, includes: filtering out multiple candidate tool confidence scores corresponding to the target surrounding environment image that meet the preset sorting number requirements from the sorting results, and obtaining multiple target candidate tools corresponding to the multiple candidate tool confidence scores; performing weight accumulation on the target region of each target candidate tool among the multiple target candidate tools to obtain a weight accumulation result corresponding to each target candidate tool, and comparing the weight accumulation results of each target candidate tool to obtain a maximum weight accumulation result; obtaining the target candidate tool corresponding to the maximum weight accumulation result, and using the target region of the target candidate tool corresponding to the maximum weight accumulation result as the second local exploration region; and inputting the second local exploration region into the large language model to generate the third command plan.

[0015] A second aspect of this application provides a tool usage command planning method based on a large language model and fine-tuning optimization, applied to the offline command planning stage, comprising the following steps: acquiring pre-stored planning information based on a preset offline data acquisition strategy and a retrieval enhancement generation strategy, retrieving target tool images from the planning information, calculating a first similarity between the target tool images and a plurality of preset target candidate tools, and obtaining the optimal target candidate tool with the highest first similarity; retrieving target operation region images corresponding to the optimal target candidate tool in the planning information using the retrieval enhancement generation strategy, calculating a second similarity between the target operation region images and a plurality of preset candidate regions, and filtering out at least one target operation region image whose second similarity meets a preset similarity requirement, so that the retrieval enhancement generation strategy can be based on the at least one target operation region image. The system retrieves the corresponding first command planning information from the planning information in the region image retrieval, and determines whether the first command planning information meets preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, it is used as the final command planning. If the first command planning information does not meet the preset reasonable requirements, the system retrieves the key region image corresponding to the optimal target candidate tool in the planning information through the retrieval enhancement generation strategy, and calculates the third similarity between the key region image and the multiple key region images corresponding to the multiple target candidate tools. Based on the target candidate tool corresponding to the target key region image with the highest third similarity, the system generates the corresponding second command planning information. Based on the second command planning information and the preset reasonable requirements, the system performs preset second-level visible exploration and / or invisible exploration operations to generate the final command planning.

[0016] A third aspect of this application provides a tool usage command planning device based on a large language model and fine-tuning optimization, applied in the online command planning stage, comprising: a command planning module, configured to acquire target human instructions, and based on a preset online data acquisition strategy, generate an image of the target's surrounding environment, and execute a command planning operation according to a preset large language model, the target human instructions, and the image of the target's surrounding environment to generate target candidate tools and a first command plan in the image of the target's surrounding environment, and determine whether the first command plan meets preset reasonable requirements; if the first command plan meets the preset reasonable requirements, then the first command plan is adopted as the final command plan; and a first exploration module, configured to, if the first command plan does not meet the preset reasonable requirements, perform a first-level visible exploration operation on the target candidate tools to generate a first local exploration region and a second command plan corresponding to the local exploration region, and determine whether the second command plan meets the preset reasonable requirements; if the second command plan does not meet the preset reasonable requirements, then the second command plan is adopted as the final command plan. If the second command plan meets the preset reasonable requirements, then the second command plan is used as the final command plan. The second exploration module is used to perform a second-level visible exploration operation on the first local exploration area to obtain a second local exploration area if the second command plan does not meet the preset reasonable requirements, and to perform a command planning operation on the second local exploration area to generate a third command plan corresponding to the second local exploration area. It also determines whether the third command plan meets the preset reasonable requirements. If the third command plan does not meet the preset reasonable requirements, it controls the target embodied intelligent robot to perform an invisible exploration operation to generate an invisible target area. The target embodied intelligent robot then explores the invisible target area. If the third command plan meets the preset reasonable requirements, then the third command plan is used as the final command plan. The module also acquires and stores the planning information corresponding to the final command plan, and uses the planning information to perform an offline command plan generation operation during the offline command planning stage.

[0017] Optionally, in one embodiment of this application, the command planning module includes: an online data acquisition unit, configured to generate target tool labels and target tool appearance properties corresponding to the target human command based on the target human command and the large language model, and randomly select an initial image and multiple interference images from a preset dataset according to the target tool labels, and randomly stitch the initial image and multiple interference images to obtain an image of the target's surrounding environment; a first acquisition unit, configured to input the target tool labels into a preset visual target detection model to generate different candidate tool schematic diagrams, and obtain the candidate tool confidence scores corresponding to different candidate tools according to the image of the target's surrounding environment, the different candidate tool schematic diagrams, and the visual target detection model; and a coarse screening unit, configured to sort the candidate tool confidence scores in descending order. The system first performs a sorting process to obtain a sorting result, and then filters out multiple target candidate tools that meet a preset confidence level based on the sorting result. It also obtains the sorting index of the multiple target candidate tool confidence levels in the sorting result and the target candidate tools corresponding to the multiple target candidate tool confidence levels. A fine-tuning unit performs background filtering and tool recombination operations on the target candidate tools to obtain a recombined candidate tool map. The recombined candidate tool map, the target tool label, and the appearance properties of the target tool are input into the large language model to filter out the target sorting index corresponding to the target human command. A segmentation unit performs segmentation and filtering operations on the target candidate tools corresponding to the target sorting index based on a preset segmentation model to obtain a corresponding segmentation map. The segmentation map is then input into the large language model to generate the first command plan.

[0018] Optionally, in one embodiment of this application, the first exploration module includes: a second acquisition unit, configured to filter the confidence scores of the top N target candidate tools among the multiple target candidate tool confidence scores according to the ranking result, and acquire the target candidate tools corresponding to the confidence scores of the top N target candidate tools, where N is a positive integer; an addition unit, configured to add the target candidate tools corresponding to the target ranking index to the target candidate tools corresponding to the confidence scores of the top N target candidate tools, so as to obtain multiple target candidate tools to be filtered; and a filtering unit, configured to perform splicing and background filtering operations on the multiple target candidate tools to be filtered, so as to generate the first local exploration region, and input the first local exploration region into the large language model to generate the second command plan.

[0019] Optionally, in one embodiment of this application, the second exploration module includes: a third acquisition unit, configured to filter out multiple candidate tool confidence scores corresponding to the target surrounding environment image that meet the preset sorting number requirements from the sorting results, and acquire multiple target candidate tools corresponding to the multiple candidate tool confidence scores; a weight accumulation unit, configured to accumulate the weights of the target region of each target candidate tool among the multiple target candidate tools to obtain the weight accumulation result corresponding to each target candidate tool, and compare the weight accumulation result of each target candidate tool to obtain the maximum value of the weight accumulation result; a fourth acquisition unit, configured to acquire the target candidate tool corresponding to the maximum value of the weight accumulation result, and use the target region of the target candidate tool corresponding to the maximum value of the weight accumulation result as the second local exploration region; and a generation unit, configured to input the second local exploration region into the large language model to generate the third command plan.

[0020] A fourth aspect of this application provides a tool usage command planning device based on a large language model and fine-tuning optimization, applied in the offline command planning stage, comprising: a retrieval module, configured to acquire pre-stored planning information based on a preset offline data acquisition strategy and a retrieval enhancement generation strategy, and retrieve target tool images from the planning information, and calculate a first similarity between the target tool images and a preset plurality of target candidate tools, and obtain the optimal target candidate tool with the highest first similarity; a first calculation module, configured to retrieve target operation region images corresponding to the optimal target candidate tool in the planning information using the retrieval enhancement generation strategy, and calculate a second similarity between the target operation region images and a preset plurality of candidate regions, and filter out at least one target operation region image whose second similarity meets a preset similarity requirement, so that the retrieval enhancement generation strategy is based on the at least one target operation region image. The first command planning information is retrieved from the planning information and its corresponding first command planning information is determined. If the first command planning information meets the preset reasonable requirements, it is used as the final command planning information. The second calculation module is used to retrieve the key area image corresponding to the optimal target candidate tool in the planning information through the retrieval enhancement generation strategy if the first command planning information does not meet the preset reasonable requirements. The third similarity between the key area image and the multiple key area images corresponding to the multiple target candidate tools is calculated respectively. Based on the target candidate tool corresponding to the target key area image with the highest third similarity, the corresponding second command planning information is generated. The offline exploration module is used to perform preset second-level visible exploration and / or invisible exploration operations based on the second command planning information and the preset reasonable requirements to generate the final command planning information.

[0021] A fifth aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the tool usage command planning method based on a large language model and fine-tuning optimization as described in the above embodiments.

[0022] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described tool-use command planning method based on a large language model and fine-tuning optimization.

[0023] A seventh aspect of this application provides a computer program product, including a computer program that is executed to implement the above-described tool usage command planning method based on a large language model and fine-tuning optimization.

[0024] Therefore, the embodiments of this application have the following beneficial effects:

[0025] The embodiments of this application, after inputting human commands and images of the surrounding environment, first go through the basic process of command planning to generate corresponding command plans, and then query the human to determine if the command plan is reasonable. If reasonable, the command plan is taken as the final result and the task ends; if unreasonable, a first-level visible exploration is performed, a second selection of candidate tools is made, and corresponding command plans are generated. At this point, the human is again queried for the reasonableness of the command plan; if reasonable, the task ends as the final result; if unreasonable, a second-level visible exploration is performed, using the explored local area as a new surrounding scene image for command planning again. At this point, the human is again queried for the reasonableness of the command plan; if reasonable, the task ends as the final result; if unreasonable, an invisible exploration is performed, generating invisible areas where the tools needed to complete the human command may exist, for the embodied intelligent robot to further explore, and then the task ends. This application enables the robot to generate corresponding execution steps for any human command in complex and unknown scenarios, and to execute them efficiently and reasonably. This solves the problem that existing technologies struggle to effectively balance the universality of scenario instructions with the executableness of generated execution steps.

[0026] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0027] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 A flowchart illustrating a command planning method based on a large language model and fine-tuning optimization applied to the online command planning stage according to an embodiment of this application;

[0029] Figure 2 This is a schematic diagram of the basic framework of a command planning method according to an embodiment of this application;

[0030] Figure 3 This is a schematic diagram of the execution architecture of a command planning method according to an embodiment of this application;

[0031] Figure 4 This is an example image of a dataset in a GPT-4-involved online command planning framework provided according to an embodiment of this application;

[0032] Figure 5 This is a scene diagram of the surroundings of an embodied intelligent robot according to an embodiment of this application;

[0033] Figure 6 A candidate tool diagram for human reference provided according to one embodiment of this application;

[0034] Figure 7 A candidate tool diagram for inputting GPT4 according to an embodiment of this application;

[0035] Figure 8 This is a schematic diagram of segmentation labels for a selected tool according to an embodiment of this application;

[0036] Figure 9 This is a schematic diagram of the overall flow of a command planning method for unknown scenarios according to an embodiment of this application;

[0037] Figure 10 This application provides a novel candidate tool graph generated from a first-level visible exploration, according to one embodiment of the present application.

[0038] Figure 11 This is a schematic diagram illustrating the results of a complex indoor scene and a second level of visible exploration according to an embodiment of this application;

[0039] Figure 12 This is a schematic diagram illustrating the effect of invisible exploration in a complex indoor scene according to an embodiment of this application;

[0040] Figure 13 A flowchart illustrating a command planning method based on a large language model and fine-tuning optimization applied to the offline command planning stage, according to an embodiment of this application;

[0041] Figure 14 This is an example image of a surrounding environment generated by DALL-E 3 according to an embodiment of this application;

[0042] Figure 15 This is an example image of a real-world photograph of the surrounding environment provided according to an embodiment of this application;

[0043] Figure 16 An example diagram of a command planning device used in the online command planning phase according to an embodiment of this application, which is based on a large language model and fine-tuning optimization.

[0044] Figure 17 An example diagram of a command planning device used in the offline command planning phase according to an embodiment of this application, based on a large language model and fine-tuning optimization.

[0045] Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0046] Among them, 10-a command planning device based on a large language model and fine-tuning optimization for the online command planning stage; 20-a command planning device based on a large language model and fine-tuning optimization for the offline command planning stage; 101-command planning module; 102-first exploration module; 103-second exploration module; 201-retrieval module; 202-first calculation module; 203-second calculation module; 204-offline exploration module; 1801-memory; 1802-processor; 1803-communication interface. Detailed Implementation

[0047] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0048] The following describes a tool usage command planning method based on a large language model and fine-tuning optimization, according to embodiments of this application, with reference to the accompanying drawings. Addressing the problems mentioned in the background art, this application provides a tool usage command planning method based on a large language model and fine-tuning optimization. In this method, a basic command planning process based on prior knowledge acquisition, environmental target detection, and execution step generation is employed. Through human-computer interaction, and combining two interactive visible and invisible exploration strategies, online command planning is performed. When facing unknown scenarios and tasks, important information from human-recognized command plans is stored. When a sufficient number of command plans are stored, when faced with the same human instruction, an offline command planning framework based on RAG (Retrieval-augmented Generation) is used to retrieve results from the stored valid command plans at each stage. This application enables robots to generate corresponding execution steps for any human instruction in complex and unknown scenarios, and to execute them efficiently and reasonably. This solves the problem that existing technologies struggle to effectively balance the universality of scenario instructions with the executability of generated execution steps.

[0049] Specifically, Figure 1 This is a flowchart illustrating a command planning method based on a large language model and fine-tuning optimization, used in the online command planning phase, as provided in an embodiment of this application.

[0050] like Figure 1 As shown, the tool applied to the online command planning phase, based on a large language model and fine-tuning optimization, uses a command planning method that includes the following steps:

[0051] In step S101, the target human instruction is obtained, and based on the preset online data acquisition strategy, an image of the target's surrounding environment is generated. Then, according to the preset large language model, the target human instruction, and the image of the target's surrounding environment, a command planning operation is performed to generate a target candidate tool and a first command plan in the image of the target's surrounding environment. It is then determined whether the first command plan meets the preset reasonable requirements. If the first command plan meets the preset reasonable requirements, then the first command plan is used as the final command plan.

[0052] The command planning strategy in this application mainly includes three processes: prior knowledge acquisition, environmental target detection, and execution step generation. Figure 2 As shown.

[0053] Specifically, embodiments of this application first utilize the knowledge of the large language model GPT-4 to obtain the labels of the tools needed to execute human natural language commands, as well as the appearance properties of the tools. Second, the visual object detection model Grounding DINO and the multimodal GPT-4 are used sequentially. The former takes the tool labels as input, while the latter takes the tool labels and appearance properties as input. This process performs a two-step screening of the surrounding indoor environment scene to accurately detect the tools needed to complete the human command. Furthermore, embodiments of this application can segment and label the detected tool images using Semantic-SAM and then input them into the large language model GPT-4 to output the final command plan for the human command.

[0054] It should be noted that the command planning method of the embodiments of this application designs multimodal prompt words input by a large language model in an effort to improve the large model’s ability to understand external information and the quality of the generated results, thereby enabling a precise operation step with execution-level details to be given to natural language instructions given by humans in complex indoor scenes.

[0055] Optionally, in one embodiment of this application, obtaining the target human instruction and generating an image of the target's surrounding environment based on a preset online data acquisition strategy, and executing a command planning operation according to a preset large language model, the target human instruction, and the image of the target's surrounding environment to generate target candidate tools and a first command plan in the image of the target's surrounding environment, includes: generating target tool labels and target tool appearance properties corresponding to the target human instruction based on the target human instruction and the large language model; randomly selecting an initial image and multiple interference images from a preset dataset according to the target tool labels; randomly stitching the initial image and multiple interference images to obtain an image of the target's surrounding environment; inputting the target tool labels into a preset visual target detection model to generate different candidate tool schematic diagrams; and performing a command planning operation according to the image of the target's surrounding environment and the different candidate tool schematic diagrams. The system obtains the confidence scores of different candidate tools from the visual object detection model; sorts the candidate tool confidence scores in descending order to obtain the ranking results, and selects multiple target candidate tool confidence scores that meet the preset confidence requirements based on the ranking results. It also obtains the ranking labels of multiple target candidate tool confidence scores in the ranking results and the target candidate tools corresponding to these confidence scores; performs background filtering and tool recombination operations on the target candidate tools to obtain a recombined candidate tool map, and inputs the recombined candidate tool map, target tool labels, and target tool appearance properties into a large language model to filter out the target ranking labels corresponding to the target human commands; based on a preset segmentation model, it performs segmentation and filtering operations on the target candidate tools corresponding to the target ranking labels to obtain the corresponding segmentation maps, and inputs the segmentation maps into the large language model to generate the first command plan.

[0056] It should be noted that the detailed implementation framework of the command planning method in this application embodiment is as follows: Figure 3 As shown, the specific execution logic of each functional module in this implementation framework is as follows:

[0057] 1. Prior knowledge acquisition module:

[0058] This module mainly uses the large language model GPT-4 to generate common knowledge based on common sense, such as tool labels and tool appearance properties, to complete human natural language commands without considering the surrounding environment. This knowledge is then used in the subsequent environmental target detection module and the visual perception model's perception of the surrounding indoor environment.

[0059] Specifically, the labels of the tools required to complete the instructions are given directly by GPT-4 to the human instructions. GPT-4 then takes into account the labels of the selected tools and the details of the human instructions to determine the appearance properties that the selected tools should have if they are oriented towards completing human instructions.

[0060] 2. Environmental Target Detection Module:

[0061] In the online command planning framework stage, this application embodiment needs to construct a surrounding scene dataset; specifically, this application embodiment can, for the input human command, manually specify the most suitable tool label to complete the command, and according to the label, randomly select an image from the ImageNet dataset, and then manually randomly select several interference images from ImageNet, and randomly stitch all the selected images into a large image, thereby using the latter as the input simulated surrounding scene.

[0062] For example, for the instruction "I want to chop bones," an example of images used to construct the dataset would be as follows: Figure 4 As shown, the diagram includes actual bone-cutting knives extracted from ImageNet, as well as several manually selected knives and other knives unsuitable for cutting bones as distractors. Since the tools in the diagram are large and clear, this type of scenario helps the "online" framework generate reasonable command plans and store them.

[0063] Therefore, the environmental target detection module of this application embodiment can perform a two-step screening process of coarse and fine screening for the required tools in the surrounding indoor scene images. In actual execution, the embodiment of this application first uses the visual target detection model Grounding DINO to detect and outline candidate tools that may meet the conditions in the surrounding environment based on the tool's label, thereby achieving preliminary screening; secondly, after preprocessing the outlined candidate tools, the candidate tools and the appearance requirements of the tools are input into the multimodal GPT-4, so that it can accurately detect the tools needed to complete human instructions.

[0064] Specifically, in the coarse screening stage, in view of the different needs of humans and GPT-4 for reference judgment and precise selection of tools, the embodiments of this application can generate two candidate tool schematic diagrams based on the detection results of Grounding DINO.

[0065] The candidate tool illustrations for human inspection and reference utilize Grounding DINO to detect the top 9 objects with the highest confidence in the current scene and outline them with blue boxes. The top left corner is marked with a number derived from the candidate tool's confidence ranking (ranging from 1 to 9), thus restoring the positional relationship of the candidate tools in the original image. If GPT-4 subsequently selects a suitable tool (numbered 1 to 9), the corresponding tool can be quickly found in this image, achieving the purpose of human inspection and reference. The candidate tool illustrations used for input to GPT-4, based on the detection and outlining of the top 9 candidate tools with the highest confidence using Grounding DINO, add two steps: background filtering and tool recombination. The former uses the segmentation module in Grounded SAM to filter the background outside the candidate tools to white (i.e., only retaining the tool image within the candidate tool mask area); the latter recombines and arranges the background-filtered candidate tools for GPT-4 observation.

[0066] In the fine-tuning stage, the screening strategy mainly utilizes GPT-4's visual perception capabilities to design prompts, enabling it to select the most suitable tool from the previously generated candidate tool image to complete the human instruction. In the design of the prompts for this question, this embodiment of the application can force GPT-4 to decompose the question regarding tool selection through question template settings, thereby improving selection accuracy.

[0067] Specifically, given the visual information of the surrounding environment, the embodiments of this application set GPT-4 to first give the appearance properties that the tool should have based on the tool label required to complete the instruction, taking into account the details of the human instruction; and then, based on the appearance properties of the tool, select the label of the most suitable tool from the candidate tool diagram.

[0068] It should be noted that in actual implementation, those skilled in the art can choose other suitable network models based on the actual situation, such as CLIP replacing Grounding DINO for environmental target detection, etc., without making specific limitations.

[0069] 3. Execution Step Generation Module:

[0070] This module primarily takes the images to be understood by GPT-4, segments and labels them using an advanced segmentation model, and then inputs them into GPT-4, enabling the latter to generate the results of the execution steps.

[0071] Specifically, in this module, the embodiments of this application first utilize the Semantic-SAM segmentation model, selecting multiple granularities to segment the tool images corresponding to the tool labels selected by GPT-4 in the environment target detection module. Simultaneously, using the tool mask generated by Grounded SAM in the environment target detection module, the segmented tools are filtered again to remove task-irrelevant background segmentation and labels. After generating the tool segmentation map, following the design principles of SoM prompts, the segmentation map is input into GPT-4, enabling the latter to generate command planning based on the segmentation labels of each tool region, tailored to human instructions.

[0072] The command planning and generation process described above will be explained in detail below through a specific embodiment and with reference to the accompanying drawings.

[0073] Assume the human's command is "I want to chop bones," and the surrounding environment of the embodied intelligent robot is as follows: Figure 5 As shown, an example of using the above command planning method to plan human instructions is shown below.

[0074] After the prior knowledge acquisition module, the specific embodiments of this application can obtain the tool label "cleaver" required to complete human instructions through GPT-4, and the tool appearance characteristics are "A cleaver designed for chopping bones should have a thick, heavy blade made of" High-carbon steel, with a broad and tough edge to withstand the impact of cutting through hard materials. Briefly, the cleaver for chopping bones is a robust and heavy-duty knife, characterized by its broad, thick blade optimized for breaking down meat and bones with ease. Thus, GPT-4 correctly describes the "bone cleaver" tool required to complete the bone-cutting command, along with its corresponding appearance and properties.

[0075] In the environmental target detection module, embodiments of this application can generate a tool candidate image for human reference and an input tool candidate image to GPT4 based on tool labels and appearance properties, as shown below. Figure 6 , Figure 7 As shown. By Figure 6 and Figure 7 As can be seen, the former accurately preserves the positional relationships between candidate tools, facilitating human inspection and reference; the latter removes interfering factors other than the candidate tools, making it easier for GPT-4 to select the correct tool. GPT-4 ultimately selected the tool represented by label 3 as the most suitable tool for cutting bone, such as... Figure 7 As shown, since cutting bones requires a knife with a relatively wide and thick blade, it can be seen that GPT-4 selected the most suitable tool, verifying the effectiveness of the command planning method in the embodiments of this application.

[0076] For tool number 3, the region segmentation labeling diagram is as follows: Figure 8As shown, the corresponding tool usage method output is: "To chop bones using this cleaver, the key part of the tool to use is theblade, which is indicated by the area labeled with the numeric ID 1. This part of the cleaver is designed for heavy chopping actions and is capable ofcutting through bone due to its heft and sharp edge. You would need to grip the handle for a secure grasp, which is firmly indicated by the numeric ID 8. It's important to keep your fingers away from the blade area to avoid any potential injuries. Carefully position the blade edge (area 1) against the bone you to chop. Then, using controlled force, bringing the blade down onto the bone in a chopping motion, allowing the weight and sharpness of the cleaver to do most of the work in cutting through the bone. Always exercise caution when using such a tool and ensure that you're using proper techniques to "Prevent accidents." It is evident that for the task of cutting bones, GPT-4 believes that the tool corresponding to label 3 should be used, and the method of use is to hold the handle area represented by 8 and cut the bone with the blade area represented by 1. This provides the correct command plan, thereby further verifying the effectiveness of the command planning method.

[0077] After obtaining the corresponding command plan (i.e., the first command plan), the embodiments of this application need to ask humans whether the command plan is reasonable (i.e., humans judge whether the command plan meets the preset reasonable requirements). If it is reasonable, the command plan is taken as the final result and the task is ended.

[0078] In step S102, if the first command plan does not meet the preset reasonable requirements, a first-level visible exploration operation is performed on the target candidate tool to generate a first local exploration area and a second command plan corresponding to the local exploration area. It is then determined whether the second command plan meets the preset reasonable requirements. If the second command plan meets the preset reasonable requirements, the second command plan is used as the final command plan.

[0079] In addition, if the first command plan is unreasonable, a first-level visible exploration is performed to select candidate tools a second time and generate a corresponding command plan (i.e., the second command plan). At this time, the human command plan is asked again whether it is reasonable. If it is reasonable, it is taken as the final result and the task ends.

[0080] Optionally, in one embodiment of this application, a first-level visible exploration operation is performed on the target candidate tools to generate a first local exploration region and a second command plan corresponding to the local exploration region. This includes: filtering the confidence scores of the top N target candidate tools from multiple target candidate tool confidence scores based on the ranking results, and obtaining the target candidate tools corresponding to the confidence scores of the top N target candidate tools, where N is a positive integer; adding the target candidate tools corresponding to the target ranking index to the target candidate tools corresponding to the confidence scores of the top N target candidate tools to obtain multiple target candidate tools to be filtered; performing concatenation and background filtering operations on the multiple target candidate tools to be filtered to generate a first local exploration region, and inputting the first local exploration region into a large language model to generate a second command plan.

[0081] It should be noted that the basic process of the command planning method described above is primarily based on the assumption of an ideal indoor environment, where the required tools are relatively large and easier to detect and perceive. However, in more general cases, the tools needed to complete human commands may not be near the embodied intelligent robot, but rather at a considerable distance, or there may be no suitable tools within the robot's field of vision. In such cases, the visual perception model used in the command planning method may not be able to effectively detect and select the target tool, leading to the failure of the final command planning.

[0082] To address the aforementioned problem of command planning failure caused by the inefficient detection of imperfect environments by visual perception models, this application utilizes visual perception models such as multimodal GPT-4 and Grounding DINO. Based on the assumption of imperfect conditions such as small tool size and low quality in the surrounding indoor environment, an environmental perception exploration strategy for available tools in the surrounding environment is designed. Furthermore, two types of visual perception exploration strategies are designed for the two cases of whether the required tools in the surrounding environment are directly visible or not: visible exploration and invisible exploration. This achieves a more complete and reasonable extraction of information about the surrounding environment and improves adaptability to unknown environments and unknown conditions.

[0083] Specifically, for non-ideal situations where the tools required for command planning are visible but difficult to detect accurately, this application embodiment designs two interactive visible exploration strategies (first-level visible exploration and second-level visible exploration); for non-ideal situations where the required tools are not directly visible in the current scene, this application embodiment designs an invisible exploration strategy.

[0084] The first-level visible exploration strategy is implemented under the following conditions: although the required tool is visible in the surrounding environment and is relatively large (close to the robot and can be detected by the target detection module Grounding DINO in the first stage), the tool details are difficult to distinguish due to external factors, and the candidate tools are more interfered with due to similar appearances. Therefore, the accuracy of the original command planning method for selecting tools by GPT-4 is reduced.

[0085] To address the aforementioned issues, and based on the assumption in the Grounding DINO algorithm that candidate tools with higher confidence are more likely to be the actual required tools, the core of the first-level visible exploration strategy is as follows: After GPT-4 selects the required tools within the framework of the original command planning method, the selected tools are added to the top N (e.g., top 5) tools with confidence among the 9 tools detected by Grounding DINO. GPT-4 then performs a secondary selection on the aforementioned narrowed-down candidate tools.

[0086] Specifically, when the tool number selected by GPT-4 in the first round is 6 to 9, this exploration method stitches together the selected tool image with the tool images numbered 1 to 5, a total of 6 images, and filters the background to form a candidate tool image, which is then input into GPT-4 for secondary selection. When the tool number selected by GPT-4 in the first round is 1 to 5, this exploration method stitches together the tool images numbered 1 to 5, a total of 5 images, and filters the background to form a candidate tool image, which is then input into GPT-4 for secondary selection.

[0087] Because the first-level visible exploration strategy eliminates tools with low confidence detected by Grounding DINO, it allows GPT-4 to mix previously selected tools with tools with high confidence detected by Grounding DINO within a smaller range for secondary selection, thus removing some interfering tools. Therefore, the use of this first-level visible exploration strategy can improve the accuracy of GPT-4 in selecting the desired tool among 9 candidate tools.

[0088] In step S103, if the second command plan does not meet the preset reasonable requirements, a second-level visible exploration operation is performed on the first local exploration area to obtain the second local exploration area. Then, a command planning operation is performed on the second local exploration area to generate the third command plan corresponding to the second local exploration area. It is determined whether the third command plan meets the preset reasonable requirements. If the third command plan does not meet the preset reasonable requirements, the target embodied intelligent robot is controlled to perform an invisible exploration operation to generate an invisible area of ​​the target. The invisible area of ​​the target is explored by the target embodied intelligent robot. If the third command plan meets the preset reasonable requirements, the third command plan is used as the final command plan. The planning information corresponding to the final command plan is obtained and stored so as to use the planning information to perform the offline command planning generation operation in the offline command planning stage.

[0089] Furthermore, if the second command planning does not meet the preset reasonable requirements, a second-level visible exploration is performed, such as... Figure 9 As shown, the explored local area (i.e., the first local exploration area) is used as a new surrounding scene image to re-plan the command, resulting in the third command plan. At this point, the human is asked again whether the command plan is reasonable. If it is reasonable, the task ends as the final result. If it is unreasonable, invisible exploration is carried out to generate invisible areas where the tools required to complete the human command may exist, for the embodied intelligent robot to explore further, and the task ends.

[0090] Therefore, based on the basic process of the command planning method described above, the embodiments of this application combine the human-computer interaction with the environmental exploration strategy in the embodiments of this application, so that the command planning method designed based on the assumption of an ideal indoor environment can be widely applied to other non-ideal, complex and changeable indoor scenarios, thereby achieving the predetermined goals of the method design.

[0091] Optionally, in one embodiment of this application, a second-level visible exploration operation is performed on the first local exploration region to obtain a second local exploration region, and a command planning operation is performed on the second local exploration region to generate a third command plan corresponding to the second local exploration region. This includes: filtering multiple candidate tool confidence scores corresponding to target surrounding environment images that meet preset sorting number requirements from the sorting results, and obtaining multiple target candidate tools corresponding to the multiple candidate tool confidence scores; weighting the target to be detected in the target region of each target candidate tool among the multiple target candidate tools to obtain a weighted accumulation result corresponding to each target candidate tool, and comparing the weighted accumulation results of each target candidate tool to obtain the maximum value of the weighted accumulation result; obtaining the target candidate tool corresponding to the maximum value of the weighted accumulation result, and using the target region of the target candidate tool corresponding to the maximum value of the weighted accumulation result as the second local exploration region; and inputting the second local exploration region into a large language model to generate a third command plan.

[0092] In actual implementation, the second-level visibility exploration strategy of this application embodiment mainly targets the following situation: the required tool is visible in the surrounding environment but is small in size (far from the robot). In this case, the required tool cannot be detected by Grounding DINO, causing GPT-4 to fail to select the correct tool, thus causing the command planning task to fail. In this situation, the tool required to complete the human command is more likely to be located in an area outside the top 9 confidence levels, but still near a position with high confidence among all targets detected by Grounding DINO.

[0093] Based on the above, the core of the second-level visible exploration strategy is as follows: For candidate tools ranked 10th to 18th in confidence in the surrounding environment original image by Grounding DINO, the detected targets in the surrounding area of ​​each of the above 9 tools are weighted and accumulated (the higher the confidence of the surrounding detected targets, the more weights are accumulated) and compared, so that the surrounding area of ​​the candidate tool with the highest weight is used as the area to be detected by the visual perception module in the next step, thus realizing the second-level visible exploration.

[0094] Specifically, the second-level visible exploration strategy focuses on detecting target objects ranked 10-40 in the detection results of Grounding DINO in the original image of the surrounding environment. This exploration strategy traverses nine square regions (square regions with a radius of 250 pixels centered on the target object) around the target objects ranked 10-18. For each square region, it counts all target objects ranked 10-40 that have a containment or intersection relationship with it, and accumulates the weight of each object (accumulated weight is 40 minus the confidence ranking). This yields nine accumulated weights. The nine accumulated weights are then sorted, and the square region corresponding to the largest accumulated weight is selected as the region for further exploration, thus completing the goal of the second-level exploration.

[0095] Furthermore, for non-ideal situations where the required tool is not directly visible in the current scene (in which case neither directly using Grounding DINO to detect the original scene map nor using a visible exploration strategy can detect and select the required tool), an invisible exploration strategy was designed. Its goal is to identify the areas where the required tool may exist in the current scene, so that the embodied intelligent robot can explore these areas later. The core of the strategy is to utilize the rich common sense knowledge of GPT-4, which, based on the existing information, provides labels for the areas where the required tool may be located. The visual target detection module then detects the actual area corresponding to the label in the current scene, which is the area that the robot is expected to explore further.

[0096] Therefore, the embodiments of this application can autonomously initiate interactive active exploration when there is no clearly selectable tool in the field of vision or no selectable tool exists, and provide areas or locations where target tools may exist, which facilitates subsequent tool selection and task execution.

[0097] The following describes the execution effect of the environment exploration strategy designed in this application through a specific embodiment and in conjunction with the accompanying drawings.

[0098] In one specific embodiment of this application, for Figure 5 The scene and the bone-cutting command, the results of the first level of visible exploration are as follows: Figure 10 As shown, the first level of visible exploration achieved the desired effect, narrowing down the range of candidate tools through reasonable selection; experiments have also shown that GPT-4 can be used in... Figure 10 Select the tool corresponding to label 3 correctly.

[0099] Assuming the embodied intelligent robot is located in a more complex indoor environment, such as Figure 11As shown, if the human command is still set to "I want to chop bones," then because the actual required bone-chopping knife is too small, Grounding DINO cannot detect the necessary tool. Therefore, the correct command plan cannot be generated using the command planning strategy and the first-level visible exploration. The result of performing a second-level visible exploration in this scene is as follows: Figure 11 As shown in the blue box, the area contains the bone-cutting knife needed for the bone-cutting task, thus verifying the rationality of the second-level visible exploration strategy logic.

[0100] exist Figure 11 In the complex indoor scene shown, assuming the above command planning process and the first and second level visible exploration, the selected tools still do not meet human requirements, then in Figure 11 The results of invisible exploration in the scene are as follows Figure 12 As shown. Therefore, it can be seen that, based on the premise that the tools required to complete the instructions are not directly visible, Figure 12 The area where the bone-cutting knife needed to cut the bone might be located (i.e., in the drawer) was reasonably defined, achieving the expected result of the invisible exploration strategy design.

[0101] The following specific embodiment demonstrates the experimental comparison and illustration of the execution effect of the designed command planning method when there are interfering tools in the surrounding scene that are highly similar to the required tool but cannot complete the task or are unsuitable for completing the task. The method can accurately eliminate interference, select the required tool, and perform command planning.

[0102] Experiments show that the command planning method designed in the specific embodiments of this application can select the truly suitable tool from numerous interfering tools and perform corresponding command planning (such as selecting a bone-cutting knife from several different types of knives to cut bones, or selecting a slotted spoon from several ladles to filter, etc.). Existing object detection methods, as well as embodied intelligence models such as SayCan, PaLM-E, and TaPA, perform poorly in this type of task. For example, assuming the given human instruction is "I want to chopbones," for a given command like... Figure 5 In scenarios where not only the bone-cutting knife we ​​actually need exists, but also many other types of knives belonging to the same category, but not suitable for cutting bones, there are also several other types of knives. In this case, the command planning method designed in the embodiments of this application can select... Figure 6The bone-cutting knife (labeled 3) is used for command planning, and interference is eliminated. Table 1 shows the command planning accuracy of the command planning method in this embodiment compared to the embodied intelligence model TaPA, corresponding to four human commands: "I want to chop bones," "I want to take cooked food out of the pot as well as filter out the liquid," "I want to flip the pancake in the skillet," and "I want to fry a pancake." The two command planning methods were tested on the corresponding constructed ImageNet stitched datasets. Table 1 shows that the command planning method designed in this embodiment is better than or equal to TaPA for all four commands. For command 2, TaPA's command planning accuracy is extremely low, while this embodiment achieves a relatively high level.

[0103] Table 1

[0104]

[0105]

[0106] Since selecting tools from several similar tools to complete human instructions is a common task in indoor settings such as kitchens, the embodiments of this application can effectively improve the overall effect and versatility of command planning for embodied intelligent robots.

[0107] In summary, the embodiments of this application can provide the large language model with more and more detailed information about the surrounding environment and objects, enabling it to automatically plan and provide a precise and executable subtask at the execution level in response to high-level natural language commands given by humans in complex indoor scenes, and finally provide visual guidance for the execution of the executable method and operation area, so that the given operation steps can be truly executed by the robot at the physical level.

[0108] According to the embodiments of this application, a command planning method based on a large language model and fine-tuning optimization is used in the online command planning stage. After inputting human commands and images of the surrounding environment, the method first goes through its basic process to generate corresponding command plans. The human is then queried about the rationality of the command plan. If rational, the command plan is used as the final result, and the task ends. If not rational, a first-level visible exploration is performed, where candidate tools are selected a second time, and corresponding command plans are generated. The human is then queried again about the rationality of the command plan. If rational, the task ends as the final result. If not rational, a second-level visible exploration is performed, where the explored local area is used as a new image of the surrounding scene for command planning. The human is then queried again about the rationality of the command plan. If rational, the task ends as the final result. If not rational, an invisible exploration is performed, generating invisible areas where the tools needed to complete the human command might exist, for the embodied intelligent robot to explore further, and then the task ends. This application enables the robot to generate corresponding execution steps for any human command in complex and unknown scenarios, and to execute them efficiently and rationally.

[0109] Figure 13 This is a flowchart illustrating a tool-based command planning method for offline command planning, which is applied to the offline command planning stage and is based on a large language model and fine-tuning optimization, as provided in an embodiment of this application.

[0110] like Figure 13 As shown, the tool applied to the offline command planning phase, based on a large language model and fine-tuning optimization, uses a command planning method that includes the following steps:

[0111] In step S1301, based on the preset offline data acquisition strategy and retrieval enhancement generation strategy, the pre-stored planning information is obtained, and the target tool image in the planning information is retrieved. The first similarity between the target tool image and multiple preset target candidate tools is calculated, and the optimal target candidate tool with the highest first similarity is obtained.

[0112] In step S1302, the retrieval enhancement generation strategy is used to retrieve the target operation area image corresponding to the optimal target candidate tool in the planning information, and the second similarity between the target operation area image and a plurality of preset candidate areas is calculated respectively. At least one target operation area image whose second similarity meets the preset similarity requirement is selected, so that the retrieval enhancement generation strategy retrieves the corresponding first command planning information in the planning information based on at least one target operation area image, and determines whether the first command planning information meets the preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, the first command planning information is used as the final command planning.

[0113] In step S1303, if the first command planning information does not meet the preset reasonable requirements, the key area image corresponding to the optimal target candidate tool in the planning information is retrieved by the retrieval enhancement generation strategy, and the third similarity between the key area image and the multiple key area images corresponding to the multiple target candidate tools is calculated respectively. Based on the target candidate tool corresponding to the target key area image with the highest third similarity, the corresponding second command planning information is generated.

[0114] In step S1304, based on the second command planning information and preset reasonable requirements, preset second-level visible exploration and / or invisible exploration operations are performed to generate the final command plan.

[0115] Since GPT-4 is involved in many aspects of the aforementioned command planning framework, such as candidate tool selection and command planning generation, although the aforementioned command planning method successfully utilizes GPT-4's rich knowledge to improve the accuracy of command planning, GPT-4 requires setting up an agent and calling APIs online, which results in the original command planning method having a relatively long running time and being relatively affected by network speed.

[0116] To meet the needs of offline operation and physical deployment, in addition to the online command planning framework with GPT-4 involved as described above, this application also designs an offline command planning framework based on RAG. The basic idea is as follows: when facing an unknown scenario or unknown task, the online command planning framework (including the environment exploration stage) described above is used first to store important information of the command planning recognized by humans (such as appropriate tool tags, images, usage methods, etc. corresponding to human instructions); when there are enough stored command plannings, when facing the same human instruction, the "offline" command planning framework based on RAG retrieval is used again. At each stage, the results retrieved by RAG from the valid information of the correct command planning already stored are used to replace the results generated by GPT-4, so as to shorten the running time and facilitate physical deployment.

[0117] Specifically, the overall framework of the RAG-based offline command planning method designed in the embodiments of this application is similar to... Figure 2 , Figure 9 The online command planning frameworks they represent are basically the same, all including a prior knowledge acquisition module, an environment target detection module, an execution step generation module, and a subsequent three-level environment exploration stage. The difference lies in the specific implementation methods of each major module and stage.

[0118] For example, in the prior knowledge acquisition module, the offline framework directly retrieves key information such as the required tool tags through RAG retrieval instead of generating them using GPT-4. During the environmental target detection module, the offline framework retrieves images of suitable tools from the stored database using RAG, calculates their similarity with candidate tools, and selects the tool with the highest similarity as the most suitable tool, replacing the GPT-4 selection process. In the execution step generation module, the offline framework similarly retrieves images of the appropriate tool's operation area and calculates their similarity with candidate areas to select several key areas of the tool (such as the handle and blade of a knife), and generates corresponding command plans according to the retrieved command planning method, replacing the GPT-4 command planning generation process. For the environmental exploration phase, since GPT-4's involvement in the second-level visible and invisible exploration is minimal and non-critical, both exploration methods largely follow the design of the online framework.

[0119] For the first level of visible exploration, the offline framework uses a similarity comparison method for the key functional areas of the tools (such as the blade of a knife or the head of a slotted spoon). The key areas of the stored suitable tools are compared with the key areas of the candidate tools. The candidate tool corresponding to the key area with the highest similarity is selected as the suitable tool, which replaces the secondary selection of tools by GPT-4 in the "online" framework.

[0120] Furthermore, the command planning method in this embodiment is designed to first use an online framework involving GPT-4 to store key information for correct and reasonable command planning for unknown tasks; then, it uses an offline framework centered on RAG retrieval to generate command plans for "seen" tasks through retrieval and comparison. As described above, the online framework stage stores and prepares correct command plans, similar to a training process; while the offline framework stage, for tasks already seen, uses the stored command planning results in unknown scenarios to perform faster and more accurate command planning, similar to a testing process. During the training process in the online framework stage, the requirements for the surrounding scene are: the candidate tool should be as large and clear as possible, without requiring complexity of the scene; while during the testing process in the offline framework stage, the requirements for the surrounding scene are: it should reflect the real indoor scene as much as possible.

[0121] Since the requirements for scene images vary at different stages of the designed command planning method, the construction methods of the dataset also differ. The construction method of the surrounding scene dataset for the online command planning framework stage has been explained above. The construction method of the dataset for the offline command planning framework stage will be explained below.

[0122] During the offline command planning framework phase, the dataset was constructed using two methods: generation via DALL-E 3 and actual camera capture. An example of an indoor scene image generated via DALL-E 3 is shown below. Figure 14 As shown, by Figure 14 It can be seen that, after inputting appropriate prompts, the indoor scene images generated by DALL-E 3 not only maintain fidelity but also include common indoor items such as cups, chairs, sofas, and table lamps, meeting the needs of offline framework testing; indoor scenes obtained through actual camera photography, such as... Figure 15 As shown, this is a realistic indoor scene that meets the requirements for offline framework testing.

[0123] Therefore, compared with existing embodied intelligence work involving large models, the command planning offline framework in this application embodiment significantly reduces the running time, improves the execution accuracy, forms a sparse few-sample database to achieve generalization on multiple tasks, and is convenient for offline deployment on real machines.

[0124] The tool-based command planning method for offline command planning proposed in this application, which is based on a large language model and fine-tuning optimization, first passes through an online command planning framework (including an environment exploration phase) when facing unknown scenarios and tasks. This framework stores important information about human-recognized command plans (such as appropriate tool tags, images, and usage methods corresponding to human instructions). When enough command plans are stored, when faced with the same human instruction, the method then passes through an offline command planning framework based on RAG retrieval. At each stage, the results retrieved from the valid information of the stored correct command plans using RAG are used to replace the results generated by GPT-4, thereby shortening the running time and facilitating deployment on real machines.

[0125] Secondly, referring to the accompanying drawings, a tool-based command planning device based on a large language model and fine-tuning optimization, according to an embodiment of this application, is described.

[0126] Figure 16 This is a block diagram of a tool-based command planning device for online command planning, which is based on a large language model and fine-tuning optimization and is applied to the online command planning stage according to an embodiment of this application.

[0127] like Figure 16 As shown, the tool based on a large language model and fine-tuning optimization applied to the online command planning stage uses a command planning device 10, which includes: a command planning module 101, a first exploration module 102, and a second exploration module 103.

[0128] The command planning module 101 is used to acquire the target human instructions, generate images of the target's surrounding environment based on a preset online data acquisition strategy, and perform command planning operations according to a preset large language model, the target human instructions, and the images of the target's surrounding environment to generate target candidate tools and a first command plan in the images of the target's surrounding environment. It also determines whether the first command plan meets preset reasonable requirements. If the first command plan meets preset reasonable requirements, it is used as the final command plan.

[0129] The first exploration module 102 is used to perform a first-level visible exploration operation on the target candidate tool if the first command plan does not meet the preset reasonable requirements, so as to generate a first local exploration area and a second command plan corresponding to the local exploration area, and determine whether the second command plan meets the preset reasonable requirements. If the second command plan meets the preset reasonable requirements, the second command plan is used as the final command plan.

[0130] The second exploration module 103 is used to perform a second-level visible exploration operation on the first local exploration area to obtain a second local exploration area if the second command plan does not meet the preset reasonable requirements, and to perform a command planning operation on the second local exploration area to generate a third command plan corresponding to the second local exploration area. It also determines whether the third command plan meets the preset reasonable requirements. If the third command plan does not meet the preset reasonable requirements, it controls the target embodied intelligent robot to perform an invisible exploration operation to generate an invisible area of ​​the target, and explores the invisible area of ​​the target through the target embodied intelligent robot. If the third command plan meets the preset reasonable requirements, it uses the third command plan as the final command plan, and acquires and stores the planning information corresponding to the final command plan, so as to use the planning information to perform an offline command planning generation operation in the offline command planning stage.

[0131] Optionally, in one embodiment of this application, the command planning module 101 includes: an online data acquisition unit, a first acquisition unit, a coarse screening unit, a fine screening unit, and a segmentation unit.

[0132] The online data acquisition unit is used to generate target tool labels and target tool appearance properties corresponding to target human instructions based on the target human instructions and the large language model. It also randomly selects an initial image and multiple interference images from a preset dataset based on the target tool labels, and randomly stitches the initial image and multiple interference images together to obtain an image of the target's surrounding environment.

[0133] The first acquisition unit is used to input the target tool label into a preset visual target detection model to generate different candidate tool schematic diagrams, and to obtain the candidate tool confidence scores corresponding to different candidate tools based on the target surrounding environment image, different candidate tool schematic diagrams and the visual target detection model.

[0134] The coarse screening unit is used to sort the confidence scores of candidate tools in descending order to obtain the sorting results, and to filter out multiple target candidate tool confidence scores that meet the preset confidence requirements based on the sorting results. It also obtains the sorting index of the multiple target candidate tool confidence scores in the sorting results and the target candidate tools corresponding to the multiple target candidate tool confidence scores.

[0135] The fine-tuning unit is used to perform background filtering and tool reorganization operations on the target candidate tools to obtain a reorganized candidate tool map. The reorganized candidate tool map, target tool labels, and target tool appearance properties are then input into the large language model to filter out the target sorting number corresponding to the target human command.

[0136] The segmentation unit is used to segment and filter the target candidate tools corresponding to the target ranking index based on the preset segmentation model to obtain the corresponding segmentation map, and input the segmentation map into the large language model to generate the first command plan.

[0137] Optionally, in one embodiment of this application, the first exploration module 102 includes: a second acquisition unit, an addition unit, and a filtering unit.

[0138] The second acquisition unit is used to filter the confidence scores of the top N target candidate tools from the multiple target candidate tool confidence scores according to the sorting results, and to acquire the target candidate tools corresponding to the confidence scores of the top N target candidate tools, where N is a positive integer.

[0139] The addition unit is used to add the target candidate tool corresponding to the target sorting index to the target candidate tool corresponding to the confidence level of the first N target candidate tools, so as to obtain multiple target candidate tools to be screened.

[0140] The filtering unit is used to perform splicing and background filtering operations on multiple candidate tools to be screened to generate a first local exploration region, and input the first local exploration region into the large language model to generate a second command plan.

[0141] Optionally, in one embodiment of this application, the second exploration module 103 includes: a third acquisition unit, a weight accumulation unit, a fourth acquisition unit, and a generation unit.

[0142] The third acquisition unit is used to filter out multiple candidate tool confidence scores corresponding to target surrounding environment images that meet the preset sorting number requirements from the sorting results, and to acquire multiple target candidate tools corresponding to the multiple candidate tool confidence scores.

[0143] The weight accumulation unit is used to accumulate the weights of the target objects to be detected in the target region of each of the multiple target candidate tools to obtain the weight accumulation result corresponding to each target candidate tool, and compare the weight accumulation results of each target candidate tool to obtain the maximum value of the weight accumulation result.

[0144] The fourth acquisition unit is used to acquire the target candidate tool corresponding to the maximum value of the weight accumulation result, and to use the target region of the target candidate tool corresponding to the maximum value of the weight accumulation result as the second local exploration region.

[0145] The generation unit is used to input the second local exploration region into the large language model to generate the third command plan.

[0146] It should be noted that the foregoing explanation of the tool usage command planning method embodiment based on large language model and fine-tuning optimization applied to the online command planning stage also applies to the tool usage command planning device based on large language model and fine-tuning optimization applied to the online command planning stage, and will not be repeated here.

[0147] According to the embodiments of this application, a command planning device based on a large language model and fine-tuning optimization is proposed for use in the online command planning stage. After inputting human commands and images of the surrounding environment, the device first goes through the basic command planning process to generate a corresponding command plan. The human is then asked whether the command plan is reasonable. If reasonable, the command plan is used as the final result and the task ends. If unreasonable, a first-level visible exploration is performed, a second selection of candidate tools is made, and a corresponding command plan is generated. The human is then asked again whether the command plan is reasonable. If reasonable, the task ends as the final result. If unreasonable, a second-level visible exploration is performed, using the explored local area as a new surrounding scene image for command planning. The human is asked again whether the command plan is reasonable. If reasonable, the task ends as the final result. If unreasonable, an invisible exploration is performed, generating invisible areas where the tools needed to complete the human command may exist, for the embodied intelligent robot to explore further, and then the task ends. This application enables the robot to generate corresponding execution steps for any human command in complex and unknown scenarios, and to execute them efficiently and reasonably.

[0148] Figure 17 This is a block diagram of a tool-based command planning device for offline command planning, which is based on a large language model and fine-tuning optimization and is applied to the offline command planning stage according to an embodiment of this application.

[0149] like Figure 17 As shown, the command planning device 20, which is based on a large language model and fine-tuning optimization and is applied to the offline command planning stage, includes: a retrieval module 201, a first calculation module 202, a second calculation module 203, and an offline exploration module 204.

[0150] The retrieval module 201 is used to obtain pre-stored planning information based on a preset offline data collection strategy and a retrieval enhancement generation strategy, retrieve target tool images from the planning information, calculate the first similarity between the target tool images and multiple preset target candidate tools, and obtain the optimal target candidate tool with the highest first similarity.

[0151] The first calculation module 202 is used to retrieve the target operation area image corresponding to the optimal target candidate tool in the planning information using the retrieval enhancement generation strategy, and calculate the second similarity between the target operation area image and a plurality of preset candidate areas, and filter out at least one target operation area image whose second similarity meets the preset similarity requirements, so that the retrieval enhancement generation strategy retrieves the corresponding first command planning information in the planning information based on at least one target operation area image, and determines whether the first command planning information meets the preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, the first command planning information is used as the final command planning.

[0152] The second calculation module 203 is used to retrieve the key area image corresponding to the optimal target candidate tool in the planning information through a retrieval enhancement generation strategy if the first command planning information does not meet the preset reasonable requirements, and calculate the third similarity between the key area image and the multiple key area images corresponding to the multiple target candidate tools respectively, and generate the corresponding second command planning information based on the target candidate tool corresponding to the target key area image with the highest third similarity.

[0153] The offline exploration module 204 is used to perform preset second-level visible and / or invisible exploration operations based on the second command planning information and preset reasonable requirements, so as to generate the final command plan.

[0154] It should be noted that the foregoing explanation of the tool usage command planning method embodiment based on large language model and fine-tuning optimization applied to the offline command planning stage also applies to the tool usage command planning device based on large language model and fine-tuning optimization applied to the offline command planning stage embodiment, and will not be repeated here.

[0155] According to the embodiments of this application, a tool-based command planning device for offline command planning, which is based on a large language model and fine-tuning optimization, first passes through an online command planning framework (including an environment exploration phase) when facing unknown scenarios and tasks. This framework stores important information of human-recognized command plans (such as appropriate tool tags, images, and usage methods corresponding to human instructions). When enough command plans are stored, when faced with the same human instruction, the device then passes through an offline command planning framework based on RAG retrieval. At each stage, the results retrieved from the valid information of the stored correct command plans using RAG replace the results generated by GPT-4, thereby shortening the running time and facilitating deployment on real machines.

[0156] Figure 18 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0157] The memory 1801, the processor 1802, and the computer program stored on the memory 1801 and executable on the processor 1802.

[0158] When the processor 1802 executes the program, it implements the command planning method based on a large language model and fine-tuning optimization provided in the above embodiments.

[0159] Furthermore, electronic devices also include:

[0160] Communication interface 1803 is used for communication between memory 1801 and processor 1802.

[0161] Memory 1801 is used to store computer programs that can run on processor 1802.

[0162] The memory 1801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0163] If the memory 1801, processor 1802, and communication interface 1803 are implemented independently, then the communication interface 1803, memory 1801, and processor 1802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 18The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0164] Optionally, in a specific implementation, if the memory 1801, processor 1802, and communication interface 1803 are integrated on a single chip, then the memory 1801, processor 1802, and communication interface 1803 can communicate with each other through an internal interface.

[0165] The processor 1802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0166] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described tool usage command planning method based on a large language model and fine-tuning optimization.

[0167] This application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-described tool usage command planning method based on a large language model and fine-tuning optimization.

[0168] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0169] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0170] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0171] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0172] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0173] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0175] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A tool based on a large language model and fine-tuning optimization uses a command planning method, applied to the online command planning stage, characterized in that... Includes the following steps: The system acquires the target human command and generates an image of the target's surrounding environment based on a preset online data acquisition strategy. It then executes a command planning operation based on a preset large language model, the target human command, and the image of the target's surrounding environment to generate a target candidate tool and a first command plan in the image of the target's surrounding environment. The system then determines whether the first command plan meets preset reasonable requirements. If the first command plan meets the preset reasonable requirements, the first command plan is used as the final command plan. If the first command plan does not meet the preset reasonable requirements, a first-level visible exploration operation is performed on the target candidate tool to generate a first local exploration area and a second command plan corresponding to the local exploration area. It is then determined whether the second command plan meets the preset reasonable requirements. If the second command plan meets the preset reasonable requirements, the second command plan is used as the final command plan. If the second command plan does not meet the preset reasonable requirements, a second-level visible exploration operation is performed on the first local exploration area to obtain a second local exploration area. A command planning operation is then performed on the second local exploration area to generate a third command plan corresponding to the second local exploration area. It is determined whether the third command plan meets the preset reasonable requirements. If the third command plan does not meet the preset reasonable requirements, the target embodied intelligent robot is controlled to perform an invisible exploration operation to generate an invisible target area. The invisible target area is then explored by the target embodied intelligent robot. If the third command plan meets the preset reasonable requirements, the third command plan is used as the final command plan. The planning information corresponding to the final command plan is acquired and stored so that the planning information can be used to perform an offline command plan generation operation in the offline command planning stage. The process of acquiring target human commands, generating images of the target's surrounding environment based on a preset online data acquisition strategy, and executing command planning operations according to a preset large language model, the target human commands, and the images of the target's surrounding environment to generate target candidate tools and a first command plan from the images of the target's surrounding environment includes: Based on the target human instruction and the large language model, target tool labels and target tool appearance properties corresponding to the target human instruction are generated. Then, based on the target tool label, an initial image and multiple interference images are randomly selected from a preset dataset. The initial image and multiple interference images are then randomly spliced ​​together to obtain an image of the target's surrounding environment. The target tool label is input into a preset visual target detection model to generate different candidate tool schematic diagrams, and the candidate tool confidence scores corresponding to different candidate tools are obtained based on the target surrounding environment image, the different candidate tool schematic diagrams and the visual target detection model. The confidence scores of the candidate tools are sorted in descending order to obtain a sorting result. Based on the sorting result, multiple target candidate tool confidence scores that meet the preset confidence requirements are selected. The sorting index of the multiple target candidate tool confidence scores in the sorting result and the target candidate tools corresponding to the multiple target candidate tool confidence scores are obtained. Background filtering and tool reorganization operations are performed on the target candidate tools to obtain a reorganized candidate tool map. The reorganized candidate tool map, the target tool label, and the appearance properties of the target tool are then input into the large language model to filter out the target ranking number corresponding to the target human command. Based on a preset segmentation model, the target candidate tools corresponding to the target ranking index are segmented and filtered to obtain the corresponding segmentation map, and the segmentation map is input into the large language model to generate the first command plan; The step of performing a first-level visible exploration operation on the target candidate tool to generate a first local exploration region and a second command plan corresponding to the local exploration region includes: Based on the sorting results, filter the confidence scores of the top N target candidate tools from the multiple target candidate tool confidence scores, and obtain the target candidate tools corresponding to the confidence scores of the top N target candidate tools, where N is a positive integer; Add the target candidate tool corresponding to the target ranking index to the target candidate tool corresponding to the confidence level of the first N target candidate tools to obtain multiple target candidate tools to be screened; The multiple candidate tools to be screened are spliced ​​and background filtered to generate the first local exploration region, and the first local exploration region is input into the large language model to generate the second command plan; The step of performing a second-level visible exploration operation on the first local exploration region to obtain a second local exploration region, and performing a command planning operation on the second local exploration region to generate a third command plan corresponding to the second local exploration region, includes: From the sorting results, select multiple candidate tool confidence scores corresponding to the target's surrounding environment images that meet the preset sorting number requirements, and obtain multiple target candidate tools corresponding to the multiple candidate tool confidence scores; The target regions of each of the multiple target candidate tools are weighted and accumulated to obtain the weight accumulation result corresponding to each target candidate tool. The weight accumulation results of each target candidate tool are compared to obtain the maximum value of the weight accumulation result. Obtain the target candidate tool corresponding to the maximum value of the weight accumulation result, and use the target region of the target candidate tool corresponding to the maximum value of the weight accumulation result as the second local exploration region; The second local exploration region is input into the large language model to generate the third command plan.

2. A tool usage command planning method based on a large language model and fine-tuning optimization, employing the tool usage command planning method based on a large language model and fine-tuning optimization as described in claim 1, applied to the offline command planning stage, characterized in that... Includes the following steps: Based on a preset offline data acquisition strategy and a retrieval enhancement generation strategy, pre-stored planning information is obtained, and target tool images in the planning information are retrieved. The first similarity between the target tool images and a plurality of preset target candidate tools is calculated, and the optimal target candidate tool with the highest first similarity is obtained. The search enhancement generation strategy retrieves the target operation area image corresponding to the optimal target candidate tool in the planning information, calculates the second similarity between the target operation area image and a plurality of preset candidate areas, and filters out at least one target operation area image whose second similarity meets the preset similarity requirements. This allows the search enhancement generation strategy to retrieve the corresponding first command planning information in the planning information based on the at least one target operation area image, and determines whether the first command planning information meets the preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, then the first command planning information is used as the final command planning. If the first command planning information does not meet the preset reasonable requirements, the key area image corresponding to the optimal target candidate tool in the planning information is retrieved through the retrieval enhancement generation strategy, and the third similarity between the key area image and the multiple key area images corresponding to the multiple target candidate tools is calculated respectively. Based on the target candidate tool corresponding to the target key area image with the highest third similarity, the corresponding second command planning information is generated. Based on the second command planning information and the preset reasonable requirements, preset second-level visible exploration and / or invisible exploration operations are performed to generate the final command plan.

3. A tool usage command planning device based on a large language model and fine-tuning optimization, used to implement the tool usage command planning method based on a large language model and fine-tuning optimization applied to the online command planning stage as described in claim 1, characterized in that, include: The command planning module is used to acquire the target human command, generate an image of the target's surrounding environment based on a preset online data acquisition strategy, and perform command planning operations according to a preset large language model, the target human command, and the image of the target's surrounding environment to generate a target candidate tool and a first command plan in the image of the target's surrounding environment. The module also determines whether the first command plan meets preset reasonable requirements. If the first command plan meets the preset reasonable requirements, the first command plan is used as the final command plan. The first exploration module is used to perform a first-level visible exploration operation on the target candidate tool if the first command plan does not meet the preset reasonable requirements, so as to generate a first local exploration area and a second command plan corresponding to the local exploration area, and determine whether the second command plan meets the preset reasonable requirements. If the second command plan meets the preset reasonable requirements, the second command plan is used as the final command plan. The second exploration module is used to perform a second-level visible exploration operation on the first local exploration area to obtain a second local exploration area if the second command plan does not meet the preset reasonable requirements, and to perform a command planning operation on the second local exploration area to generate a third command plan corresponding to the second local exploration area. It also determines whether the third command plan meets the preset reasonable requirements. If the third command plan does not meet the preset reasonable requirements, it controls the target embodied intelligent robot to perform an invisible exploration operation to generate an invisible target area. The target embodied intelligent robot then explores the invisible target area. If the third command plan meets the preset reasonable requirements, it uses the third command plan as the final command plan and acquires and stores the planning information corresponding to the final command plan to use the planning information to perform an offline command plan generation operation during the offline command planning stage. The command planning module includes: An online data acquisition unit is used to generate target tool labels and target tool appearance properties corresponding to the target human instructions based on the target human instructions and the large language model, and to randomly select an initial image and multiple interference images from a preset dataset according to the target tool labels, and to randomly stitch the initial image and multiple interference images together to obtain an image of the target's surrounding environment. The first acquisition unit is used to input the target tool label into a preset visual target detection model to generate different candidate tool schematic diagrams, and to obtain the candidate tool confidence scores corresponding to different candidate tools based on the target surrounding environment image, the different candidate tool schematic diagrams and the visual target detection model; The coarse screening unit is used to sort the confidence scores of the candidate tools in descending order to obtain a sorting result, and to filter out multiple target candidate tool confidence scores that meet the preset confidence score requirements based on the sorting result, and to obtain the sorting index of the multiple target candidate tool confidence scores in the sorting result and the target candidate tool corresponding to the multiple target candidate tool confidence scores; The fine-screening unit is used to perform background filtering and tool recombination operations on the target candidate tools to obtain a recombined candidate tool image, and input the recombined candidate tool image, the target tool label, and the appearance properties of the target tool into the large language model to filter out the target sorting number corresponding to the target human command; The segmentation unit is used to perform segmentation and filtering operations on the target candidate tools corresponding to the target ranking index based on a preset segmentation model to obtain the corresponding segmentation map, and input the segmentation map into the large language model to generate the first command plan.

4. A tool usage command planning device based on a large language model and fine-tuning optimization, used to implement the tool usage command planning method based on a large language model and fine-tuning optimization applied to the offline command planning stage as described in claim 2, characterized in that, include: The retrieval module is used to obtain pre-stored planning information based on a preset offline data collection strategy and a retrieval enhancement generation strategy, retrieve target tool images from the planning information, calculate the first similarity between the target tool images and a plurality of preset target candidate tools, and obtain the optimal target candidate tool with the highest first similarity. The first calculation module is used to retrieve the target operation area image corresponding to the optimal target candidate tool in the planning information using the retrieval enhancement generation strategy, and calculate the second similarity between the target operation area image and a plurality of preset candidate areas, and filter out at least one target operation area image whose second similarity meets the preset similarity requirements, so that the retrieval enhancement generation strategy retrieves the corresponding first command planning information in the planning information based on the at least one target operation area image, and determines whether the first command planning information meets the preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, the first command planning information is used as the final command planning. The second calculation module is used to retrieve the key area image corresponding to the optimal target candidate tool in the planning information through the retrieval enhancement generation strategy if the first command planning information does not meet the preset reasonable requirements, and calculate the third similarity between the key area image and the multiple key area images corresponding to the multiple target candidate tools respectively, and generate the corresponding second command planning information based on the target candidate tool corresponding to the target key area image with the highest third similarity. The offline exploration module is used to perform preset second-level visible exploration and / or invisible exploration operations based on the second command planning information and the preset reasonable requirements, so as to generate the final command plan.

5. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the tool use command planning method based on a large language model and fine-tuning optimization as described in claim 1 or claim 2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the tool use command planning method based on a large language model and fine-tuning optimization as described in claim 1 or claim 2.

7. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the tool use command planning method based on a large language model and fine-tuning optimization as described in claim 1 or claim 2.

Images