Tool use command planning method based on large language model and fine tuning optimization

Abstract

The invention relates to a tool use command planning method based on a large language model and fine tuning optimization, and the method comprises the steps: carrying out the basic processes of command planning method based on priori knowledge acquisition, environment target detection, execution step generation and the like, and carrying out the man-machine interaction. On-line command planning is carried out by combining two interactive visible exploration strategies and invisible exploration strategies, and important information of command planning accepted by human beings is stored in the face of unknown scenes and unknown tasks; and when the stored command plans are enough, facing the same human instruction, the result is retrieved from the stored effective information of the correct command plans by utilizing the retrieval enhancement generation strategy in each link through an offline command planning framework retrieved based on the retrieval enhancement generation strategy. Therefore, the problem that in the prior art, the universality of the scene instruction and the performability of the generation execution step are difficult to effectively consider is solved.

Description

Technical Field

[0001] The present application relates to the technical field of command planning and design, and in particular to a tool-based command planning method based on a large language model and fine-tuning optimization. Background Art

[0002] At present, the development of large language models (LLMs) has attracted wide attention in the academic community. Relying on huge data sets from sources such as the Internet and advanced hardware training facilities, large language models have excellent induction, understanding, reasoning and other capabilities in many fields. Therefore, in the past two years, research on large models has emerged in an endless stream at home and abroad. One of the more popular directions is to use large language models for embodied intelligent robots, that is, to enable large language models to use their rich knowledge to give action steps that robots can execute based on human natural language instructions, so that robots can complete human instructions from a physical level.

[0003] To achieve the above goals, the current mainstream implementation idea in academia is that humans first give natural language instructions to the robot that do not express information about the surrounding environment; then the large language model gives corresponding action steps based on the human's natural language instructions, taking into account its own rich knowledge and information about the surrounding environment.

[0004] Currently, the SayCan model can use the semantic understanding ability of the large language model to process natural language instructions given by humans, and use the latter to decompose the received instructions into several small executable task units. Then, through the large language model, the contribution rate of a specific action step to the completion of the final task is evaluated, and through reinforcement learning, the probability of the action step being completed in reality is evaluated, so as to multiply the contribution rate and the completion probability to obtain the final value function, and select the most appropriate action step by selecting the largest value function value.

[0005] Secondly, the PaLM-E model has a significant end-to-end multimodal nature. It can encode the natural language instruction information given by humans and the information in the external environment into vectors of the same format in some way, and input them into the model for training at the same time. While completing the requirements of human instructions, it can also complete other multimodal tasks (such as inputting pictures for scene description, etc.), and performs well on OK-VQA.

[0006] Thirdly, the LM-Nav model does not require labeled data or fine-tuning of large models, and can effectively follow human natural language instructions to complete navigation tasks. It emphasizes the use of multiple dedicated models to work together, including the vision-navigation module (VNM), the vision-language module (VLM), and the large language model module (LLM). By utilizing the VNM and VLM modules, the LM-Nav model can extract navigation landmarks from human natural language instructions and thus navigate to the specified location.

[0007] In addition, the goal of the TaPA (TAsk Planning Agent) model is to provide execution steps in natural language that are in line with human habits for common tasks of natural human instructions. The idea is to use the CLIP method to identify object labels in the environment around the embodied intelligent robot; then use the object labels in natural language as prompts, and combine them into an overall designed prompt, so that GPT-3.5 can generate a human instruction-execution step dataset under the prompt; then use the generated dataset to fine-tune the LLaMA pre-trained large model, so as to obtain a large model that can achieve the expected effect.

[0008] However, the current academic research on embodied intelligence command planning tasks rarely takes into account the universality of scene instructions and the executableness of generated execution steps. Several influential works, while focusing on the accuracy and executableness of command planning, often impose strict restrictions on human instructions and task scenarios. For example, the SayCan model has strict restrictions on the task scenarios and action content of embodied intelligent robots, and the effect is poor for untrained scenarios and complex instructions; the PaLM-E model also has strict restrictions on task scenarios; and when working to enhance the universality of command planning under complex human instructions and task scenario conditions, there is a high possibility that the command planning will be general and lack actual execution details due to the difficulty in fully extracting detailed information from the surrounding environment. For example, the action steps generated by the TaPA model for human instructions lack execution details, which only conform to human cognitive habits and cannot be truly executed by robots.

[0009] In summary, it is difficult for the existing technology to effectively balance the universality of scene instructions and the executability of generated execution steps, which needs to be solved urgently. Summary of the invention

[0010] The present application provides a tool usage command planning method based on a large language model and fine-tuning optimization to solve the problem that the existing technology is difficult to effectively balance the universality of scene instructions and the executableness of generated execution steps.

[0011] The first aspect of the present application is a method for planning tool usage commands based on a large language model and fine-tuning optimization, which is applied to the online command planning stage and includes the following steps: obtaining target human instructions, and based on a preset online data collection strategy, generating a target surrounding environment picture, and performing command planning operations according to the preset large language model, the target human instructions and the target surrounding environment picture to generate a target candidate tool and a first command plan in the target surrounding environment picture, and judging whether the first command plan meets the preset reasonable requirements. If the first command plan meets the preset reasonable requirements, the first command plan is used as the final command plan; if the first command plan does not meet the preset reasonable requirements, a first-level visible exploration operation is performed on the target candidate tool to generate a first local exploration area and a second command plan corresponding to the local exploration area, and judging whether the second command plan meets the preset reasonable requirements. If the second command plan meets the preset reasonable requirements, If the preset reasonable requirement is met, the second command plan is used as the final command plan; if the second command plan does not meet the preset reasonable requirement, a second-level visible exploration operation is performed on the first local exploration area to obtain a second local exploration area, and a command planning operation is performed on the second local exploration area to generate a third command plan corresponding to the second local exploration area, and it is judged whether the third command plan meets the preset reasonable requirement; if the third command plan does not meet the preset reasonable requirement, the target embodied intelligent robot is controlled to perform an invisible exploration operation to generate a target invisible area, and the target invisible area is explored by the target embodied intelligent robot; if the third command plan meets the preset reasonable requirement, the third command plan is used as the final command plan, and the planning information corresponding to the final command plan is obtained and stored, so as to use the planning information to perform an offline command plan generation operation in the offline command planning stage.

[0012] Optionally, in one embodiment of the present application, the target human instruction is obtained, and based on a preset online data collection strategy, a target surrounding environment picture is generated, and a command planning operation is performed according to a preset large language model, the target human instruction and the target surrounding environment picture to generate a target candidate tool and a first command plan in the target surrounding environment picture, including: based on the target human instruction and the large language model, a target tool label and a target tool appearance property corresponding to the target human instruction are generated, and according to the target tool label, an initial picture and multiple interference pictures are randomly selected from a preset data set, and a random splicing operation is performed on the initial picture and the multiple interference pictures to obtain the target surrounding environment picture; the target tool label is input into a preset visual target detection model to generate different candidate tool schematics, and according to the target surrounding environment picture, the different candidate tool schematics and the The visual target detection model obtains the candidate tool confidences corresponding to different candidate tools; sorts the candidate tool confidences in descending order to obtain a sorting result, and screens out multiple target candidate tool confidences that meet preset confidence requirements based on the sorting result, and obtains the sorting labels of the multiple target candidate tool confidences in the sorting result and the target candidate tools corresponding to the multiple target candidate tool confidences; performs background filtering and tool reorganization operations on the target candidate tools to obtain a reorganized candidate tool graph, and inputs the reorganized candidate tool graph, the target tool label and the target tool appearance properties into the large language model to screen out the target sorting label corresponding to the target human instruction; based on a preset segmentation model, performs segmentation and filtering operations on the target candidate tools corresponding to the target sorting label to obtain a corresponding segmentation map, and inputs the segmentation map into the large language model to generate the first command plan.

[0013] Optionally, in one embodiment of the present application, the first level visible exploration operation is performed on the target candidate tool to generate a first local exploration area and a second command plan corresponding to the local exploration area, including: filtering the top N target candidate tool confidences among the multiple target candidate tool confidences according to the sorting result, and obtaining the target candidate tools corresponding to the top N target candidate tool confidences, where N is a positive integer; adding the target candidate tools corresponding to the target sorting number to the target candidate tools corresponding to the top N target candidate tool confidences to obtain a plurality of target candidate tools to be screened; performing splicing and background filtering operations on the plurality of target candidate tools to be screened to generate the first local exploration area, and inputting the first local exploration area into the large language model to generate the second command plan.

[0014] Optionally, in one embodiment of the present application, the second-level visible exploration operation is performed on the first local exploration area to obtain a second local exploration area, and a command planning operation is performed on the second local exploration area to generate a third command plan corresponding to the second local exploration area, including: screening out multiple candidate tool confidences corresponding to the target surrounding environment picture that meets the preset sorting label requirements from the sorting results, and obtaining multiple target candidate tools corresponding to the multiple candidate tool confidences; weighting the targets to be detected in the target area of ​​each target candidate tool in the multiple target candidate tools to obtain the weight accumulation result corresponding to each target candidate tool, and comparing the weight accumulation results of each target candidate tool to obtain the maximum weight accumulation result; obtaining the target candidate tool corresponding to the maximum weight accumulation result, and taking the target area of ​​the target candidate tool corresponding to the maximum weight accumulation result as the second local exploration area; inputting the second local exploration area into the large language model to generate the third command plan.

[0015] The second aspect of the present application provides a tool usage command planning method based on a large language model and fine-tuning optimization, which is applied to the offline command planning stage, and includes the following steps: based on a preset offline data collection strategy and a retrieval enhancement generation strategy, obtaining pre-stored planning information, and retrieving the target tool image in the planning information, and respectively calculating the first similarity between the target tool image and a plurality of preset target candidate tools, and obtaining the optimal target candidate tool with the highest first similarity; using the retrieval enhancement generation strategy to retrieve the target operation area image corresponding to the optimal target candidate tool in the planning information, and respectively calculating the second similarity between the target operation area image and a plurality of preset candidate areas, and screening out at least one target operation area image whose second similarity meets the preset similarity requirement, so that the retrieval enhancement generation strategy is based on the at least one target operation area image. The area image retrieves the corresponding first command planning information in the planning information, and determines whether the first command planning information meets the preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, the first command planning information is used as the final command planning; if the first command planning information does not meet the preset reasonable requirements, the key area image corresponding to the optimal target candidate tool in the planning information is retrieved through the retrieval enhancement generation strategy, and the third similarity between the key area image and the multiple key area images corresponding to the multiple target candidate tools is calculated respectively, and the corresponding second command planning information is generated based on the target candidate tool corresponding to the target key area image with the highest third similarity; based on the second command planning information and the preset reasonable requirements, a preset second-level visible exploration and / or invisible exploration operation is performed to generate a final command plan.

[0016] The third aspect of the present application is a tool use command planning device based on a large language model and fine-tuning optimization, which is applied to the online command planning stage, including: a command planning module, which is used to obtain target human instructions, and based on a preset online data collection strategy, generate a target surrounding environment picture, and perform command planning operations according to the preset large language model, the target human instructions and the target surrounding environment picture to generate a target candidate tool and a first command plan in the target surrounding environment picture, and judge whether the first command plan meets the preset reasonable requirements. If the first command plan meets the preset reasonable requirements, the first command plan is used as the final command plan; a first exploration module, which is used to perform a first level visible exploration operation on the target candidate tool if the first command plan does not meet the preset reasonable requirements, so as to generate a first local exploration area and a second command plan corresponding to the local exploration area, and judge whether the second command plan meets the preset reasonable requirements. If the second command plan meets the preset reasonable requirements, If the second command plan meets the preset reasonable requirements, the second command plan is used as the final command plan; a second exploration module is used to perform a second-level visible exploration operation on the first local exploration area to obtain a second local exploration area if the second command plan does not meet the preset reasonable requirements, and perform a command planning operation on the second local exploration area to generate a third command plan corresponding to the second local exploration area, and judge whether the third command plan meets the preset reasonable requirements; if the third command plan does not meet the preset reasonable requirements, control the target embodied intelligent robot to perform an invisible exploration operation to generate a target invisible area, and explore the target invisible area through the target embodied intelligent robot; if the third command plan meets the preset reasonable requirements, use the third command plan as the final command plan, and obtain and store the planning information corresponding to the final command plan, so as to use the planning information to perform an offline command plan generation operation in the offline command planning stage.

[0017] Optionally, in one embodiment of the present application, the command planning module includes: an online data acquisition unit, which is used to generate a target tool label and a target tool appearance property corresponding to the target human instruction based on the target human instruction and the large language model, and randomly select an initial image and multiple interference images in a preset data set according to the target tool label, and randomly splice the initial image and the multiple interference images to obtain the target surrounding environment image; a first acquisition unit, which is used to input the target tool label into a preset visual target detection model to generate different candidate tool schematics, and obtain candidate tool confidences corresponding to different candidate tools according to the target surrounding environment image, the different candidate tool schematics and the visual target detection model; a coarse screening unit, which is used to sort the candidate tool confidences in descending order. A method for selecting a target tool for a human operator is provided for selecting a target tool for a human operator who has a plurality of target tools that are of a predetermined confidence level. The method comprises the following steps: a first step is to select a target tool for a human operator who has a plurality of target tools that are of a predetermined confidence level and a plurality of target tools that are of a predetermined confidence level. The method comprises the following steps: a first step is to select a target tool for a human operator who has a plurality of target tools that are of a predetermined confidence level and a plurality of target tools that are of a predetermined confidence level. The method comprises the following steps: a first step is to select a target tool for a human operator who has a plurality of target tools that are of a predetermined confidence level and a plurality of target tools that are of a predetermined confidence level. The method comprises the following steps: a first step is to select a target tool for a human operator who has a plurality of target tools that are of a predetermined confidence level and a plurality of target tools that are of a predetermined confidence level. The method comprises the following steps:

[0018] Optionally, in one embodiment of the present application, the first exploration module includes: a second acquisition unit, used to filter the top N target candidate tool confidences among the multiple target candidate tool confidences according to the sorting result, and obtain the target candidate tools corresponding to the top N target candidate tool confidences, where N is a positive integer; an adding unit, used to add the target candidate tools corresponding to the target sorting number to the target candidate tools corresponding to the top N target candidate tool confidences to obtain a plurality of target candidate tools to be screened; a filtering unit, used to perform splicing and background filtering operations on the plurality of target candidate tools to be screened to generate the first local exploration area, and input the first local exploration area into the large language model to generate the second command plan.

[0019] Optionally, in one embodiment of the present application, the second exploration module includes: a third acquisition unit, used to filter out multiple candidate tool confidences corresponding to the target surrounding environment picture that meets the preset sorting label requirements from the sorting results, and obtain multiple target candidate tools corresponding to the multiple candidate tool confidences; a weight accumulation unit, used to weight the targets to be detected in the target area of ​​each target candidate tool in the multiple target candidate tools to obtain the weight accumulation result corresponding to each target candidate tool, and compare the weight accumulation results of each target candidate tool to obtain the maximum weight accumulation result; a fourth acquisition unit, used to obtain the target candidate tool corresponding to the maximum weight accumulation result, and use the target area of ​​the target candidate tool corresponding to the maximum weight accumulation result as the second local exploration area; a generation unit, used to input the second local exploration area into the large language model to generate the third command plan.

[0020] The fourth aspect of the present application provides a tool usage command planning device based on a large language model and fine-tuning optimization, which is applied to the offline command planning stage, including: a retrieval module, which is used to obtain pre-stored planning information based on a preset offline data collection strategy and a retrieval enhancement generation strategy, and retrieve the target tool image in the planning information, and respectively calculate the first similarity between the target tool image and a plurality of preset target candidate tools, and obtain the optimal target candidate tool with the highest first similarity; a first calculation module, which is used to use the retrieval enhancement generation strategy to retrieve the target operation area image corresponding to the optimal target candidate tool in the planning information, and respectively calculate the second similarity between the target operation area image and a plurality of preset candidate areas, and screen out at least one target operation area image whose second similarity meets the preset similarity requirement, so that the retrieval enhancement generation strategy is based on the at least one target operation area image. a first command planning information corresponding to the first command planning information in the planning information, and judging whether the first command planning information meets the preset reasonable requirements, and if the first command planning information meets the preset reasonable requirements, taking the first command planning information as the final command planning; a second calculation module, for retrieving the key area picture corresponding to the optimal target candidate tool in the planning information through the retrieval enhancement generation strategy if the first command planning information does not meet the preset reasonable requirements, and respectively calculating the third similarity between the key area picture and the multiple key area pictures corresponding to the multiple target candidate tools, and generating the corresponding second command planning information based on the target candidate tool corresponding to the target key area picture with the highest third similarity; an offline exploration module, for performing a preset second-level visible exploration and / or invisible exploration operation based on the second command planning information and the preset reasonable requirements to generate a final command plan.

[0021] The fifth aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a tool usage command planning method based on a large language model and fine-tuning optimization as described in the above embodiments.

[0022] The sixth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, which, when executed by a processor, implements the tool command planning method based on a large language model and fine-tuning optimization as described above.

[0023] The seventh aspect embodiment of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned tool usage command planning method based on a large language model and fine-tuning optimization.

[0024] Therefore, the embodiments of the present application have the following beneficial effects:

[0025] The embodiment of the present application can generate the corresponding command plan by first going through the basic process of the command planning method after inputting human instructions and surrounding environment pictures, and ask the human whether the command plan is reasonable. If it is reasonable, the command plan is used as the final result to end the task; if it is unreasonable, the first level of visible exploration is performed, and the candidate tools are selected twice and the corresponding command plan is generated. At this time, continue to ask the human whether the command plan is reasonable. If it is reasonable, the task is ended as the final result; if it is unreasonable, the second level of visible exploration is performed, and the local area obtained by the exploration is used as a new surrounding scene picture to re-plan the command. At this time, continue to ask the human whether the command plan is reasonable. If it is reasonable, the task is ended as the final result; if it is unreasonable, invisible exploration is performed to generate invisible areas where the tools required to complete the human instructions may exist, for the embodied intelligent robot to further explore, and end the task. The present application enables the robot to generate corresponding execution steps for any human instructions in complex unknown scenes, and can be executed efficiently and reasonably. The present application enables the robot to generate corresponding execution steps for any human instructions in complex unknown scenes, and can be executed efficiently and reasonably. This solves the problem that the prior art is difficult to effectively balance the universality of scene instructions and the executability of generated execution steps.

[0026] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A flowchart of a command planning method for a tool based on a large language model and fine-tuning optimization applied to an online command planning stage according to an embodiment of the present application;

[0029] Figure 2 A basic framework diagram of a command planning method provided according to an embodiment of the present application;

[0030] Figure 3 A schematic diagram of an execution architecture of a command planning method provided according to an embodiment of the present application;

[0031] Figure 4 This is an example diagram of a data set image in an online command planning framework involving GPT-4 according to an embodiment of the present application;

[0032] Figure 5 A surrounding scene diagram of an embodied intelligent robot provided according to an embodiment of the present application;

[0033] Figure 6 A tool candidate diagram for human reference provided according to an embodiment of the present application;

[0034] Figure 7 A tool candidate graph for inputting into GPT4 provided according to an embodiment of the present application;

[0035] Figure 8 A schematic diagram of segmentation numbering of selected tools provided according to an embodiment of the present application;

[0036] Fig. 9 A schematic diagram of the overall flow of a command planning method for unknown scenarios provided according to an embodiment of the present application;

[0037] Fig.10 A new candidate tool map generated by a first-level visible exploration according to an embodiment of the present application;

[0038] Fig.11 A schematic diagram of a complex indoor scene and a result of a second-level visible exploration provided according to an embodiment of the present application;

[0039] Fig.12 A schematic diagram of an effect of performing invisible exploration in a complex indoor scene according to an embodiment of the present application;

[0040] Fig.13 A flowchart of a command planning method for a tool based on a large language model and fine-tuning optimization applied to an offline command planning stage according to an embodiment of the present application;

[0041] Fig.14 This is an example of a surrounding environment picture generated by DALL-E 3 according to an embodiment of the present application;

[0042] Fig.15 This is an example of a surrounding environment picture taken by a real machine according to an embodiment of the present application;

[0043] Fig.16 An example diagram of a command planning device used in a tool based on a large language model and fine-tuning optimization applied to an online command planning stage according to an embodiment of the present application;

[0044] Fig.17 An example diagram of a command planning device used in a tool based on a large language model and fine-tuning optimization applied to an offline command planning stage according to an embodiment of the present application;

[0045] Fig.18 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0046] Among them, 10-a tool based on a large language model and fine-tuning optimization uses a command planning device applied in the online command planning stage, 20-a tool based on a large language model and fine-tuning optimization uses a command planning device applied in the offline command planning stage; 101-a command planning module, 102-a first exploration module, 103-a second exploration module; 201-a retrieval module, 202-a first calculation module, 203-a second calculation module, 204-an offline exploration module; 1801-a memory, 1802-a processor, 1803-a communication interface. DETAILED DESCRIPTION

[0047] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0048] The following is a description of a tool use command planning method based on a large language model and fine-tuning optimization of an embodiment of the present application with reference to the accompanying drawings. In view of the problems mentioned in the above background technology, the present application provides a tool use command planning method based on a large language model and fine-tuning optimization, in which the basic flow of command planning methods such as prior knowledge acquisition, environmental target detection, and execution step generation is performed, and the human-computer interaction link is used, and two interactive visible exploration strategies and invisible exploration strategies are combined to perform online command planning. When facing unknown scenes and unknown tasks, the important information of command planning recognized by humans is stored; when there are enough stored command plans, facing the same human instructions, the offline command planning framework based on RAG (Retrieval-augmented Generation) retrieval is used to retrieve the results from the valid information of the stored correct command planning using RAG in each link. The present application enables the robot to generate corresponding execution steps for any human instructions in complex unknown scenes, and can execute them efficiently and reasonably. Thus, the problem that the prior art is difficult to effectively balance the universality of scene instructions and the executableness of generated execution steps is solved.

[0049] Specifically, Figure 1 A flowchart of a command planning method for a tool based on a large language model and fine-tuning optimization used in an online command planning stage provided in an embodiment of the present application.

[0050] like Figure 1 As shown, the command planning method used by the tool based on a large language model and fine-tuning optimization applied to the online command planning stage includes the following steps:

[0051] In step S101, the target human instructions are obtained, and based on the preset online data collection strategy, a target surrounding environment picture is generated, and a command planning operation is performed according to the preset large language model, the target human instructions and the target surrounding environment picture to generate a target candidate tool and a first command plan in the target surrounding environment picture, and determine whether the first command plan meets the preset reasonable requirements. If the first command plan meets the preset reasonable requirements, the first command plan is used as the final command plan.

[0052] The command planning strategy of the embodiment of the present application mainly includes three processes: prior knowledge acquisition, environmental target detection, and execution step generation. Figure 2 shown.

[0053] Specifically, the embodiments of the present application can first use the knowledge of the large language model GPT-4 to obtain the labels of the tools needed to execute human natural language instructions, as well as the appearance properties of the tools; secondly, it passes through the visual target detection model Grounding DINO and the multimodal GPT-4 in turn. The former inputs the tool labels, and the latter inputs the tool labels and the appearance properties of the tools, and performs a two-step screening of the required tools for the surrounding indoor environment scene, and accurately detects the tools required to complete human instructions; further, the embodiments of the present application can input the tool images obtained by detection into the large language model GPT-4 after Semantic-SAM segmentation and labeling, so as to output the final required command planning for human instructions.

[0054] It should be noted that the command planning method of the embodiment of the present application attempts to improve the large model's ability to understand external information and the quality of generated results by designing multimodal prompt words for the large language model input, thereby achieving an accurate and detailed operation procedure with execution level for natural language instructions given by humans in complex indoor scenes.

[0055] Optionally, in one embodiment of the present application, a target human instruction is obtained, and based on a preset online data collection strategy, a target surrounding environment picture is generated, and a command planning operation is performed according to a preset large language model, the target human instruction and the target surrounding environment picture to generate a target candidate tool and a first command plan in the target surrounding environment picture, including: based on the target human instruction and the large language model, a target tool label and target tool appearance properties corresponding to the target human instruction are generated, and according to the target tool label, an initial picture and multiple interference pictures are randomly selected from a preset data set, and an initial picture and multiple interference pictures are randomly spliced ​​to obtain a target surrounding environment picture; the target tool label is input into a preset visual target detection model to generate different candidate tool schematics, and according to the target surrounding environment picture, different candidate tool schematics are generated. and a visual target detection model to obtain candidate tool confidences corresponding to different candidate tools; sort the candidate tool confidences in descending order to obtain a sorting result, and screen out multiple target candidate tool confidences that meet preset confidence requirements based on the sorting result, and obtain the sorting labels of the multiple target candidate tool confidences in the sorting result and the target candidate tools corresponding to the multiple target candidate tool confidences; perform background filtering and tool reorganization operations on the target candidate tools to obtain a reorganized candidate tool graph, and input the reorganized candidate tool graph, the target tool label and the target tool appearance properties into the large language model to screen out the target sorting label corresponding to the target human instruction; based on a preset segmentation model, perform segmentation and filtering operations on the target candidate tools corresponding to the target sorting label to obtain a corresponding segmentation graph, and input the segmentation graph into the large language model to generate a first command plan.

[0056] It should be noted that the detailed implementation framework of the command planning method in the embodiment of the present application is as follows: Figure 3 As shown, the specific execution logic of each functional module in the implementation framework is as follows:

[0057] 1. Prior knowledge acquisition module:

[0058] This module mainly consists of a large language model GPT-4, which generates common sense-based general knowledge such as tool labels and tool appearance properties required to complete human instructions based on human natural language instructions without considering the surrounding environment, so that the latter can be used in subsequent environmental target detection modules and the visual perception model's perception of the surrounding indoor environment.

[0059] Specifically, the labels of the tools required to complete the instructions are given directly by GPT-4 based on the human instructions, and then GPT-4 comprehensively considers the labels of the selected tools and the detailed details of the human instructions to obtain the appearance properties that the selected tool should have if it is guided by completing human instructions.

[0060] 2. Environmental target detection module:

[0061] In the online command planning framework stage, the embodiments of the present application need to construct a surrounding scene dataset; specifically, the embodiments of the present application can target the input human instructions, manually specify the most appropriate tool label to complete the instructions, and based on the label, randomly select a picture in the ImageNet dataset, and then randomly select several interference pictures from ImageNet, and randomly splice all the selected pictures into a large picture, so as to use the latter as the input simulated surrounding scene.

[0062] For example, for the command “I want to chop bones.”, the following are examples of images used to construct the dataset: Figure 4 As shown in the figure, it includes the bone cutting knives that are really needed and extracted from ImageNet, as well as several manually selected knives such as knives that are not suitable for bone cutting as interference items. Since the tools in the figure are large and clear, this type of scenario helps the "online" framework to generate reasonable command plans and store them.

[0063] Furthermore, the environmental target detection module of the embodiment of the present application can perform a two-step screening process of coarse screening and fine screening of the required tools in the surrounding indoor scene pictures. In the actual implementation process, the embodiment of the present application first detects and frames the candidate tools that may meet the conditions in the surrounding environment according to the tool labels by the visual target detection model Grounding DINO to achieve preliminary screening; secondly, after pre-processing the framed candidate tools, the candidate tools and the requirements for the appearance properties of the tools are input into the multimodal GPT-4, so that it can accurately detect the tools required to complete human instructions.

[0064] Specifically, in the rough screening stage, in view of the different requirements of humans and GPT-4 for reference judgment and precise selection of tools, the embodiment of the present application can generate two candidate tool schematics based on the detection results of the tools by Grounding DINO.

[0065] The schematic diagram of candidate tools for human inspection and reference uses Grounding DINO to detect and frame the top 9 objects in the current scene with confidence, and marks the serial number (ranging from 1 to 9) obtained from the candidate tool confidence ranking in the upper left corner, thereby restoring the positional relationship of the candidate tools in the original image; if GPT-4 subsequently selects the number of the appropriate tool (1 to 9), the corresponding tool can be quickly found in the image, which can achieve the purpose of human inspection and reference; the schematic diagram of candidate tools for input into GPT-4 uses Grounding DINO to detect and frame the top 9 candidate tools with confidence, and adds two steps of background filtering and tool reorganization. The former uses the segmentation module in Grounded SAM to filter the background outside the candidate tool to white (that is, only the tool image within the candidate tool mask range is retained); the latter recombines and arranges the candidate tools after background filtering for GPT-4 observation.

[0066] In the fine screening stage, the screening strategy mainly uses the visual perception ability of GPT-4 to design prompt words, so that it can select the label of the most suitable tool to complete the human instruction from the candidate tool map generated previously. In the prompt word design of this question, the embodiment of the present application can force GPT-4 to decompose the problem for tool selection through the question template setting, thereby improving the selection accuracy.

[0067] Specifically, since the visual information of the surrounding environment is given, the embodiment of the present application sets GPT-4 to first give the appearance properties that the tool should have based on the tool label required to complete the instruction and comprehensively consider the details of the human instruction; secondly, based on the appearance properties of the tool, select the most suitable tool label from the candidate tool map.

[0068] It should be noted that, in the actual implementation process, technicians in the field can also select other appropriate network models according to actual conditions, such as CLIP instead of Grounding DINO for environmental target detection and other operations, and no specific limitation is made accordingly.

[0069] 3. Execute steps to generate modules:

[0070] This module mainly divides and labels the pictures to be understood by GPT-4 through an advanced segmentation model, and then inputs them into GPT-4, so that the latter can generate the execution step results.

[0071] Specifically, in this module, the embodiment of the present application can first use the Semantic-SAM segmentation model to select multiple granularities to segment the tool image corresponding to the tool label selected by GPT-4 in the environmental target detection module. At the same time, the tool mask generated by Grounded SAM in the environmental target detection module is used to filter the segmented tools again to filter out the segmentation and labeling of the background that is not related to the task; after generating the segmentation map of the tool, the segmentation map can be input into GPT-4 according to the idea of ​​SoM prompt word design, so that the latter generates command planning for human instructions based on the segmentation labels of each area of ​​the tool.

[0072] The command plan generation process is described in detail below through a specific embodiment and in conjunction with the accompanying drawings.

[0073] Assume that the human instruction is “I want to chop bones.” and the surrounding scene of the embodied intelligent robot is as follows: Figure 5 As shown, an example of using the above command planning method to plan commands for human instructions is shown below.

[0074] After the prior knowledge acquisition module, the specific embodiment of the present application can obtain through GPT-4 the tool label required to complete the human instruction as "cleaver", the tool appearance property as "A cleaver designed for chopping bones should have a thick, heavy blade made of high-carbon steel,with a broad and tough edge to withstand the impact of cutting through hard materials. Briefly,the cleaver for chopping bones is a robust and heavy-duty knife,characterized by its broad,thick blade optimized for breaking down meat and bones withease.",It can be seen that GPT-4 correctly gives the "bone chopper" tool and the corresponding appearance properties required to complete the bone cutting instruction.

[0075] In the environmental object detection module, the embodiment of the present application can generate a tool candidate map for human reference and a tool candidate map input to GPT4 according to the tool label and appearance properties, respectively. Figure 6 , Figure 7 As shown. Figure 6 and Figure 7 It can be seen that the former truly retains the positional relationship between the candidate tools, which is convenient for human inspection and reference; the latter removes interference factors other than the candidate tools, making it easier for GPT-4 to select the correct tool. GPT-4 finally selected the tool represented by number 3 as the most suitable tool for cutting bones. Figure 7 As shown, since bone cutting requires a knife with a wider and thicker blade, it can be seen that GPT-4 has selected the most suitable tool, verifying the effectiveness of the command planning method in the embodiment of the present application.

[0076] For the selected label 3 tool, the region segmentation label diagram is as follows Figure 8As shown, the corresponding tool usage method output is: "To chop bones using this cleaver, the key part of the tool to use is theblade, which is indicated by the area labeled with the numeric ID 1. This part of the cleaver is designed for heavy chopping actions and is capable ofcutting through bone due to its heft and sharp edge. You would need to grip the handle for a secure grasp, which is firmly indicated by the numeric ID 8. It's important to keep your fingers away from the blade area to avoid any potential injuries. Carefully position the blade edge (area 1) against the bone you to chop. Then, using controlled force, bringing the blade down onto the bone in a chopping motion, allowing the weight and sharpness of the cleaver to do most of the work in cutting through the bone. Always exercise caution when using such a tool and ensure that you're using proper techniques to preventaccidents." It can be seen that for the task of cutting bones, GPT-4 believes that the tool corresponding to label 3 should be used, and the method of use is to hold the handle area represented by 8 and use the blade area represented by 1 to cut bones, giving the correct command planning, which further verifies the effectiveness of the command planning method.

[0077] After obtaining the corresponding command plan (i.e., the first command plan), the embodiment of the present application needs to ask humans whether the command plan is reasonable (i.e., judge by humans whether the command plan meets the preset reasonable requirements). If reasonable, the command plan is used as the final result to end the task.

[0078] In step S102, if the first command plan does not meet the preset reasonable requirements, a first-level visible exploration operation is performed on the target candidate tool to generate a first local exploration area and a second command plan corresponding to the local exploration area, and determine whether the second command plan meets the preset reasonable requirements. If the second command plan meets the preset reasonable requirements, the second command plan is used as the final command plan.

[0079] In addition, if the first command plan is unreasonable, the first level of visible exploration is performed, a secondary selection of candidate tools is made and a corresponding command plan (i.e., the second command plan) is generated. At this time, the human is asked whether the command plan is reasonable. If it is reasonable, the task is terminated as the final result.

[0080] Optionally, in one embodiment of the present application, a first level visible exploration operation is performed on the target candidate tool to generate a first local exploration area and a second command plan corresponding to the local exploration area, including: filtering the top N target candidate tool confidences among multiple target candidate tool confidences according to the sorting results, and obtaining the target candidate tools corresponding to the top N target candidate tool confidences, where N is a positive integer; adding the target candidate tools corresponding to the target sorting number to the target candidate tools corresponding to the top N target candidate tool confidences to obtain multiple target candidate tools to be screened; performing splicing and background filtering operations on the multiple target candidate tools to be screened to generate a first local exploration area, and inputting the first local exploration area into the large language model to generate a second command plan.

[0081] It should be noted that the basic process of the above command planning method is mainly based on the assumption that the surrounding indoor environment is relatively ideal, that is, the tools required in indoor scenes are larger in size and easier to detect and perceive. However, in more general cases, the tools required to complete human instructions may not be near the embodied intelligent robot, but are far away, or there may be no suitable tools within the robot's field of view. In this case, the visual perception model used in the above command planning method may not be able to effectively detect and select the target tool, resulting in the failure of the final command planning.

[0082] In response to the above-mentioned problem of command planning failure caused by the inefficient detection of non-ideal environments by visual perception models, the embodiments of the present application utilize visual perception models such as multimodal GPT-4, Grounding DINO, etc., and design an environmental perception exploration strategy for the available tools in the surrounding environment based on the assumption that the tools in the surrounding indoor environment are small in size and low in quality. In addition, two types of visual perception exploration strategies, visible exploration and invisible exploration, are designed for the two situations where the tools required in the surrounding environment are directly visible or not, respectively, thereby achieving more comprehensive and reasonable extraction of surrounding environmental information and improving adaptability to unknown environments and unknown situations.

[0083] Specifically, for the non-ideal situation where the tools required for command planning are visible in the field of vision but difficult to detect accurately, the embodiments of the present application design two interactive visible exploration strategies (first-level visible exploration and second-level visible exploration); for the non-ideal situation where the required tools are not directly visible in the current scene, the embodiments of the present application design an invisible exploration strategy.

[0084] Among them, the premise for executing the first-level visible exploration strategy is: although the required tools in the surrounding environment are visible and large in size (close to the robot, and can be detected by the target detection module Grounding DINO), the tool details are difficult to distinguish due to external reasons and the candidate tools cause greater interference due to their similar appearance. Using the original command planning method, the accuracy of tool selection by GPT-4 is reduced.

[0085] In response to the above problems, and based on the assumption in the logic of the Grounding DINO algorithm that the candidate tools with higher confidence are more likely to be the real required tools, the core of the first-level visible exploration strategy design is: after GPT-4 selects the required tools under the framework of the original command planning method, the selected tools are added to several tools with confidence in the top N (such as the top 5) of the 9 tools detected by Grounding DINO, and GPT-4 makes a secondary selection of the above-mentioned narrowed candidate tools.

[0086] Specifically, when the tools selected by GPT-4 for the first time are numbered 6 to 9, the exploration method will splice and filter the background of the selected tool pictures with the tool pictures numbered 1 to 5, a total of 6 pictures, to form candidate tool pictures, which are input into GPT-4 for secondary selection; when the tools selected by GPT-4 for the first time are numbered 1 to 5, the exploration method will splice and filter the background of the tool pictures numbered 1 to 5, a total of 5 pictures, to form candidate tool pictures, which are input into GPT-4 for secondary selection.

[0087] Since the first-level visible exploration strategy excludes tools with low Grounding DINO detection confidence, it allows GPT-4 to mix the tools selected last time into tools with higher Grounding DINO detection confidence in a smaller range for secondary selection, thereby removing some interfering tools. Therefore, the use of this first-level visible exploration strategy can improve the accuracy of GPT-4's selection of the required tools among the 9 candidate tools.

[0088] In step S103, if the second command plan does not meet the preset reasonable requirements, a second-level visible exploration operation is performed on the first local exploration area to obtain a second local exploration area, and a command planning operation is performed on the second local exploration area to generate a third command plan corresponding to the second local exploration area, and it is determined whether the third command plan meets the preset reasonable requirements. If the third command plan does not meet the preset reasonable requirements, the target embodied intelligent robot is controlled to perform an invisible exploration operation to generate a target invisible area, and the target invisible area is explored by the target embodied intelligent robot. If the third command plan meets the preset reasonable requirements, the third command plan is used as the final command plan, and the planning information corresponding to the final command plan is obtained and stored, so as to use the planning information to perform the offline command plan generation operation in the offline command planning stage.

[0089] Furthermore, if the second command plan does not meet the preset reasonable requirements, the second level visible exploration is performed, such as Fig. 9 As shown, the local area obtained by exploration (i.e., the first local exploration area) is used as a new surrounding scene picture to re-plan the command to obtain the third command plan; at this time, the human is continued to be asked whether the command plan is reasonable. If it is reasonable, the task is terminated as the final result; if it is unreasonable, invisible exploration is performed to generate invisible areas where the tools required to complete the human instructions may exist, for the embodied intelligent robot to conduct further exploration, and the task is terminated.

[0090] Therefore, based on the basic process of the above-mentioned command planning method, the embodiments of the present application are combined with the environmental exploration strategy in the embodiments of the present application through the link of human-computer interaction, so that the command planning method designed based on the assumption of an ideal indoor environment can be widely used in other non-ideal, complex and changeable indoor scenes, thereby achieving the predetermined goals of the method design.

[0091] Optionally, in one embodiment of the present application, a second-level visible exploration operation is performed on the first local exploration area to obtain a second local exploration area, and a command planning operation is performed on the second local exploration area to generate a third command plan corresponding to the second local exploration area, including: screening out multiple candidate tool confidences corresponding to the target surrounding environment picture that meets the preset sorting label requirements from the sorting results, and obtaining multiple target candidate tools corresponding to the multiple candidate tool confidences; weighting the targets to be detected in the target area of ​​each target candidate tool in the multiple target candidate tools to obtain the weight accumulation result corresponding to each target candidate tool, and comparing the weight accumulation result of each target candidate tool to obtain the maximum weight accumulation result; obtaining the target candidate tool corresponding to the maximum weight accumulation result, and taking the target area of ​​the target candidate tool corresponding to the maximum weight accumulation result as the second local exploration area; inputting the second local exploration area into the large language model to generate a third command plan.

[0092] In the actual implementation process, the second-level visible exploration strategy of the embodiment of the present application is mainly aimed at the situation where the required tool is visible in the surrounding environment but is small in size (far away from the robot). At this time, Grounding DINO cannot detect the required tool, which causes GPT-4 to be unable to select the correct tool, and then the command planning task fails. In this case, the tool required to complete the human instruction is more likely to be located outside the top 9 confidence levels, but in an area near the top confidence level among all detected targets in Grounding DINO.

[0093] Based on the above situation, the core of the second-level visible exploration strategy is: for the candidate tools whose detection confidence ranks 10 to 18 in the original image of the surrounding environment by Grounding DINO, the weights of the detection targets in the surrounding areas of each of the above 9 tools are accumulated (the higher the confidence of the surrounding detection targets, the greater the accumulated weight) and compared, so that the surrounding area of ​​the candidate tool with the highest weight is used as the area for the next step of visual perception module detection, realizing the second-level visible exploration.

[0094] Specifically, the second-level visible exploration strategy focuses on the target objects whose detection confidence ranks 10 to 40 in the detection results of Grounding DINO in the original image of the surrounding environment; the exploration strategy traverses the fixed-size areas around the target objects ranked 10 to 18 (square areas with a radius of 250 pixels centered on the center of the target object), that is, 9 square areas, respectively. For each square area, all target objects with confidence ranking 10 to 40 that have a containment or intersection relationship with it are counted, and the weight of each object is accumulated (the accumulated weight is 40-confidence ranking), thereby counting 9 accumulated weights; then the sizes of the 9 accumulated weights are sorted, and the square area corresponding to the largest accumulated weight is used as the area for further exploration, thereby completing the goal of the second-level exploration.

[0095] In addition, for the non-ideal situation where the required tools are not directly visible in the current scene (at this time, whether directly using Grounding DINO to detect the original scene graph or using a visible exploration strategy, the required tools cannot be detected and selected), an invisible exploration strategy is designed. Its goal is to give the area where the required tools may exist in the current scene, so that the embodied intelligent robot can subsequently explore the above area. The core of the strategy is to utilize the rich common sense knowledge of GPT-4, and based on the currently available information, give labels for the areas where the required tools may be located, so that the visual object detection module can detect the actual area corresponding to the label in the current scene, that is, the area that the robot is expected to explore further.

[0096] Therefore, when no selectable tool is clearly visible or does not exist in the field of view, the embodiments of the present application can autonomously initiate interactive active exploration and provide areas or locations where the target tool may exist, thereby facilitating subsequent tool selection and task execution.

[0097] The following is an explanation of the execution effect of the environment exploration strategy designed in the embodiment of the present application through a specific embodiment and combined with the accompanying drawings.

[0098] In a specific embodiment of the present application, for Figure 5 The scene and bone cutting instructions, the first level can see the results of the exploration as follows Fig.10 As shown in the figure, the first-level visible exploration achieves the desired effect and narrows the scope of candidate tools through reasonable trade-offs. After experiments, GPT-4 can also Fig.10 Correctly select the tool corresponding to number 3.

[0099] Assume that the embodied intelligent robot is located in a more complex indoor scene, such as Fig.11As shown in the figure, the human command is still set to "I want to chop bones." Since the size of the bone cutter actually required is too small, Grounding DINO cannot detect the required tool at this time, so the correct command plan cannot be generated through the above command planning strategy and the first level visible exploration. The result of the second level visible exploration in this scene is shown in Fig.11 As shown in the area outlined in the blue box, it can be seen that the blue box contains the bone cutting knife required for the bone cutting task, thereby verifying the rationality of the logic of the second-level visible exploration strategy.

[0100] exist Fig.11 In the complex indoor scene shown in the figure, assuming that the command planning process and the first and second level visible exploration still do not meet human requirements, then Fig.11 The results of invisible exploration in the scene are as follows Fig.12 As shown. From this, we can see that based on the premise that the tools required to complete the instructions are not directly visible, Fig.12 The area where the bone cutting knife needed for bone cutting may be located (i.e., in the drawer) is reasonably framed, achieving the expectations of the invisible exploration strategy design.

[0101] The following is another specific embodiment to experimentally compare and illustrate the execution effect of the designed command planning method when there are interfering tools in the surrounding scene that are highly similar to the required tool but cannot complete the task or are not suitable for completing the task. The method can accurately eliminate the interference, select the required tool and perform command planning.

[0102] Through experiments, it can be seen that the command planning method designed in the specific embodiment of the present application can select the truly suitable tool from many interfering tools and perform corresponding command planning (such as selecting a bone cutting knife from several different types of knives to cut bones, selecting a colander from several pot spoons to filter, etc.), while the existing target detection methods and embodied intelligent models such as SayCan, PaLM-E, and TaPA are less effective in completing such tasks. For example, assuming that the given human instruction is "I want to chopbones.", for example Figure 5 In the scenario, there is not only the bone cutting knife we ​​really need, but also many other types of knives that belong to the same category of knives but are not suitable for cutting bones. At this time, the command planning method designed in the embodiment of the present application can select Figure 6The bone cutting knife with the middle label number 3 performs corresponding command planning to eliminate interference. The command planning accuracy of the command planning method of the embodiment of the present application and the embodied intelligent model TaPA are shown in Table 1, corresponding to four human instructions "I want to chop bones.", "I want to take cooked food out of the pot as well as filter out the liquid.", "I want to flip the pancake in the skillet.", "I want to fry a pancake.", and the two command planning methods are tested on the correspondingly constructed ImageNet splicing data sets. It can be seen from Table 1 that the effect of the command planning method designed in the embodiment of the present application is better than or equal to TaPA for the four instructions, and for instruction No. 2, the command planning accuracy of TaPA is extremely low, while relatively, the present application has reached a higher level.

[0103] Table 1

[0104]

[0105]

[0106] Since selecting tools from a number of similar tools to complete human instructions is a common task in indoor scenes such as kitchens, the embodiments of the present application can effectively improve the overall effect and versatility of command planning for embodied intelligent robots.

[0107] In summary, the embodiments of the present application can provide the large language model with more and more detailed information about surrounding objects, so that it can automatically plan and give a precise executable operation step sub-task at the execution level in response to high-level natural language instructions given by humans in complex indoor scenes, and ultimately give visual guidance for the execution of executable methods and operation areas, so that the given operation steps can be truly executed by the robot at the physical level.

[0108] According to the tool use command planning method based on a large language model and fine-tuning optimization applied to the online command planning stage proposed in the embodiment of the present application, after inputting human instructions and surrounding environment pictures, the basic process of the command planning method is first passed to generate the corresponding command plan, and ask the human whether the command plan is reasonable. If it is reasonable, the command plan is used as the final result to end the task; if it is unreasonable, the first level of visible exploration is performed, and the candidate tools are selected for a second time and the corresponding command plan is generated. At this time, continue to ask whether the human command plan is reasonable. If it is reasonable, the task is ended as the final result; if it is unreasonable, the second level of visible exploration is performed, and the local area obtained by the exploration is used as a new surrounding scene picture to re-plan the command. At this time, continue to ask whether the human command plan is reasonable. If it is reasonable, the task is ended as the final result; if it is unreasonable, invisible exploration is performed to generate invisible areas where the tools required to complete the human instructions may exist, for the embodied intelligent robot to further explore, and end the task. This application enables the robot to generate corresponding execution steps for any human instruction in complex unknown scenes, and can execute them efficiently and reasonably.

[0109] Fig.13 A flowchart of a command planning method for a tool based on a large language model and fine-tuning optimization used in an offline command planning stage provided in an embodiment of the present application.

[0110] like Fig.13 As shown, the command planning method used by the tool based on a large language model and fine-tuning optimization in the offline command planning stage includes the following steps:

[0111] In step S1301, based on the preset offline data collection strategy and retrieval enhancement generation strategy, the pre-stored planning information is obtained, and the target tool image in the planning information is retrieved, and the first similarity between the target tool image and the preset multiple target candidate tools is calculated respectively, and the optimal target candidate tool with the highest first similarity is obtained.

[0112] In step S1302, the retrieval enhancement generation strategy is used to retrieve the target operation area image corresponding to the optimal target candidate tool in the planning information, and the second similarity between the target operation area image and the preset multiple candidate areas is calculated respectively, and at least one target operation area image whose second similarity meets the preset similarity requirement is screened out, so that the retrieval enhancement generation strategy retrieves the corresponding first command planning information in the planning information according to at least one target operation area image, and determines whether the first command planning information meets the preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, the first command planning information is used as the final command planning.

[0113] In step S1303, if the first command planning information does not meet the preset reasonable requirements, the key area image corresponding to the optimal target candidate tool in the planning information is retrieved through the retrieval enhancement generation strategy, and the third similarities between the key area image and the multiple key area images corresponding to the multiple target candidate tools are calculated respectively, and based on the target candidate tool corresponding to the target key area image with the highest third similarity, the corresponding second command planning information is generated.

[0114] In step S1304, based on the second command planning information and preset reasonable requirements, a preset second level visible exploration and / or invisible exploration operation is performed to generate a final command plan.

[0115] Since GPT-4 is involved in many aspects of the aforementioned command planning method framework, such as candidate tool selection and command planning generation, although the aforementioned command planning method successfully uses the rich knowledge of GPT-4 to improve the accuracy of command planning, the operation of GPT-4 requires the setting of a proxy and online API calls, which results in the original command planning method running for a relatively long time and is relatively greatly affected by the network speed.

[0116] For the needs of offline operation and actual machine deployment, in addition to the online command planning framework involving GPT-4 introduced above, the embodiment of the present application also designs an offline command planning framework based on RAG. Its basic idea is: when facing unknown scenes and unknown tasks, first go through the online command planning framework introduced above (including the environmental exploration link) to store important information of command plans recognized by humans (such as appropriate tool labels, pictures, usage methods, etc. corresponding to human instructions); when there are enough stored command plans, facing the same human instructions, at this time, go through the "offline" command planning framework based on RAG retrieval, and use the results retrieved by RAG from the valid information of the stored correct command plans in each link to replace the results generated by GPT-4, so as to achieve the goal of shortening the running time and facilitating actual machine deployment.

[0117] Specifically, the overall framework of the RAG-based offline command planning method designed in the embodiment of the present application is Figure 2 , Fig. 9 The online command planning frameworks represented are basically the same, all of which include a priori knowledge acquisition module, an environmental target detection module, an execution step generation module, and a subsequent three-level environmental exploration link. The difference lies in the specific implementation methods of each main module and link.

[0118] For example, for the prior knowledge acquisition module, the offline framework directly obtains the required tool labels and other key information through RAG retrieval instead of GPT-4 generation; during the execution of the environmental target detection module, the offline framework retrieves the stored images of suitable tools through RAG, and calculates the similarity with the candidate tools respectively, so that the tool with the highest similarity is selected as the most suitable tool, replacing the selection process of GPT-4; in the execution step generation module, the offline framework also retrieves the images of the operating area of ​​the suitable tool and calculates the similarity with the candidate areas respectively, selects several key areas of the tool (such as the handle and blade of the knife, etc.), and generates the corresponding command planning according to the corresponding command planning method retrieved, replacing the command planning generation process of GPT-4; for the environmental exploration process, since the parts in which GPT-4 participates in the second-level visible exploration and invisible exploration are small and non-critical, the two exploration methods basically follow the design in the online framework.

[0119] For the first-level visible exploration, the offline framework uses a similarity comparison method for the key areas of tool function (such as the blade of a knife and the head of a colander) to compare the corresponding key areas of the stored suitable tools with the key areas of the candidate tools. The candidate tool corresponding to the key area with the highest similarity is the selected suitable tool, replacing the secondary selection of the tool by GPT-4 in the "online" framework.

[0120] In addition, the design of the command planning method in the embodiment of the present application plans to first pass through the online framework involving GPT-4 to store the key information of the correct and reasonable command planning corresponding to the unknown task; and then pass through the offline framework with RAG retrieval as the core, through the retrieval and comparison method, to generate the command planning for the "seen" task. From the above description, it can be seen that the online framework stage stores and prepares the correct command planning, which is similar to the training process; and the offline framework stage uses the stored command planning results for the tasks that have been seen in unknown scenes to perform faster and more accurate command planning, which is similar to the testing process. During the training process of the online framework stage, the requirements for the surrounding scene are: the size of the candidate tool is as large as possible, as clear and complete as possible, and there is no requirement for the complexity of the scene; and during the testing process of the offline framework stage, the requirements for the surrounding scene are: reflect the situation of the real indoor scene as much as possible.

[0121] Since the designed command planning method has different requirements for scene images at different stages, the dataset is constructed in different ways. The above has explained the construction method of the surrounding scene dataset in the online command planning framework stage, and the following will explain the construction of the dataset in the offline command planning framework stage.

[0122] In the offline command planning framework stage, the dataset is constructed using two methods: generation through DALL-E 3 and real-machine shooting. Examples of indoor scene pictures generated by DALL-E 3 are shown below: Fig.14 As shown by Fig.14 It can be seen that after inputting appropriate prompt words, the indoor scene pictures generated by DALL-E 3 are true to life and also contain common objects in indoor scenes such as cups, chairs, sofas, table lamps, etc., which can meet the needs of offline framework testing; the indoor scenes obtained by real machine shooting are as follows Fig.15 As shown, as a real indoor scene, it meets the requirements for offline framework testing.

[0123] Therefore, compared with the embodied intelligent work involving existing large models, the command planning offline framework in the embodiment of the present application significantly reduces the running time and improves the execution accuracy, forms a sparse few-sample database to achieve generalization on multiple tasks, and facilitates offline deployment on real machines.

[0124] According to the tool usage command planning method based on a large language model and fine-tuning optimization applied to the offline command planning stage proposed in the embodiment of the present application, when facing unknown scenarios and unknown tasks, the important information of the command plan recognized by humans is first passed through the online command planning framework (including the environment exploration link) to store the important information of the command plan (such as appropriate tool labels, pictures, usage methods, etc. corresponding to human instructions); when there are enough stored command plans, when facing the same human instructions, the offline command planning framework based on RAG retrieval is used again, and the results retrieved by RAG from the valid information of the stored correct command plan are used in each link to replace the results generated by GPT-4, so as to achieve the goal of shortening the running time and facilitating actual machine deployment.

[0125] Secondly, the tool usage command planning device based on a large language model and fine-tuning optimization proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0126] Fig.16 It is a block diagram of a command planning device used in an online command planning stage of a tool based on a large language model and fine-tuning optimization according to an embodiment of the present application.

[0127] like Fig.16 As shown, the tool-based command planning device 10 applied to the online command planning stage based on a large language model and fine-tuning optimization includes: a command planning module 101, a first exploration module 102 and a second exploration module 103.

[0128] Among them, the command planning module 101 is used to obtain target human instructions, and based on a preset online data collection strategy, generate a target surrounding environment picture, and perform command planning operations according to a preset large language model, target human instructions and target surrounding environment picture to generate a target candidate tool and a first command plan in the target surrounding environment picture, and determine whether the first command plan meets the preset reasonable requirements. If the first command plan meets the preset reasonable requirements, the first command plan is used as the final command plan.

[0129] The first exploration module 102 is used to perform a first level visible exploration operation on the target candidate tool if the first command plan does not meet the preset reasonable requirements, so as to generate a first local exploration area and a second command plan corresponding to the local exploration area, and determine whether the second command plan meets the preset reasonable requirements. If the second command plan meets the preset reasonable requirements, the second command plan is used as the final command plan.

[0130] The second exploration module 103 is used to perform a second-level visible exploration operation on the first local exploration area to obtain a second local exploration area if the second command plan does not meet the preset reasonable requirements, and perform a command planning operation on the second local exploration area to generate a third command plan corresponding to the second local exploration area, and judge whether the third command plan meets the preset reasonable requirements. If the third command plan does not meet the preset reasonable requirements, control the target embodied intelligent robot to perform an invisible exploration operation to generate a target invisible area, and explore the target invisible area through the target embodied intelligent robot. If the third command plan meets the preset reasonable requirements, use the third command plan as the final command plan, and obtain and store the planning information corresponding to the final command plan, so as to use the planning information to perform an offline command plan generation operation in the offline command planning stage.

[0131] Optionally, in one embodiment of the present application, the command planning module 101 includes: an online data acquisition unit, a first acquisition unit, a coarse screening unit, a fine screening unit and a segmentation unit.

[0132] Among them, the online data acquisition unit is used to generate a target tool label and a target tool appearance property corresponding to the target human instruction based on the target human instruction and the large language model, and randomly select an initial image and multiple interference images in a preset data set according to the target tool label, and randomly splice the initial image and the multiple interference images to obtain a target surrounding environment image.

[0133] The first acquisition unit is used to input the target tool label into a preset visual target detection model to generate different candidate tool schematics, and obtain the candidate tool confidences corresponding to different candidate tools based on the target surrounding environment picture, different candidate tool schematics and the visual target detection model.

[0134] The coarse screening unit is used to sort the confidences of the candidate tools in descending order to obtain a sorting result, and screen out multiple target candidate tool confidences that meet preset confidence requirements based on the sorting result, and obtain the sorting labels of the multiple target candidate tool confidences in the sorting result and the target candidate tools corresponding to the multiple target candidate tool confidences.

[0135] The fine screening unit is used to perform background filtering and tool reorganization operations on the target candidate tools to obtain a reorganized candidate tool graph, and input the reorganized candidate tool graph, the target tool label and the target tool appearance properties into the large language model to screen out the target ranking label corresponding to the target human instruction.

[0136] The segmentation unit is used to segment and filter the target candidate tools corresponding to the target sorting labels based on a preset segmentation model to obtain a corresponding segmentation map, and input the segmentation map into the large language model to generate a first command plan.

[0137] Optionally, in one embodiment of the present application, the first exploration module 102 includes: a second acquisition unit, an adding unit and a filtering unit.

[0138] Among them, the second acquisition unit is used to screen the first N target candidate tool confidences among multiple target candidate tool confidences according to the sorting result, and obtain the target candidate tools corresponding to the first N target candidate tool confidences, wherein N is a positive integer.

[0139] The adding unit is used to add the target candidate tools corresponding to the target ranking number to the target candidate tools corresponding to the confidence levels of the first N target candidate tools to obtain a plurality of target candidate tools to be screened.

[0140] The filtering unit is used to perform splicing and background filtering operations on multiple candidate target tools to be screened to generate a first local exploration area, and input the first local exploration area into the large language model to generate a second command plan.

[0141] Optionally, in one embodiment of the present application, the second exploration module 103 includes: a third acquisition unit, a weight accumulation unit, a fourth acquisition unit and a generation unit.

[0142] Among them, the third acquisition unit is used to filter out multiple candidate tool confidences corresponding to the target surrounding environment pictures that meet the preset sorting label requirements from the sorting results, and obtain multiple target candidate tools corresponding to the multiple candidate tool confidences.

[0143] The weight accumulation unit is used to weight the targets to be detected in the target area of ​​each target candidate tool among multiple target candidate tools to obtain the weight accumulation result corresponding to each target candidate tool, and compare the weight accumulation result of each target candidate tool to obtain the maximum value of the weight accumulation result.

[0144] The fourth acquisition unit is used to acquire the target candidate tool corresponding to the maximum value of the weight accumulation result, and use the target area of ​​the target candidate tool corresponding to the maximum value of the weight accumulation result as the second local exploration area.

[0145] A generating unit is used to input the second local exploration area into the large language model to generate a third command plan.

[0146] It should be noted that the aforementioned explanation of the embodiment of the tool command planning method based on a large language model and fine-tuning optimization applied to the online command planning stage is also applicable to the tool command planning device based on a large language model and fine-tuning optimization applied to the online command planning stage, and will not be repeated here.

[0147] According to the tool use command planning device based on a large language model and fine-tuning optimization applied to the online command planning stage proposed in the embodiment of the present application, after inputting human instructions and surrounding environment pictures, the basic process of the command planning method is first passed to generate the corresponding command plan, and ask the human whether the command plan is reasonable. If it is reasonable, the command plan is used as the final result to end the task; if it is unreasonable, the first level of visible exploration is performed, and the candidate tools are selected for a second time and the corresponding command plan is generated. At this time, continue to ask whether the human command plan is reasonable. If it is reasonable, the task is ended as the final result; if it is unreasonable, the second level of visible exploration is performed, and the local area obtained by the exploration is used as a new surrounding scene picture to re-plan the command. At this time, continue to ask whether the human command plan is reasonable. If it is reasonable, the task is ended as the final result; if it is unreasonable, invisible exploration is performed to generate invisible areas where the tools required to complete the human instructions may exist, for the embodied intelligent robot to further explore, and end the task. This application enables the robot to generate corresponding execution steps for any human instruction in complex unknown scenes, and can execute them efficiently and reasonably.

[0148] Fig.17 It is a block diagram of a command planning device used in an offline command planning stage using a tool based on a large language model and fine-tuning optimization according to an embodiment of the present application.

[0149] like Fig.17 As shown, the tool use command planning device 20 based on a large language model and fine-tuning optimization applied to the offline command planning stage includes: a retrieval module 201, a first calculation module 202, a second calculation module 203 and an offline exploration module 204.

[0150] Among them, the retrieval module 201 is used to obtain pre-stored planning information based on a preset offline data collection strategy and a retrieval enhancement generation strategy, and retrieve the target tool image in the planning information, and respectively calculate the first similarity between the target tool image and a plurality of preset target candidate tools, and obtain the optimal target candidate tool with the highest first similarity.

[0151] The first calculation module 202 is used to use the retrieval enhancement generation strategy to retrieve the target operation area image corresponding to the optimal target candidate tool in the planning information, and respectively calculate the second similarity between the target operation area image and the preset multiple candidate areas, and screen out at least one target operation area image whose second similarity meets the preset similarity requirement, so that the retrieval enhancement generation strategy retrieves the corresponding first command planning information in the planning information according to at least one target operation area image, and determines whether the first command planning information meets the preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, the first command planning information is used as the final command planning.

[0152] The second calculation module 203 is used to retrieve the key area image corresponding to the optimal target candidate tool in the planning information through the retrieval enhancement generation strategy if the first command planning information does not meet the preset reasonable requirements, and respectively calculate the third similarities between the key area image and multiple key area images corresponding to the multiple target candidate tools, and generate the corresponding second command planning information based on the target candidate tool corresponding to the target key area image with the highest third similarity.

[0153] The offline exploration module 204 is used to perform preset second-level visible exploration and / or invisible exploration operations based on the second command planning information and preset reasonable requirements to generate a final command plan.

[0154] It should be noted that the aforementioned explanation of the embodiment of the tool use command planning method based on a large language model and fine-tuning optimization applied to the offline command planning stage is also applicable to the tool use command planning device based on a large language model and fine-tuning optimization applied to the offline command planning stage, and will not be repeated here.

[0155] According to the tool use command planning device based on a large language model and fine-tuning optimization applied to the offline command planning stage proposed in the embodiment of the present application, when facing unknown scenes and unknown tasks, it first passes through an online command planning framework (including an environmental exploration phase) to store important information of command plans recognized by humans (such as appropriate tool labels, pictures, usage methods, etc. corresponding to human instructions); when there are enough stored command plans, when facing the same human instructions, it passes through an offline command planning framework based on RAG retrieval, and uses RAG to retrieve the results from the valid information of the stored correct command plans in each link, instead of the results generated by GPT-4, so as to achieve the goal of shortening the running time and facilitating actual machine deployment.

[0156] Fig.18 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0157] A memory 1801 , a processor 1802 , and a computer program stored in the memory 1801 and executable on the processor 1802 .

[0158] When the processor 1802 executes the program, the tool using command planning method based on the large language model and fine-tuning optimization provided in the above embodiment is implemented.

[0159] Furthermore, the electronic device further comprises:

[0160] The communication interface 1803 is used for communication between the memory 1801 and the processor 1802 .

[0161] The memory 1801 is used to store computer programs that can be executed on the processor 1802 .

[0162] The memory 1801 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0163] If the memory 1801, the processor 1802 and the communication interface 1803 are implemented independently, the communication interface 1803, the memory 1801 and the processor 1802 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.18Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0164] Optionally, in a specific implementation, if the memory 1801, the processor 1802 and the communication interface 1803 are integrated on a chip, the memory 1801, the processor 1802 and the communication interface 1803 can communicate with each other through an internal interface.

[0165] The processor 1802 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0166] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the tool-use command planning method based on a large language model and fine-tuning optimization as described above.

[0167] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned tool usage command planning method based on a large language model and fine-tuning optimization.

[0168] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0169] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0170] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0171] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0172] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0173] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0175] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A tool-use command planning method based on a large language model and fine-tuning optimization, applied to the online command planning stage, characterized in that: The following steps are involved: Obtaining a target human instruction, and based on a preset online data collection strategy, generating a target surrounding environment picture, and performing a command planning operation according to a preset large language model, the target human instruction, and the target surrounding environment picture to generate a target candidate tool and a first command plan in the target surrounding environment picture, and determining whether the first command plan meets preset reasonable requirements, and if the first command plan meets the preset reasonable requirements, using the first command plan as the final command plan; If the first command plan does not meet the preset reasonable requirements, a first level visible exploration operation is performed on the target candidate tool to generate a first local exploration area and a second command plan corresponding to the local exploration area, and it is determined whether the second command plan meets the preset reasonable requirements. If the second command plan meets the preset reasonable requirements, the second command plan is used as the final command plan; If the second command plan does not meet the preset reasonable requirements, a second-level visible exploration operation is performed on the first local exploration area to obtain a second local exploration area, and a command planning operation is performed on the second local exploration area to generate a third command plan corresponding to the second local exploration area, and it is determined whether the third command plan meets the preset reasonable requirements. If the third command plan does not meet the preset reasonable requirements, the target embodied intelligent robot is controlled to perform an invisible exploration operation to generate a target invisible area, and the target invisible area is explored by the target embodied intelligent robot. If the third command plan meets the preset reasonable requirements, the third command plan is used as the final command plan, and the planning information corresponding to the final command plan is obtained and stored, so as to use the planning information to perform an offline command plan generation operation in the offline command planning stage.

2. The method according to claim 1, characterized in that The step of acquiring the target human instruction and generating a target surrounding environment picture based on a preset online data collection strategy, and performing a command planning operation according to a preset large language model, the target human instruction and the target surrounding environment picture to generate a target candidate tool and a first command plan in the target surrounding environment picture includes: Based on the target human instruction and the large language model, a target tool label and a target tool appearance property corresponding to the target human instruction are generated, and according to the target tool label, an initial picture and multiple interference pictures are randomly selected from a preset data set, and a random splicing operation is performed on the initial picture and the multiple interference pictures to obtain the target surrounding environment picture; Inputting the target tool label into a preset visual target detection model to generate different candidate tool schematics, and obtaining candidate tool confidences corresponding to different candidate tools according to the target surrounding environment picture, the different candidate tool schematics and the visual target detection model; Sorting the candidate tool confidences in descending order to obtain a sorting result, and screening out multiple target candidate tool confidences that meet preset confidence requirements according to the sorting result, and obtaining the sorting labels of the multiple target candidate tool confidences in the sorting result and the target candidate tools corresponding to the multiple target candidate tool confidences; Performing background filtering and tool reorganization operations on the target candidate tool to obtain a reorganized candidate tool graph, and inputting the reorganized candidate tool graph, the target tool label, and the target tool appearance properties into the large language model to filter out a target ranking label corresponding to the target human instruction; Based on a preset segmentation model, segmentation and filtering operations are performed on the target candidate tools corresponding to the target sorting labels to obtain corresponding segmentation maps, and the segmentation maps are input into the large language model to generate the first command plan.

3. The method according to claim 2, characterized in that The performing a first level visible exploration operation on the target candidate tool to generate a first local exploration area and a second command plan corresponding to the local exploration area includes: Filtering the top N target candidate tool confidences among the multiple target candidate tool confidences according to the ranking result, and acquiring the target candidate tools corresponding to the top N target candidate tool confidences, where N is a positive integer; Adding the target candidate tools corresponding to the target ranking number to the target candidate tools corresponding to the confidences of the first N target candidate tools to obtain a plurality of target candidate tools to be screened; The plurality of candidate target tools to be screened are concatenated and background filtered to generate the first local exploration area, and the first local exploration area is input into the large language model to generate the second command plan.

4. The method according to claim 3, characterized in that The performing a second level visible exploration operation on the first local exploration area to obtain a second local exploration area, and performing a command planning operation on the second local exploration area to generate a third command plan corresponding to the second local exploration area, includes: Filtering out multiple candidate tool confidences corresponding to the target surrounding environment picture that meet the preset sorting label requirements from the sorting results, and obtaining multiple target candidate tools corresponding to the multiple candidate tool confidences; Accumulating weights of the targets to be detected in the target area of ​​each target candidate tool among the multiple target candidate tools to obtain a weight accumulation result corresponding to each target candidate tool, and comparing the weight accumulation results of each target candidate tool to obtain a maximum value of the weight accumulation results; Acquire the target candidate tool corresponding to the maximum value of the weight accumulation result, and use the target area of ​​the target candidate tool corresponding to the maximum value of the weight accumulation result as the second local exploration area; The second local exploration region is input into the large language model to generate the third command plan.

5. A tool-use command planning method based on a large language model and fine-tuning optimization, applied to the offline command planning stage, characterized in that: The following steps are involved: Based on a preset offline data collection strategy and a retrieval enhancement generation strategy, pre-stored planning information is obtained, and a target tool image in the planning information is retrieved, and a first similarity between the target tool image and a plurality of preset target candidate tools is respectively calculated, and an optimal target candidate tool having the highest first similarity is obtained; The target operation area image corresponding to the optimal target candidate tool in the planning information is retrieved by using the retrieval enhancement generation strategy, and the second similarity between the target operation area image and the preset multiple candidate areas is calculated respectively, and at least one target operation area image whose second similarity meets the preset similarity requirement is screened out, so that the retrieval enhancement generation strategy retrieves the corresponding first command planning information in the planning information according to the at least one target operation area image, and determines whether the first command planning information meets the preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, the first command planning information is used as the final command planning; If the first command planning information does not meet the preset reasonable requirement, the key area picture corresponding to the optimal target candidate tool in the planning information is retrieved through the retrieval enhancement generation strategy, and the third similarities between the key area picture and the multiple key area pictures corresponding to the multiple target candidate tools are respectively calculated, and based on the target candidate tool corresponding to the target key area picture with the highest third similarity, the corresponding second command planning information is generated; Based on the second command planning information and the preset reasonable requirements, a preset second-level visible exploration and / or invisible exploration operation is performed to generate a final command plan.

6. A tool-use command planning device based on a large language model and fine-tuning optimization, applied to the online command planning stage, characterized in that: include: A command planning module, used to obtain a target human instruction, generate a target surrounding environment picture based on a preset online data collection strategy, and perform a command planning operation according to a preset large language model, the target human instruction and the target surrounding environment picture to generate a target candidate tool and a first command plan in the target surrounding environment picture, and determine whether the first command plan meets a preset reasonable requirement. If the first command plan meets the preset reasonable requirement, the first command plan is used as the final command plan; A first exploration module is used for performing a first level visible exploration operation on the target candidate tool to generate a first local exploration area and a second command plan corresponding to the local exploration area if the first command plan does not meet the preset reasonable requirements, and judging whether the second command plan meets the preset reasonable requirements, and if the second command plan meets the preset reasonable requirements, taking the second command plan as the final command plan; The second exploration module is used to perform a second-level visible exploration operation on the first local exploration area to obtain a second local exploration area if the second command plan does not meet the preset reasonable requirements, and perform a command planning operation on the second local exploration area to generate a third command plan corresponding to the second local exploration area, and judge whether the third command plan meets the preset reasonable requirements; if the third command plan does not meet the preset reasonable requirements, control the target embodied intelligent robot to perform an invisible exploration operation to generate a target invisible area, and explore the target invisible area through the target embodied intelligent robot; if the third command plan meets the preset reasonable requirements, use the third command plan as the final command plan, and obtain and store the planning information corresponding to the final command plan, so as to use the planning information to perform an offline command plan generation operation in the offline command planning stage.

7. A tool-use command planning device based on a large language model and fine-tuning optimization, applied to the offline command planning stage, characterized in that: include: A retrieval module, configured to obtain pre-stored planning information based on a preset offline data collection strategy and a retrieval enhancement generation strategy, retrieve a target tool image in the planning information, respectively calculate a first similarity between the target tool image and a plurality of preset target candidate tools, and obtain an optimal target candidate tool having the highest first similarity; A first calculation module, used to use the retrieval enhancement generation strategy to retrieve the target operation area image corresponding to the optimal target candidate tool in the planning information, and respectively calculate the second similarity between the target operation area image and the preset multiple candidate areas, and screen out at least one target operation area image whose second similarity meets the preset similarity requirement, so that the retrieval enhancement generation strategy retrieves the corresponding first command planning information in the planning information according to the at least one target operation area image, and determines whether the first command planning information meets the preset reasonable requirements. If the first command planning information meets the preset reasonable requirements, the first command planning information is used as the final command planning; A second calculation module is used for retrieving the key area picture corresponding to the optimal target candidate tool in the planning information through the retrieval enhancement generation strategy if the first command planning information does not meet the preset reasonable requirement, and respectively calculating the third similarity between the key area picture and the multiple key area pictures corresponding to the multiple target candidate tools, and generating the corresponding second command planning information based on the target candidate tool corresponding to the target key area picture with the highest third similarity; The offline exploration module is used to perform a preset second-level visible exploration and / or invisible exploration operation based on the second command planning information and the preset reasonable requirements to generate a final command plan.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the tool usage command planning method based on a large language model and fine-tuning optimization as described in any one of claims 1 to 4 or claim 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement a tool usage command planning method based on a large language model and fine-tuning optimization as described in any one of claims 1 to 4 or claim 5.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the tool usage command planning method based on a large language model and fine-tuning optimization as described in any one of claims 1 to 4 or claim 5.

Images