Unmanned equipment autonomous task planning and execution method based on large model agent framework

Abstract

The invention provides an unmanned equipment autonomous task planning and execution method based on a large model agent framework. The method comprises the following steps: constructing an agent meeting a custom task requirement, and generating a prompt template for the agent; for a natural language task requirement input by a user, generating a multi-dimensional cue word through the constructed cue template of the intelligent agent; the cue words are input into the large model, and unstructured decision step information is sequentially output according to decision steps; a feedback mechanism is constructed, and the user confirms or adjusts the decision step information to obtain feedback information; combining the feedback information with the cue word to form a new cue word, and repeatedly iterating until the user confirms to obtain a decision result; and performing structured analysis according to a decision result, and completing corresponding decision steps, namely completing task execution of the unmanned equipment. According to the invention, powerful technical support is provided for wide application of unmanned equipment in the fields of industry, rescue and the like.

Description

Technical Field

[0001] The present invention relates to an unmanned equipment autonomous task planning and execution method, in particular to an unmanned equipment autonomous task planning and execution method based on a large model intelligent body framework. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] With the rapid development of artificial intelligence technology, intelligent unmanned equipment systems have become an important part of modern science and technology. However, most of the existing intelligent unmanned equipment systems rely on a single model or fixed algorithm, which is difficult to meet the complex and changing task requirements. At the same time, in the creation and configuration process of unmanned equipment systems, there are often problems such as complex operation and low efficiency, which limits the widespread application of intelligent unmanned systems.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide an unmanned equipment autonomous task planning and execution method based on a large model intelligent agent framework in response to the deficiencies in the prior art.

[0006] In order to solve the above technical problems, the present invention discloses an unmanned equipment autonomous task planning and execution method based on a large model intelligent agent framework, comprising the following steps:

[0007] Step 1, agent creation and configuration, setting the name, introduction, type, large model, role prompt, knowledge base and tool set of the agent, building an agent that meets the customized task requirements; generating a prompt template for the agent, the prompt template includes: role prompt, output format requirements, candidate tool set, task description, execution process and user feedback;

[0008] Step 2: task requirement analysis: for the natural language task requirement input by the user, a multi-dimensional prompt word is generated through the prompt template of the intelligent agent constructed in step 1;

[0009] Step 3, input the prompt words generated in step 2 into the large model, and output unstructured decision step information in sequence according to the decision steps, wherein each decision step information includes the decision process of the large model, the tools to be used and their parameters;

[0010] Step 4: Based on the unstructured decision step information obtained in step 3, a feedback mechanism is constructed, and the user confirms or adjusts the decision step information in step 3 to obtain feedback information; the above feedback information is combined with the prompt word in step 2 to form a new prompt word and steps 3 to 4 are repeatedly executed iteratively until the user confirms and obtains the decision result;

[0011] In step 5, the intelligent agent performs structured analysis based on the decision result obtained in step 4, extracts the parameters required for each tool, carries out intelligent agent tool set calling and instruction generation, and completes the corresponding decision steps, thereby completing the task execution of the unmanned equipment.

[0012] Furthermore, the feedback mechanism described in step 5 specifically includes:

[0013] According to the settings in step 1, the agent performs tasks in two operation modes: human-in-the-loop HITL and human-out-of-the-loop HOTL.

[0014] Furthermore, the agent tool set calling and instruction generation described in step 5 specifically include:

[0015] Toolset interface standardization, dynamic toolset selection and calling, toolset feedback and optimization, parameter extraction and instruction generation.

[0016] Furthermore, the creation and configuration of the agent described in step 1 specifically includes:

[0017] Step 1-1, the user inputs the name and introduction of the agent to be created, and generates an agent ID as the unique identifier of the agent. The introduction is used to describe the basic functions and application scenarios of the agent;

[0018] Step 1-2, the user selects the type of the agent, which includes question-answering type, instruction type, accusation type and equipment type;

[0019] Steps 1-3: Users select from multiple candidate large models based on task requirements;

[0020] Step 1-4, in the role prompt setting, the user enters a custom role prompt or selects the initial role prompt generated by the large model, and adjusts it to meet the task requirements;

[0021] Steps 1-5, the user can associate the agent with a local knowledge base, which provides supporting data for task execution;

[0022] Step 1-6, the user selects the agent mode according to the specific task, the mode includes intelligent selection mode, process arrangement mode and autonomous planning mode, wherein when the mode is autonomous planning mode, the human-out-of-the-loop HOTL mode or the human-in-the-loop HITL mode is further selected;

[0023] In steps 1-7, the user needs to select and associate a toolset for the agent. The toolset is pre-set by the user, and the agent automatically calls the tools in the toolset when performing a task.

[0024] Furthermore, the task requirement analysis described in step 2 specifically includes:

[0025] Step 2-1, the user inputs task requirements in natural language to the agent; wherein the requirements include specific task objectives, operation constraints and expected results;

[0026] Step 2-2, the agent processes the input task requirements using natural language processing (NLP) methods to identify key task elements; wherein the processing includes word segmentation, grammatical parsing, and semantic analysis;

[0027] Step 2-3: Based on the key task elements identified in step 2-2, the intelligent agent parses and generates multi-dimensional prompt words in combination with the prompt template to guide the reasoning process of the large model.

[0028] Furthermore, the generation of multi-dimensional prompt words in step 2-3 specifically includes:

[0029] Using key task elements, the multi-dimensional prompt words are expressed as follows:

[0030] I={I r ,I t ,I h ,I p ,I d ,I f}

[0031] Where: I r is the role description of the intelligent agent, I t Describe the candidate toolset. h For historical mission records, I p is the current task description, I d It is the current task execution process record, I f Provide feedback to users;

[0032] Agent Role Description I r , used to detail the agent’s position and role in the task;

[0033] Candidate Toolset Description I t, which lists the available tools and their functions related to the current task;

[0034] Historical Mission Record I h , used to summarize the records of similar tasks in the past, including task content, execution methods and results;

[0035] Current Task Description I p , used to describe the current task information;

[0036] Current task execution process record I d , used to provide the execution status of the previous decision step under the current task, including the execution content and execution results, etc., for reference by the next decision step;

[0037] User FeedbackI f , used to record the feedback given by users during task execution.

[0038] Furthermore, the unstructured decision step information is outputted sequentially according to the decision steps described in step 3, specifically including:

[0039] The prompt words generated in step 2 are input into the large model to make a decision, and the unstructured text decision results including the task execution plan, tool calling sequence and corresponding tool parameters are output.

[0040] Furthermore, the feedback mechanism described in step 4 specifically includes:

[0041] Step 4-1, the agent performs the task in the human-in-the-loop HITL operation mode, as follows:

[0042] Step 4-1-1, real-time reasoning process display, graphically displaying the reasoning path and current execution status of the large model, including decision nodes, logical judgments and parameter changes;

[0043] Step 4-1-2, execution feedback visualization, combined with real-time feedback, to show the progress and results of the current task execution;

[0044] Step 4-1-3, user feedback integration, receives feedback from users, and adjusts the reasoning direction and execution parameters according to the feedback content. The user feedback processing logic is as follows:

[0045]

[0046] Step 4-1-4, if the current decision is in line with expectations, the user provides confirmation feedback, that is, confirms the current single step, or confirms the next N steps and enters the HOTL mode in the next N steps;

[0047] Step 4-2, the agent performs the task in the human-out-of-the-loop HOTL operation mode, as follows:

[0048] Step 4-2-1: autonomous reasoning and decision-making, using large models to automatically perform task decisions;

[0049] Step 4-2-2, anomaly detection and processing, automatically identify and process anomalies or deviations in the execution process;

[0050] Step 4-2-3, automatic mode switching. When an abnormal situation occurs, it will actively switch to the human-in-the-loop HITL mode and invite users to participate in decision-making. The abnormal handling logic is as follows:

[0051]

[0052] Step 4-3: Establish a backtracking and optimization mechanism. During the reasoning and execution process, if a reasoning error or a logical dead loop is found, automatically backtrack to the previous step to reselect the path and optimize the solution. The backtracking and optimization logic is as follows:

[0053]

[0054] Further, the structural analysis described in step 5 specifically includes:

[0055] Step 5-1: Use natural language processing methods to semantically understand the decision results in the form of unstructured text and identify key task elements;

[0056] Step 5-2, extracting and converting the decision step information, tool calling sequence and corresponding tool parameters in the decision result into structured data; the structured decision data I l as follows:

[0057] I l ={S,T,T p}

[0058] Among them, S is the decision step information, which indicates the specific operations required for each step; T is the tool call plan, which indicates the tool set and call order required; T p is the specific parameter value of the required tool;

[0059] Step 5-3, match the corresponding toolset and calling strategy according to the decision step information.

[0060] Furthermore, the agent tool set calling and instruction generation described in step 5 specifically include:

[0061] Standardize tool set interfaces and design common interface protocols to support the processing and analysis of multimodal data types such as text, images, audio, and video;

[0062] Dynamic toolset selection and calling: select toolsets to assist decision making and execution based on task requirements and current resource status;

[0063] Toolset feedback and optimization: During the toolset invocation process, tool execution feedback is collected and tool selection and invocation strategies are dynamically adjusted based on the feedback results.

[0064] Based on the decision results, the task parameters are extracted and specific control instructions are generated. The generated instructions are sent to the unmanned equipment, and its execution status is monitored in real time. During the execution of the task by the unmanned equipment, feedback information is collected and the instruction generation and sending strategy is adjusted accordingly.

[0065] Beneficial effects:

[0066] This application introduces a large model as the core decision engine of the intelligent agent, combined with a tool set and custom rules, to achieve rapid creation and flexible configuration of the intelligent agent. This method not only improves the adaptability and intelligence level of the intelligent unmanned equipment system, but also simplifies the creation and configuration process, reducing the difficulty and cost of operation.

[0067] The method framework for autonomous task planning and execution of unmanned equipment based on large model intelligent agents proposed in this application aims to improve the autonomous decision-making and execution capabilities of unmanned equipment in complex environments. The framework integrates core technologies such as multi-level intent understanding, prompt template generation, large and small model collaborative scheduling, chain reasoning decision, human-machine feedback mechanism, and intelligent tool set call, realizing a complete closed loop from user task input to autonomous execution of unmanned equipment. The innovation lies in that through hierarchical intent parsing and prompt template guidance, complex tasks are decomposed and efficiently allocated to large and small models for collaborative execution, so that the system can quickly respond to dynamic environmental changes and optimize task planning and execution paths. In addition, by introducing a human-machine feedback mechanism, intelligent adjustments are achieved during task execution, which improves the adaptability and robustness of the system. This innovative framework not only improves the efficiency and accuracy of task completion of unmanned equipment, but also provides an important reference for the autonomous development of future intelligent systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0069] Figure 1 It is the overall architecture diagram of the present invention.

[0070] Figure 2 The figure is a schematic diagram of the process of autonomous mission planning and execution of unmanned equipment according to the method of the present invention. DETAILED DESCRIPTION

[0071] This paper proposes an autonomous task planning and execution method for unmanned equipment based on a large model intelligent framework, aiming to improve the autonomous decision-making and task execution capabilities of unmanned equipment in complex environments. This technology integrates core technologies such as multi-level intention understanding, chain thinking reasoning and human-machine feedback decision-making, and device control instruction services based on ROS (Robot Operating System) by efficiently integrating large model capabilities, tool sets and custom rules, providing users with customized and intelligent unmanned system solutions, and improving the autonomous decision-making capabilities and execution efficiency of unmanned equipment in complex task environments.

[0072] This technical solution is Figure 1 As shown, it includes the following key steps:

[0073] S1, Agent creation and configuration, aims to provide users with a customized platform, allowing users to build agent instances that meet specific task requirements and ensure that agents can perform tasks efficiently in complex environments. The creation and configuration steps include the settings of the agent's name, introduction, type, large model, role prompts, knowledge base, and toolset.

[0074] First, the user enters the agent name and introduction through a graphical user interface or application program interface (API) to generate an agent ID as the unique identifier of the agent. The introduction is used to describe the basic functions and application scenarios of the agent. Then, the user can select the type of agent, which includes question-answering type, instruction type, accusation type and equipment type to meet the needs of different tasks.

[0075] The intelligent agent further supports large model selection. The system provides multiple optional large models, such as Wenxin Yiyan, Tongyi Qianwen, etc. Users can select the appropriate large model according to task requirements to ensure that the intelligent agent has the best performance.

[0076] In the role prompt setting, users can enter a custom role prompt or select the initial role prompt generated by the large model and adjust it to meet the needs of specific tasks. The prompt is used to guide the large model to clarify the role, so as to generate the corresponding output results in combination with the specific task.

[0077] In addition, users can associate the agent with a local knowledge base, which provides support data for task execution. The agent has multiple tool usage modes, including intelligent selection, process orchestration, and autonomous planning. In the autonomous planning mode, you can further choose between human-out-of-the-loop (HOTL) and human-in-the-loop (HITL). Users choose the appropriate mode based on the specific task, which affects the tool invocation method and execution strategy.

[0078] Subsequently, the user needs to select and associate a toolset for the agent. The toolset is pre-set by the user, and the agent can automatically call the tools in the toolset when performing a task to meet specific task requirements.

[0079] Finally, based on the above settings, the system automatically generates a prompt template for the agent, which includes framework information such as role prompts, output format requirements, candidate tool sets, task descriptions, execution process, and user feedback. After the user enters the task requirements, prompt words can be generated based on the specific task.

[0080] S2. Task requirement analysis. The natural language task requirements entered by the user are analyzed, and multi-dimensional prompt words are generated through the intelligent agent prompt template to accurately guide the large model to make reasoning and decisions. This step involves the combination of natural language processing technology and intelligent prompt generation mechanism to ensure that the task requirements are fully and accurately understood.

[0081] First, the user inputs the task requirements in natural language to the agent through a graphical user interface or application programming interface (API). These requirements may involve specific task objectives, operational constraints, expected results, etc. The agent uses advanced natural language processing (NLP) algorithms (references: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N.Gomez, Kaiser, and IlliaPolosukhin.Attention is all you need.In Advances in neural information processing systems, 2017.) Perform word segmentation, grammatical parsing, semantic analysis and other processing steps on the input text to identify key task elements.

[0082] After analyzing the task requirements, the system will automatically generate a multi-dimensional prompt word. The purpose of this prompt word is to guide the reasoning process of the large model and ensure the accuracy and consistency of the decision results. The generated prompt word mainly includes the following key parts:

[0083] S21, Agent role description: Detailed description of the agent's position and role in the task. This part can help the big model understand the agent's function and its relationship with the user.

[0084] S22, Candidate toolset description: List the available tools and their functions related to the current task. The system automatically selects the appropriate toolset based on the task requirements to provide support for subsequent task execution.

[0085] S23. Historical task records: Summarize the records of similar tasks in the past, including task content, execution methods and results. These historical data help identify effective strategies and optimize current task decisions.

[0086] S24. Description of current task: Describe the background information and prerequisites of the current task, and clarify the environment and resources required to execute the task.

[0087] S25, past decisions / execution results: used to provide the execution status of previous decision steps under the current task, including execution content and execution results, etc., for reference by the next decision step.

[0088] S26. Feedback: includes previous feedback from users or systems, pointing out deficiencies and areas for improvement during task execution.

[0089] The prompt word structure is as follows:

[0090] T p = {R,C,H,D,P,F b}

[0091] Among them, R represents the agent role description, C is the candidate tool set description, H is the historical task record, D is the current task description, P is the past decision / execution result, and F is the b Express feedback.

[0092] S3: Input the prompt words generated in step S2 into the big model to generate a decision result in the form of unstructured text. This step is the core part of the agent's reasoning and decision-making process, and uses the powerful computing power and deep learning algorithm of the big model to achieve analysis and judgment of complex tasks.

[0093] The large model can be Wenxin Yiyan, Tongyi Qianwen or other pre-trained language models suitable for specific task requirements. The large model uses natural language understanding and generation capabilities to parse the input multi-dimensional prompts, consider the current task context and historical decision-making experience, and output an unstructured text decision result containing task solution suggestions, execution order, risk assessment, etc.

[0094] S4. The system performs structured analysis on the unstructured text decision results generated by the large model to extract the instructions that can be directly used for task execution and the tool set information required to be called. l .

[0095] S41. Use natural language processing algorithms to semantically understand unstructured text and identify key task elements.

[0096] S42, extracting and converting information such as the task execution plan and tool calling sequence described in the text into structured data.

[0097] S43. According to the task plan, match the corresponding tool set and calling strategy to ensure that the appropriate tools can be used during the task execution.

[0098] Furthermore, the parsed structured decision data includes:

[0099] I l = {S,T,t p}

[0100] Among them, S is the decision step information, which indicates the specific operations required for each step; T is the tool call plan, which indicates the tool set and call order required; T p is the specific parameter value of the required tool.

[0101] S5, Human-in-the-Loop feedback mechanism, combined with the settings of the user in step S1 in creating and configuring the intelligent agent, the intelligent agent can work in two operation modes: Human-in-the-Loop (HITL) and Human-out-of-the-Loop (HOTL) to enhance the flexibility of the decision-making process and the accuracy of task execution. Through these two modes, the system can dynamically adjust the task strategy during the task execution process and obtain user feedback when necessary to ensure that the task execution of the unmanned equipment is carried out in a controlled state.

[0102] S51, Human in the Loop (HITL) mode, the system provides users with a real-time visual interface that displays the large model reasoning process and small model execution feedback. Users can view the decision path, parameter settings, and intermediate results through this interface. Specific implementation includes:

[0103] S511, real-time reasoning process display: The system graphically displays the reasoning path and current execution status of the large model, including decision nodes, logical judgments and parameter changes.

[0104] S512. Visualization of execution feedback: Combined with the real-time feedback of the small model, users can see the progress and results of the current task execution, making it easier to discover potential problems.

[0105] S513, User feedback integration: After the user provides feedback in the interface, the system will adjust the reasoning direction and execution parameters according to the feedback content to ensure that the feedback information is reasonably integrated to optimize task execution.

[0106] User feedback processing logic:

[0107]

[0108] S52: In the human-out-of-the-loop (HOTL) mode, the system autonomously performs reasoning and decision-making, combining the strategy of the large model with the real-time feedback of the small model to achieve fully automated processing. The specific implementation includes:

[0109] S521, Autonomous reasoning and decision-making: The system uses the synergy of large and small models to automatically perform task decisions without human intervention.

[0110] S522, anomaly detection and processing: When the system detects anomalies or deviations in the execution process, it can automatically identify and process them, and switch to HITL mode when necessary.

[0111] S523, automatic mode switching: When an abnormal situation occurs, the system actively switches to HITL mode, inviting users to participate in decision-making to ensure that the task is carried out under control.

[0112] Exception handling logic:

[0113]

[0114] S53, backtracking and optimization. During the reasoning and execution process, if a reasoning error or a logical dead loop is found, the system will automatically backtrack to the previous step to reselect the path and optimize the solution. The specific implementation includes:

[0115] S531. Automatic backtracking mechanism: By monitoring the reasoning process, the system can automatically identify reasoning errors or logical dead loops and backtrack to the previous decision node.

[0116] S532, Path reselection and optimization: Combined with the feedback from the small model, the system reselects the reasoning path and adjusts the task execution plan to improve the robustness and accuracy of the overall task solution.

[0117] S533, Continuous Optimization: The system gradually optimizes the reasoning path and execution plan through continuous feedback and backtracking to ensure efficient completion of tasks.

[0118] Backtracking and optimization logic:

[0119]

[0120] S6, intelligent tool set call and command generation, to ensure that the unmanned equipment has efficient decision-making ability and accurate execution during the task execution process. This process includes multiple links such as tool set interface standardization, dynamic tool set selection and call, tool set feedback and optimization, parameter extraction and command generation.

[0121] S61. Standardization of tool set interface. The system has designed a unified tool set interface standard to support the integration and calling of various types of tools. The interface standardization ensures that the system can seamlessly interact with APIs, services or subsystem collections. Specific implementations include:

[0122] S611, Interface definition: Design common interface protocols to support the processing and analysis of multimodal data types such as text, images, audio and video.

[0123] S612, Multimodal Data Processing: The toolset has the ability to process multimodal data, providing comprehensive environmental perception and task execution support.

[0124] S613. Compatibility assurance: Interface standardization ensures the compatibility of the toolset with various external services and subsystems, enabling smooth integration and interaction.

[0125] S62, Dynamic toolset selection and calling, based on task requirements and current system resource status, the system dynamically selects the appropriate toolset to assist decision making and execution. The tool selection strategy combines factors such as task complexity, real-time performance requirements, and resource availability. The specific steps are as follows:

[0126] S621, Task analysis: Analyze task requirements and determine the type and quantity of tools required.

[0127] S622, resource assessment: select the optimal toolset based on the current system resource status (such as CPU load, memory usage).

[0128] S623, tool call: dynamically call the selected toolset and adjust the task execution strategy according to the tool output.

[0129] S63, Toolset feedback and optimization. During the toolset calling process, the system collects tool execution feedback and dynamically adjusts tool selection and calling strategies based on the feedback results to improve the toolset utilization efficiency and task execution accuracy.

[0130] The specific steps include:

[0131] S631, Feedback Collection: Monitor the execution status of the tool set and collect feedback information.

[0132] S632, Dynamic Adjustment: Adjust tool selection and calling strategies based on feedback information to improve execution efficiency.

[0133] S633, Strategy Optimization: By analyzing feedback data, continuously optimize the tool set calling strategy to achieve the best execution effect.

[0134] S64: The system extracts task parameters based on the decision result and generates specific control instructions. The instructions are used to guide the unmanned equipment to perform the corresponding tasks, and the specific implementation includes:

[0135] S641. Extract mission parameters: Extract required mission parameters from knowledge base, embodied equipment instruction set and other services to ensure instruction accuracy and effectiveness.

[0136] S642, generate control instructions: use the extracted mission parameters to generate specific control instructions to guide the unmanned equipment to perform the mission.

[0137] S643, command sending and execution: Send the generated commands to the unmanned equipment through the Robot Operating System (ROS) or other standard interfaces, and monitor its execution status in real time.

[0138] S65, execution feedback and optimization. During the unmanned equipment's mission execution, the system collects feedback information and adjusts the command generation and sending strategy accordingly. Through continuous optimization, the overall system performance and the accuracy of mission execution are improved. The specific steps include:

[0139] S651, execution status monitoring: real-time monitoring of the execution status of unmanned equipment and collection of feedback information.

[0140] S652. Feedback analysis and adjustment: Analyze feedback data and adjust instruction generation and sending strategies.

[0141] S653, Continuous Optimization: Achieve high efficiency and accuracy in task execution through continuous feedback and optimization.

[0142] The technical solution proposed in this application aims to improve the autonomous decision-making and execution capabilities of unmanned equipment in complex environments. It integrates core technologies such as multi-level intent understanding, prompt template generation, chain reasoning decision, human-computer feedback mechanism, and intelligent tool set calling, realizing a complete closed loop from user task input to autonomous execution of unmanned equipment. The innovation lies in the decomposition and efficient allocation of complex tasks to large and small models for collaborative execution through hierarchical intent parsing and prompt template guidance, so that the system can quickly respond to dynamic environmental changes and optimize task planning and execution paths. In addition, by introducing a human-computer feedback mechanism, intelligent adjustments are made during task execution, which improves the adaptability and robustness of the system. This innovative framework not only improves the efficiency and accuracy of task completion of unmanned equipment, but also provides an important reference for the autonomous development of future intelligent systems.

[0143] Example:

[0144] According to the above technical solution, the present invention proposes a specific embodiment as follows:

[0145] First, you need to create an agent and define the agent's name, introduction, large model used, role prompt words, tool set, human-in-the-loop mode, tool usage mode, etc. The system automatically generates a prompt template for the agent based on the above settings.

[0146] At the beginning of each task, the agent receives a complex task instruction in natural language input from the user: task = {x1, x2, ..., x n}, where x i is the i-th character. For example, the task instruction can be "send the nearest drone to the position (3,2) for photo reconnaissance". The system combines the task description and the prompt template to generate a multi-dimensional prompt word:

[0147] p={p role ,p tool ,p history ,p task ,p prestep ,p feedback}

[0148] Where: p role is the agent role description, p tool is the description of the candidate toolset, p history For historical mission records, p task is the current task description, p prestep It is the current task execution process record, p feedback Provide feedback to users.

[0149] The prompt word p is used as the input din1 of the first decision step and is input into the large model M. The large model outputs dout1 after reasoning and decision making. The system parses the structured thinking process, tool set name and parameter information from dout1.

[0150] For example, in the task "send the nearest drone to the position (3,2) for photo reconnaissance", based on the GPT-4o model, the first decision step input din1 is as follows (partial diagram):

[0151] ---------------input---------------:

[0152] [{'role':'system','content':'You are an unmanned equipment cluster control agent. You can accurately control the unmanned vehicle equipment and drone equipment in the cluster to handle events / tasks according to the user's instructions. Unmanned equipment is located in three-dimensional space using x-axis, y-axis and z-axis coordinates. The origin coordinate is (0,0,0), and the area range is: x value range (-10,10), y value range (-6,6), z value range is (0,10). Unmanned vehicles can only move on the plane where z=0, and drones need to move in the space where z>0. If necessary, you can query the cluster information before each mission starts. When planning the path, try to make each unmanned equipment travel the shortest distance. Each unmanned vehicle or drone can only receive one instruction at a time, so remember: each time you select a tool, you can only output one action for the same unmanned vehicle or drone, and you cannot output multiple actions. \nAnswer the following questions as best you can.You have access to the following tools:\n\nEquipment performance parameter query: query the performance parameters of the equipment according to the name and model of the equipment. You can directly ask the performance parameters of a certain equipment, or directly ask a certain performance parameter\n\tTool input examples are as follows:\n\t\tWhat is the range of the burger car?\n\nUnmanned vehicle moves to the target point: This tool can control the unmanned vehicle to move to the specified target point according to the user's requirements. It is not necessary to know the current position of the unmanned vehicle. The input of this tool is a command. \n\tThe input of the tool needs to include the following key information:\n\t\tThe x-axis coordinate of the target point (x)\n\t\tThe y-axis coordinate of the target point (y)\n\t\tThe number of the unmanned vehicle, such as 0,1,2,...\n\tThe input example of the tool is as follows:\n\t\tThe No. 2 unmanned vehicle moves to the target point (5,9)\n\nUnmanned vehicle position query: To query the current position coordinates of the unmanned vehicle, you need to provide the number of the unmanned vehicle, such as querying the position of the No. 1 unmanned vehicle\n\tThe input of the tool needs to include the following key information:\n\t\tThe number of the unmanned vehicle, such as 0,1,2,...\n\tThe input example of the tool is as follows:\n\t\tQuery the position of the No. 1 unmanned vehicle\n\nUnmanned vehicle moves in a straight line: This tool can control the unmanned vehicle to move in a straight line along the current direction according to the user's requirements, without knowing the current position of the unmanned vehicle. \n\tThe input of the tool needs to include the following key information:\n\t\tThe number of the unmanned vehicle, such as 0,1,2,...\n\t\tDistance of advance\n\tTool input example is as follows:\n\t\tUnmanned vehicle No. 2 moves forward 1 meter\n\nUnmanned vehicle moves backward in a straight line: This tool can control the unmanned vehicle to move backward in a straight line along the current direction according to the user's requirements, without knowing the current position of the unmanned vehicle. \n\tThe input of the tool needs to include the following key information:\n\t\tUnmanned vehicle number, such as 0,1,2,...\n\t\tDistance of retreat\n\tTool input example is as follows:\n\t\tUnmanned vehicle No. 2 moves backward 1 meter\n\nCluster information query: This tool is used to query the basic information of the cluster, including which equipment is in the cluster, their respective numbers, current position and initial position (default position) and other information. \n\nDrone goes to the target point: This tool can control the drone to go to the specified target point according to the user's requirements. The input of this tool is a command\n\tThe input of the tool needs to include the following key information:\n\t\tThe x-axis coordinate of the target point (x)\n\t\tThe y-axis coordinate of the target point (y)\n\t\tDrone number, such as 10,11,12,...\n\nDrone photo reconnaissance: This tool can control the drone camera to capture photos, and then analyze and judge the reconnaissance mission based on the content of the photos. \n\tTool input examples are as follows:\n\t\tUAV 2 reconnaissance confirms whether there are tank targets\n\t\tUAV 3 reconnaissance purpose: which buildings appear and the appearance color of each building\n\t\tUAV 0 reconnaissance mission: confirm whether there are men wearing black shirts\n\nUnmanned vehicle strike: This tool can control unmanned vehicles to strike (attack, destroy, destroy, etc.)\n\tThe input of the tool needs to include the following key information:\n\t\tUnmanned vehicle number, such as 0,1,2,...\n\tTool input examples are as follows:\n\t\tInitiate strike\n\nUnmanned vehicle photo reconnaissance: This tool can control the camera of the unmanned vehicle to capture photos, and then analyze and judge the reconnaissance mission based on the content of the photos.\n\nUnmanned vehicle turn left: This tool can control the unmanned vehicle to turn left or rotate counterclockwise by an angle\n\nUnmanned vehicle turn right: This tool can control the unmanned vehicle to turn right or rotate clockwise by an angle.\n\nUnmanned vehicle rotation: This tool can control the unmanned vehicle to adjust its direction or orientation and rotate it to a specific angle. When you need to explore the surrounding environment, you can also use this tool to adjust the direction, and then observe and analyze. \n\nUAV location query: query the current location coordinates of the px4 drone\n\n\nThe way you use the tools is by specifying a json blob.\nSpecifically,this json should have a`action`key(with the name of the tool to use)and a`action_input`key(with the input to the tool going here).\n\nTheonly values ​​that should be in the"action"field are:equipment performance parameter query,unmanned vehicle moves to the target point,unmanned vehicle position query,unmanned vehicle moves forward in a straight line,unmanned vehicle moves backward in a straight line,cluster information query,drone goes to the target point,drone photo reconnaissance,unmanned vehicle strikes,unmanned vehicle photo reconnaissance,unmanned vehicle turns left,unmanned vehicle turns right,unmanned vehicle turns,drone position query\n\nThe$JSON_BLOB can contain a SINGLE action,here is anexample of a valid$JSON_BLOB for single action:\n```\n{\n"action":$TOOL_NAME,\n"action_input":$INPUT\n}\n```\nOr The$JSON_BLOB can contain a list ofmultiple actions only when they need to be executed in parallel,here is anexample of a valid$JSON_BLOB for a list of multiple actions:\n```\n[{\n"action":$TOOL_NAME,\n"action_input":$INPUT\n},\n{\n"action":$TOOL_NAME,\n"action_input":$INPUT\n}]\n```\n\nALWAYS use the following format:\n\nQuestion:the input question you must answer\nThought:you should always thinkabout what to do\nAction:\n```\n$JSON_BLOB\n```\nObservation: the result of the action\n...(this Thought / Action / Observation can repeat N times)\nThought:I now know the final answer\nFinal Answer:the final answer to the original input question\n\nBegin! Reminder to always use the exact characters`FinalAnswer`when responding.'},{'role':'user','content':'Send the nearest drone to the (3,2) location for photo reconnaissance'}].

[0153] The output dout1 of the first decision step is as follows:

[0154] ---------------output--------------:

[0155] After understanding the requirements, I need to first query the cluster information to determine the current location of the drone and select the nearest drone to perform the task.

[0156]

[0157] The system parses the structured object from dout1 according to the following class structure to drive the call of specific tools: ComplexAgentAction(action_id="a5a8b5a777614c8e98deaaef66fbf744", thought="Query cluster information and determine the current location of the drone", actions="AgentAction(tool="Cluster information query", tool_input="{})]

[0158] Once the tool is successfully called, the decision step is completed.

[0159] After the decision result is generated in the previous decision step and analyzed by the system, the user can confirm or adjust the decision information through the feedback interface.

[0160] If the user directly confirms without any feedback, the system will execute the tool call and obtain the tool call result

[0161] observation.

[0162] If the user has feedback, skip the tool call and directly use the feedback as observation.

[0163] observation will be combined with the prompt word p to form an updated prompt word, and will be input into the large model M as the context of the next decision step din2 = observation + p. After reasoning and decision output dout2, the system further parses the structured thinking process, tool set name and parameter information from dout2.

[0164] The above decision-making step input and output process can be iteratively executed until the agent determines that the user's task instruction is completed, and the entire process ends.

[0165] For example, under normal circumstances, the second and third decision steps should be the drone heading to the target point and the drone taking photos for reconnaissance. If the large model has an abnormality during the reasoning process, such as ignoring the order of task execution and outputting wrong decision information, the user can give feedback, such as: "Your decision is wrong, you should pay attention to the order of execution." The intelligent agent will generate an updated prompt word din2 based on the feedback and re-input it into the large model for reasoning until the correct decision information dout2 is generated for step-by-step execution. The details are as follows (partial schematic):

[0166] [{'role':'system','content':'You are an unmanned equipment cluster control agent. You can accurately control the unmanned vehicle equipment and drone equipment in the cluster to handle events / tasks according to the user's instructions. Unmanned equipment is located in three-dimensional space using x-axis, y-axis and z-axis coordinates. The origin coordinate is (0,0,0), and the range of the area is: x value range (-10,10), y value range (-6,6), z value range is (0,10). Unmanned vehicles can only move on the plane where z=0, and drones need to move in the space where z>0. If necessary, you can query the cluster information before each mission starts. When planning the path, try to make each unmanned equipment travel the shortest distance. Each unmanned vehicle or drone can only receive one instruction at a time, so remember: each time you select a tool, you can only output one action for the same unmanned vehicle or drone, and you cannot output multiple actions. \nAnswer the following questions as best you can.You have access to the following tools:\n\nEquipment performance parameter query: query the performance parameters of the equipment according to the name and model of the equipment. You can directly ask the performance parameters of a certain equipment, or directly ask a certain performance parameter\n\tTool input examples are as follows:\n\t\tWhat is the range of the burger car?\n\nUnmanned vehicle moves to the target point: This tool can control the unmanned vehicle to move to the specified target point according to the user's requirements. It is not necessary to know the current position of the unmanned vehicle. The input of this tool is a command. \n\tThe input of the tool needs to include the following key information:\n\t\tThe x-axis coordinate of the target point (x)\n\t\tThe y-axis coordinate of the target point (y)\n\t\tThe number of the unmanned vehicle, such as 0,1,2,...\n\tThe input example of the tool is as follows:\n\t\tThe No. 2 unmanned vehicle moves to the target point (5,9)\n\nUnmanned vehicle position query: To query the current position coordinates of the unmanned vehicle, you need to provide the number of the unmanned vehicle, such as querying the position of the No. 1 unmanned vehicle\n\tThe input of the tool needs to include the following key information:\n\t\tThe number of the unmanned vehicle, such as 0,1,2,...\n\tThe input example of the tool is as follows:\n\t\tQuery the position of the No. 1 unmanned vehicle\n\nUnmanned vehicle moves in a straight line: This tool can control the unmanned vehicle to move in a straight line along the current direction according to the user's requirements, without knowing the current position of the unmanned vehicle. \n\tThe input of the tool needs to include the following key information:\n\t\tThe number of the unmanned vehicle, such as 0,1,2,...\n\t\tDistance of advance\n\tTool input example is as follows:\n\t\tUnmanned vehicle No. 2 moves forward 1 meter\n\nUnmanned vehicle moves backward in a straight line: This tool can control the unmanned vehicle to move backward in a straight line along the current direction according to the user's requirements, without knowing the current position of the unmanned vehicle. \n\tThe input of the tool needs to include the following key information:\n\t\tUnmanned vehicle number, such as 0,1,2,...\n\t\tDistance of retreat\n\tTool input example is as follows:\n\t\tUnmanned vehicle No. 2 moves backward 1 meter\n\nCluster information query: This tool is used to query the basic information of the cluster, including which equipment is in the cluster, their respective numbers, current position and initial position (default position) and other information. \n\nDrone goes to the target point: This tool can control the drone to go to the specified target point according to the user's requirements. The input of this tool is a command\n\tThe input of the tool needs to include the following key information:\n\t\tThe x-axis coordinate of the target point (x)\n\t\tThe y-axis coordinate of the target point (y)\n\t\tDrone number, such as 10,11,12,...\n\nDrone photo reconnaissance: This tool can control the drone camera to capture photos, and then analyze and judge the reconnaissance mission based on the content of the photos. \n\tTool input examples are as follows:\n\t\tUAV 2 reconnaissance confirms whether there are tank targets\n\t\tUAV 3 reconnaissance purpose: which buildings appear and the appearance color of each building\n\t\tUAV 0 reconnaissance mission: confirm whether there are men wearing black shirts\n\nUnmanned vehicle strike: This tool can control unmanned vehicles to strike (attack, fire, destroy, annihilate, destroy, etc.)\n\tThe input of the tool needs to include the following key information:\n\t\tUnmanned vehicle number, such as 0,1,2,...\n\tTool input examples are as follows:\n\t\tInitiate strike\n\nUnmanned vehicle photo reconnaissance: This tool can control the unmanned vehicle camera to capture photos, and then analyze and judge the reconnaissance mission based on the content of the photos.\n\nUnmanned vehicle turn left: This tool can control the unmanned vehicle to turn left or rotate counterclockwise by an angle\n\nUnmanned vehicle turn right: This tool can control the unmanned vehicle to turn right or rotate clockwise by an angle. \n\nRotation of unmanned vehicle: This tool can control the unmanned vehicle to adjust its direction or orientation, so that it rotates to a specific angle. When you need to explore the surrounding environment, you can also use this tool to adjust the direction, and then observe and analyze. \n\nUAV location query: query the current location coordinates of the px4 drone\n\n\nThe way you use the tools is by specifying a json blob.\nSpecifically,this json should have a`action`key(with the name of the tool to use)and a`action_input`key(with the input to the tool going here).\n\nTheonly values ​​that should be in the"action"field are:equipment performance parameter query,unmanned vehicle moves to the target point,unmanned vehicle position query,unmanned vehicle moves forward in a straight line,unmanned vehicle moves backward in a straight line,cluster information query,drone goes to the target point,drone photo reconnaissance,unmanned vehicle strikes,unmanned vehicle photo reconnaissance,unmanned vehicle turns left,unmanned vehicle turns right,unmanned vehicle turns,drone position query\n\nThe$JSON_BLOB can contain a SINGLE action,here is anexample of a valid$JSON_BLOB for single action:\n```\n{\n"action":$TOOL_NAME,\n"action_input":$INPUT\n}\n```\nOr The$JSON_BLOB can contain a list ofmultiple actions only when they need to be executed in parallel,here is anexample of a valid$JSON_BLOB for a list of multiple actions:\n```\n[{\n"action":$TOOL_NAME,\n"action_input":$INPUT\n},\n{\n"action":$TOOL_NAME,\n"action_input":$INPUT\n}]\n```\n\nALWAYS use the following format:\n\nQuestion:the input question you must answer\nThought:you should always thinkabout what to do\nAction:\n```\n$JSON_BLOB\n```\nObservation: the result of the action\n...(this Thought / Action / Observation can repeat N times)\nThought:I now know the final answer\nFinal Answer:the final answer to the original input question\n\nBegin! Reminder to always use the exact characters`FinalAnswer`when responding.'},{'role':'user','content':'Send the nearest drone to the position (3,2) for photo reconnaissance\n\nQuestion:Send the nearest drone to the position (3,2) for photo reconnaissance\nThought:First, I need to query the cluster information to know the drone number and current position. Then, I can select the drone closest to the target point and send it to the specified location for photo reconnaissance. \nAction:\n```\n{\n"action":"Cluster information query",\n"action_input":{}\n}\n```\n\nObservation: There are a total of 10 unmanned equipment in the current cluster environment, which are:\nNo. 0 tb3 unmanned vehicle equipment, currently at (0.03,0.02,-0.0), its initial position is (0.0,0.0,0.0)\nNo. 1 tb3 unmanned vehicle equipment, currently at (2.98,2.97,-0.0), its initial position is (3.0,3.0,0.0)\nNo. 2 tb3 unmanned vehicle equipment, currently at (-3.0,3.0,-0.0), its initial position is (-3.0,3.0,0.0)\nNo. 3 tb3 unmanned vehicle equipment, currently at (-3.0,-3.0,-0. 0), its initial position is (-3.0,-3.0,0.0)\nNo. 4 tb3 unmanned vehicle equipment, currently at (3.0,-3.0,-0.0), its initial position is (3.0,-3.0,0.0)\nNo. 5 tb3 unmanned vehicle equipment, currently at (6.0,-0.0,-0.0), its initial position is (6.0,0.0,0.0)\nNo. 0 iris drone equipment, when The current position is (2.03, 0.02, 2.98), and its initial position is (2.0, 0.0, 3.0)\nThe iris drone equipment No. 1 is currently at (0.03, 2.0, 3.48), and its initial position is (0.0, 2.0, 3.5)\nThe iris drone equipment No. 2 is currently at (-1.97, 0.01, 3.98), and its initial position is (-2.0, 0.0, 4.0)\nThe iris drone equipment No. 3 is currently at (0.02,-2.0,4.48), and its initial position is (0.0,-2.0,4.5)\n\nThought: From the cluster information, we can see that there are four drones, and their current positions are:\n-Drone No. 0 is at (2.03,0.02,2.98)\n-Drone No. 1 is at (0.03,2.0,3.48)\n-Drone No. 2 is at (-1.97,0.01,3.98)\n-Drone No. 3 is at (0.02,-2.0,4.48)\n\nNow I need to calculate the distance from each drone to the target point (3,2) and select the nearest drone. \n\nThe formula for calculating the distance is:\n\\[\\text{distance}=\\sqrt{(x_2-x_1)^2+(y_2-y_1)^2+(z_2-z_1)^2}\\]\n\nCalculate the distance from each drone to the target point:\n-0 drone:\\[\\sqrt{(3-2.03)^2+(2-0.02)^2+(0-2.98)^2}\\]\n-1 drone:\\[\\sqrt{(3-0.03)^2+(2-2.0)^2+(0-3.48)^2}\\]\n-2 drone:\\[\\sqrt{(3-(-1.97))^2+(2-0.01)^2+(0-3.98)^2}\\]\n-3 drone :\\[\\sqrt{(3-0.02)^2+(2-(-2.0))^2+(0-4.48)^2}\\]\n\nCalculate:\n-Drone distance 0:\\[\\sqrt{0.9409+3.9204+8.8804}=\\sqrt{13.7417}\\approx3.71\\]\n-Drone distance 1:\\[\\sqrt{8.8209+0+12.1104}=\\sqrt{20.9313}\\approx4.57\\]\n-Drone distance 2:\\[\\sqrt{24.9604+3.9601+15.8404}=\\sqrt{44.7609}\\approx 6.69\\]\n-Distance of drone No. 3:\\[\\sqrt{8.8804+16.0+20.0704}=\\sqrt{44.9508}\\approx 6.70\\]\n\nTherefore, the drone closest to the target point is No. 0. \n\nNext, I will instruct No. 0 drone to go to the target point (3,2) and take photos. \n\nAction:\n```\n[\n{\n"action":"Drone goes to the target point",\n"action_input":{\n"x":3,\n"y":2,\n"Drone number":0\n}\n},\n{\n"action":"Drone takes photos",\n"action_input":{\n"Drone number":0\n}\n}\n]\n```\n\n\nObservation: Your decision is wrong. You should pay attention to the order of execution. \nThought:'}].

[0167] Because the execution order was wrong, I needed to let UAV 0 go to the target point first, and then take photos for reconnaissance. Adjust the order and execute the operation again.

[0168]

[0169] like Figure 2 As shown, a decision-making and execution process (part) in a specific embodiment is shown, in which:

[0170] The user inputs the task instruction "Send the nearest drone to the position (3,2) for photo reconnaissance" in natural language to the "cluster control agent". The system generates multi-dimensional prompt words based on the task description and prompt template, and inputs them into the large model GPT-4o. After reasoning and decision-making, the large model outputs the first step decision, which requires querying the current number and location of the unmanned equipment cluster. After the user clicks "Instruction Issue" to confirm, the system performs a structured analysis of the decision result, successfully calls and executes the "Cluster Information Query" tool, and obtains the execution result of the first step. The agent infers the second step decision based on the result, and simultaneously issues the "drone goes to the target point" and "drone takes photos for reconnaissance", ignoring the order of task execution, and the decision information is incorrect. The user clicks the feedback button and enters the feedback information: "Note that you must go to the target point before performing reconnaissance, and there is a time sequence". The agent will re-infer the decision based on the feedback. This process can be repeated until the correct decision information is generated.

[0171] After all decision steps are completed and the tool call is successfully executed, the task execution of the unmanned equipment is also completed.

[0172] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of the method for autonomous task planning and execution of unmanned equipment based on a large model intelligent agent framework provided by the present invention and some or all steps in each embodiment. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0173] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0174] The present invention provides an idea and method for autonomous task planning and execution of unmanned equipment based on a large model intelligent body framework. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the scope of protection of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A method for autonomous task planning and execution of unmanned equipment based on a large model intelligent agent framework, characterized in that: The following steps are involved: Step 1: Create and configure the agent, set the name, introduction, type, model, role prompt, knowledge base and toolset of the agent, and build an agent that meets the requirements of custom tasks; Generate a prompt template for the agent, wherein the prompt template includes: role prompt, output format requirement, candidate tool set, task description, execution process and user feedback; Step 2: task requirement analysis: for the natural language task requirement input by the user, a multi-dimensional prompt word is generated through the prompt template of the intelligent agent constructed in step 1; Step 3, input the prompt words generated in step 2 into the large model, and output unstructured decision step information in sequence according to the decision steps, wherein each decision step information includes the decision process of the large model, the tools to be used and their parameters; Step 4: Based on the unstructured decision step information obtained in step 3, a feedback mechanism is constructed, and the user confirms or adjusts the decision step information in step 3 to obtain feedback information; the above feedback information is combined with the prompt word in step 2 to form a new prompt word and steps 3 to 4 are repeatedly executed iteratively until the user confirms and obtains the decision result; In step 5, the intelligent agent performs structured analysis based on the decision result obtained in step 4, extracts the parameters required for each tool, carries out intelligent agent tool set calling and instruction generation, and completes the corresponding decision steps, thereby completing the task execution of the unmanned equipment.

2. According to claim 1, a method for autonomous task planning and execution of unmanned equipment based on a large model intelligent agent framework is characterized in that: The feedback mechanism described in step 5 includes: According to the settings in step 1, the agent performs tasks in two operation modes: human-in-the-loop HITL and human-out-of-the-loop HOTL.

3. According to claim 1, a method for autonomous task planning and execution of unmanned equipment based on a large model intelligent agent framework is characterized in that: The agent tool set call and instruction generation described in step 5 specifically include: Toolset interface standardization, dynamic toolset selection and calling, toolset feedback and optimization, parameter extraction and instruction generation.

4. The method for autonomous task planning and execution of unmanned equipment based on a large model agent framework according to claim 1, characterized in that: The creation and configuration of the agent described in step 1 specifically includes: Step 1-1, the user inputs the name and introduction of the agent to be created, and generates an agent ID as the unique identifier of the agent. The introduction is used to describe the basic functions and application scenarios of the agent; Step 1-2, the user selects the type of the agent, which includes question-answering type, instruction type, accusation type and equipment type; Steps 1-3: Users select from multiple candidate large models based on task requirements; Step 1-4, in the role prompt setting, the user enters a custom role prompt or selects the initial role prompt generated by the large model, and adjusts it to meet the task requirements; Steps 1-5, the user can associate the agent with a local knowledge base, which provides supporting data for task execution; Step 1-6, the user selects the agent mode according to the specific task, the mode includes intelligent selection mode, process arrangement mode and autonomous planning mode, wherein when the mode is autonomous planning mode, the human-out-of-the-loop HOTL mode or the human-in-the-loop HITL mode is further selected; In steps 1-7, the user needs to select and associate a toolset for the agent. The toolset is pre-set by the user, and the agent automatically calls the tools in the toolset when performing a task.

5. The method for autonomous task planning and execution of unmanned equipment based on a large model agent framework according to claim 1 is characterized in that: The task requirement analysis described in step 2 specifically includes: Step 2-1, the user inputs task requirements in natural language to the agent; wherein the requirements include specific task objectives, operation constraints and expected results; Step 2-2, the agent processes the input task requirements using natural language processing (NLP) methods to identify key task elements; wherein the processing includes word segmentation, grammatical parsing, and semantic analysis; Step 2-3: Based on the key task elements identified in step 2-2, the intelligent agent parses and generates multi-dimensional prompt words in combination with the prompt template to guide the reasoning process of the large model.

6. The method for autonomous task planning and execution of unmanned equipment based on a large model agent framework according to claim 5 is characterized in that: The generation of multi-dimensional prompt words described in step 2-3 specifically includes: Using key task elements, the multi-dimensional prompt words are expressed as follows: I={I r ,I t ,I h ,I p ,I d ,I f } Where: I r is the role description of the intelligent agent, I t Describe the candidate toolset. h For historical mission records, I p is the current task description, I d It is the current task execution process record, I f Provide feedback to users; Agent Role Description I r , used to detail the agent’s position and role in the task; Candidate Toolset Description I t , which lists the available tools and their functions related to the current task; Historical Mission Record I h , used to summarize the records of similar tasks in the past, including task content, execution methods and results; Current Task Description I p , used to describe the current task information; Current task execution process record I d , used to provide the execution status of the previous decision step under the current task, including the execution content and execution results, etc., for reference by the next decision step; User FeedbackI f , used to record the feedback given by users during task execution.

7. The method for autonomous task planning and execution of unmanned equipment based on a large model agent framework according to claim 1, characterized in that: The unstructured decision step information is outputted sequentially according to the decision steps described in step 3, specifically including: The prompt words generated in step 2 are input into the large model to make a decision, and the unstructured text decision results including the task execution plan, tool calling sequence and corresponding tool parameters are output.

8. The method for autonomous task planning and execution of unmanned equipment based on a large model agent framework according to claim 2 is characterized in that: The feedback mechanism described in step 4 includes: Step 4-1, the agent performs the task in the human-in-the-loop HITL operation mode, as follows: Step 4-1-1, real-time reasoning process display, graphically displaying the reasoning path and current execution status of the large model, including decision nodes, logical judgments and parameter changes; Step 4-1-2, execution feedback visualization, combined with real-time feedback, to show the progress and results of the current task execution; Step 4-1-3, user feedback integration, receives feedback from users, and adjusts the reasoning direction and execution parameters according to the feedback content. The user feedback processing logic is as follows: Step 4-1-4, if the current decision is in line with expectations, the user provides confirmation feedback, that is, confirms the current single step, or confirms the next N steps and enters the HOTL mode in the next N steps; Step 4-2, the agent performs the task in the human-out-of-the-loop HOTL operation mode, as follows: Step 4-2-1: autonomous reasoning and decision-making, using large models to automatically perform task decisions; Step 4-2-2, anomaly detection and processing, automatically identify and process anomalies or deviations in the execution process; Step 4-2-3, automatic mode switching. When an abnormal situation occurs, it will actively switch to the human-in-the-loop HITL mode and invite users to participate in decision-making. The abnormal handling logic is as follows: Step 4-3: Establish a backtracking and optimization mechanism. During the reasoning and execution process, if a reasoning error or a logical dead loop is found, automatically backtrack to the previous step to reselect the path and optimize the solution. The backtracking and optimization logic is as follows:

9. The method for autonomous task planning and execution of unmanned equipment based on a large model agent framework according to claim 1, characterized in that: The structural analysis described in step 5 specifically includes: Step 5-1: Use natural language processing methods to semantically understand the decision results in the form of unstructured text and identify key task elements; Step 5-2, extracting and converting the decision step information, tool calling sequence and corresponding tool parameters in the decision result into structured data; the structured decision data I l as follows: I l ={S,T,T p } Among them, S is the decision step information, which indicates the specific operations required for each step; T is the tool call plan, which indicates the tool set and call order required; T p is the specific parameter value of the required tool; Step 5-3, match the corresponding toolset and calling strategy according to the decision step information.

10. The method for autonomous task planning and execution of unmanned equipment based on a large model intelligent agent framework according to claim 3, characterized in that: The agent tool set call and instruction generation described in step 5 specifically include: Standardize tool set interfaces and design common interface protocols to support the processing and analysis of multimodal data types such as text, images, audio, and video; Dynamic toolset selection and calling: select toolsets to assist decision making and execution based on task requirements and current resource status; Toolset feedback and optimization: During the toolset invocation process, tool execution feedback is collected and tool selection and invocation strategies are dynamically adjusted based on the feedback results. Based on the decision results, the task parameters are extracted and specific control instructions are generated. The generated instructions are sent to the unmanned equipment, and its execution status is monitored in real time. During the execution of the task by the unmanned equipment, feedback information is collected and the instruction generation and sending strategy is adjusted accordingly.

Images