Unmanned aerial vehicle inspection system and method based on multi-agent cooperation

Abstract

The invention belongs to the technical field of unmanned aerial vehicle inspection, and particularly relates to an unmanned aerial vehicle inspection system and method based on multi-agent cooperation. The system comprises an early-stage preparation agent, a task planning agent, a monitoring and execution agent, a summary and induction agent and a Memory module, and each agent comprises an unmanned aerial vehicle tool set corresponding to a function. The intelligent agents are clear in division of labor, information sharing and decision cooperation can be carried out according to inspection requirements, the response speed and collaboration of inspection tasks are improved, and the problem of information processing bottlenecks in a traditional single agent mode is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of drone inspection, and more specifically, relates to a drone inspection system and method based on multi-agent collaboration. Background Art

[0002] Drones are widely used in power inspection, agricultural monitoring, traffic management and other fields. Especially in infrastructure inspection, they have gradually become a common tool due to their flexibility and unmanned operation advantages. However, traditional drone inspection mission planning usually relies on a single system or intelligent agent to perform static planning of inspection routes, and lacks the ability to flexibly respond to environmental changes and mission emergencies. In the prior art, some solutions are based on fixed-route inspection methods, which are usually planned and managed by a single intelligent agent or centralized control system. For example, some power inspection drone systems use pre-planned paths when generating routes, or specify inspection orders through manual intervention, and perform simple monitoring feedback, making it difficult to adjust paths in real time or coordinate multiple drones to perform complex tasks. This method is easily affected by environmental changes, equipment failures, etc. during execution, making it difficult to complete inspection tasks as expected.

[0003] The existing drone inspection planning technology has the following shortcomings: Poor response flexibility: Traditional solutions lack the ability to dynamically adjust routes and often rely on fixed paths, making it difficult to respond to environmental changes or temporary mission requirements. This approach is prone to interruptions in actual execution due to external factors such as obstacles and weather changes.

[0004] Single agent control: Most traditional solutions adopt a centralized control method, coordinating inspection tasks through only a single agent or control center. It is impossible to achieve the division of labor and cooperation among multiple task roles, making it difficult for the system to effectively handle diverse task requirements, and the monitoring and feedback efficiency is low.

[0005] Lack of distributed decision-making: In the event of drone failure or environmental changes, traditional methods cannot achieve distributed collaboration and decision-making among intelligent agents, thereby reducing the integrity and reliability of inspection tasks.

[0006] Chinese patent document CN111652460A discloses an intelligent optimization method and system for collaborative inspection of multiple towers by multiple drones. The method includes: pre-establishing an information database of the inspection area; receiving inspection tasks for designated towers in the inspection area; obtaining capability information of drones in the drone base; and determining a corresponding inspection plan using a preset algorithm based on the information database, the capability information of the drones, and the inspection tasks, and issuing a drive instruction based on the inspection plan to drive the corresponding drones in the drone base to start executing the inspection tasks. The system includes: an inspection control center module, a drone control module, and an intelligent planning module. However, the invention uses multiple drones to collaborate to complete the corresponding inspection tasks, and uses a centralized control system for planning and management, lacking distributed decision-making.

[0007] In view of this, the present invention designs a drone inspection system and method based on multi-agent collaboration. Summary of the invention

[0008] The present invention aims to overcome at least one of the defects of the above-mentioned prior art and provide a drone inspection system based on multi-agent collaboration to achieve dynamic adjustment, distributed collaboration and efficient feedback of inspection tasks.

[0009] The invention also discloses a drone inspection method based on multi-agent collaboration.

[0010] The detailed technical scheme of the present invention is as follows: A drone inspection system based on multi-agent collaboration, the system includes a preliminary preparation agent, a task planning agent, a monitoring and execution agent, a summary agent and a memory module; each agent contains a drone tool set corresponding to the function; The preliminary preparation agent is responsible for helping the user complete the preliminary preparation work of the drone inspection task; The mission planning agent is responsible for recommending the most suitable route for the user's drone inspection mission and dynamically adjusting it during execution; The monitoring and execution agent is responsible for real-time monitoring of the execution of tasks during the drone inspection process; The summarizing agent is responsible for confirming the task status and generating a detailed report after the task is completed; The Memory module is responsible for information storage and information retrieval.

[0011] Preferably, the tool set for the preliminary preparation of the intelligent agent includes: logging into the control platform, querying the list of machine nests, and querying drones; The tool set of the mission planning agent includes: route list query, mission status query and mission creation; The tool set of the monitoring and execution agent includes: drone query, nest preparation, drone takeoff, drone pointing flight and drone action execution; The tool set for the summarizing and inductive agent includes: drone return and task archiving.

[0012] A drone inspection method based on multi-agent collaboration, the method comprising the following steps: S1. The user issues a command. The Setup Agent parses the user's command and calls the nest list query tool and the drone query tool to query the status of the nest and drone. If an abnormality is detected, an abnormality report is returned to the user, prompting the user to troubleshoot the fault and provide new information in an interactive way. After confirming that the equipment status is normal, the execution right is transferred to the task planning agent Dynamic Planning Agent. S2, the task planning agent Dynamic Planning Agent plans the initial route according to the inspection requirements and geographical environment provided by the user, and transmits the planning results to the monitoring and execution agent Task Management Agent; if information supplement or confirmation is required during the planning process, the user is notified to supplement or confirm; S3, the monitoring and execution agent Task Management Agent executes each step in the plan in turn, periodically monitors the task execution status, and when the task cannot continue to be executed, notifies the preliminary preparation agent Setup Agent to solve the equipment abnormality or notifies the task planning agent Dynamic Planning Agent to re-plan until the flight is completed; S4: After the inspection is completed, the Completion Agent orders the drone to return and generates an inspection report based on the records of the inspection process.

[0013] Preferably, according to the present invention, during the entire multi-agent interaction process, all agents can communicate with the Memory module to record or read important information related to their own stage.

[0014] Preferably, according to the present invention, the technical implementation of all intelligent agents follows the design pattern of "planning-action-reflection", and formatted output is used in all three stages for easy parsing.

[0015] Preferably, according to the present invention, the prompt words for the task performed by the preliminary preparation agent are as follows: a. Log in to the drone control platform and confirm user permissions; b. Check the status of the drone equipment to ensure that it is normal; c. Confirm the availability of the take-off airport and the aircraft nest to ensure that the drone can take off and return smoothly; d. If the user encounters login failure or device abnormality, provide corresponding troubleshooting suggestions and assist in reconfiguration; e. If the equipment status is normal and the airport is available, transfer to the mission planning agent for mission planning.

[0016] Preferably, according to the present invention, the prompt words for the task performed by the task planning agent are as follows: a. Plan the initial route according to the inspection requirements and geographical environment provided by the user; b. If encountering obstacles, no-fly zones or weather changes, adjust the route in real time to ensure the safe flight of the drone; c. During the mission, if the user needs to modify the inspection area or mission objectives, re-plan the route and provide an optimization plan; d. Provide route planning results and make adjustments based on user feedback; e. Assist users to customize route planning when necessary to ensure that all inspection targets are covered; f. After the route planning is confirmed to be correct, it is transferred to the monitoring and execution agent.

[0017] Preferably, according to the present invention, the prompt words for monitoring and executing the tasks performed by the agent are as follows: a. Monitor the inspection progress of drones; b. If there is any flight failure, out-of-range, equipment problem or any abnormal situation during the mission, immediately notify the user and provide corresponding solutions; c. Trigger necessary operational tasks to ensure successful completion of the task; d. Feedback all intermediate outputs of the agent to the user in real time to ensure that they understand the task progress and potential risks; e. When a mission is abnormal, it can be transferred to the preparatory agent or mission planning agent for fault repair or route adjustment; f. After the task is completed, it is transferred to the summary agent for return and task archiving.

[0018] Preferably, according to the present invention, the prompt words for summarizing the tasks performed by the agent are as follows: a. Confirm whether the mission has been completed, including whether all inspection points have been covered, whether the flight is smooth, and whether the equipment is in normal status; b. Summarize all inspection data; c. Generate a detailed inspection report based on the collected information. The report should include task progress, key events and solutions, equipment inspection results, etc.; d. After the report is generated, the user is prompted that the task has been completed and a task summary is provided; e. If there is any omission in the report, remind the user to supplement the relevant information to ensure the completeness of the report.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention introduces four key agents to realize the division of labor and collaboration in inspection tasks. The task planning agent can flexibly adjust the route according to the real-time environment and task requirements to adapt to complex inspection requirements; the monitoring and execution agent monitors the progress of the task in real time and cooperates with external modules to improve the safety of the inspection process; the summarizing agent automatically generates an inspection report and ensures the completeness of the task record through the memory module, which improves the accuracy of data processing and the efficiency of task completion; each agent has a clear division of labor and can share information and collaborate in decision-making according to inspection requirements, which improves the response speed and coordination of inspection tasks and solves the information processing bottleneck under the traditional single agent mode.

[0020] (2) The application of the "planning-action-reflection" model of the present invention enables the intelligent agent to dynamically adapt to the task environment. At the same time, through formatted phased output, it ensures the traceability and efficient optimization of the task execution process. At the same time, a memory module communication is set up to support information storage and sharing between intelligent agents, so that important information related to each stage can be stored and called across intelligent agents, thereby improving the integrity and traceability of the task process. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of a drone inspection method based on multi-agent collaboration described in the present invention.

[0022] Figure 2 It is a "planning-action-reflection" flow chart for monitoring and executing an intelligent agent in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0024] Embodiment 1, This embodiment provides a drone inspection system based on multi-agent collaboration, the system includes a preliminary preparation agent setup_agent, a task planning agent dynamic_planning_agent, a monitoring and execution agent task_management_agent, a summary agent completion_agent and a Memory module; each agent contains a drone tool set corresponding to the function; The preliminary preparation agent is responsible for helping the user complete the preliminary preparation work of the drone inspection task; The mission planning agent is responsible for recommending the most suitable route for the user's drone inspection mission and dynamically adjusting it during execution; The monitoring and execution agent is responsible for real-time monitoring of the execution of tasks during the drone inspection process; The summarizing agent is responsible for confirming the task status and generating a detailed report after the task is completed; The Memory module is responsible for information storage and information retrieval. For example, after a setup_agent execution cycle ends, the memory summarizes the text content within this cycle and stores it in the database in the form of "title-content". When the setup_agent starts the next execution, the memory module can also retrieve information saved from other agents to help itself understand the context.

[0025] The tool set for the preliminary preparation of the intelligent agent includes: logging into the control platform, querying the list of drone nests, and querying drones; The tool set of the mission planning agent includes: route list query, mission status query and mission creation; The tool set of the monitoring and execution agent includes: drone query, nest preparation, drone takeoff, drone pointing flight and drone action execution; The tool set for the summarizing and inductive agent includes: drone return and task archiving.

[0026] The UAV tool design is shown in Table 1: Table 1 UAV tool design table Tool Name Tool Description Parameter Description Return results Management platform login The user logs in to the drone control platform and obtains API access rights Token: login token Login result (success or failure) Machine nest list query Used to query the status of all machine nests none All machine nest IDs, machine nest status, etc. Drone Query Used to query the status of all drones none All drone IDs, drone types, drone nests, health status, mission status (standby, inspection, returning), etc. Route list query Used to query all inspection routes none All route IDs, route names, ordered waypoints, etc. Task status query Used to query the status of all tasks none Mission ID, status (in progress, completed), route used, drone used, etc. Task creation Create a new inspection task Route ID, nest ID, drone ID, etc. Task creation result (success + task ID or failure + failure reason) Machine nest preparation The drone rises and the top of the nest opens Nest ID, drone ID Nest preparation result (success or failure) Drone takes off The drone leaves the nest and takes off Drone ID Drone takeoff result (success or failure) Drone flight guidance Instruct the drone to fly to a designated location Drone ID, target latitude and longitude Pointing flight (success or failure) Drone action execution Instruct the drone to perform actions Drone ID, action type (warning, shouting, taking pictures, etc.) Action execution result Drone returns Drone return command Drone ID Return execution results Task Archive Summarize all inspection information Task ID, inspection record Archiving result (success or failure) Embodiment 2, like Figure 1 As shown, this embodiment provides a drone inspection method based on multi-agent collaboration, the method comprising: S1. User Agent issues a command, and Setup Agent parses the user's command and calls tools to query the status of the machine nest and drone. If an abnormality is detected, it returns an abnormality report to the user, prompting the user to troubleshoot and provide new information in an interactive way. After confirming that the device status is normal, the execution right is transferred to the task planning agent Dynamic Planning Agent; S2, the task planning agent Dynamic Planning Agent plans the initial route according to the inspection requirements and geographical environment provided by the user, and transmits the planning results to the monitoring and execution agent Task Management Agent; if information supplement or confirmation is required during the planning process, the user is notified to supplement or confirm; S3, the monitoring and execution agent Task Management Agent executes each step in the plan in turn, acts as the "co-pilot" during the UAV's mission execution (the "main pilot" role is played by external modules such as the UAV's real-time obstacle avoidance), periodically monitors the execution of the task, and when the task cannot continue to be executed, notifies the preliminary preparation agent Setup Agent to solve the equipment abnormality or notifies the task planning agent Dynamic Planning Agent to re-plan until the flight is completed; S4: After the inspection is completed, the Completion Agent orders the drone to return and generates an inspection report based on the records of the inspection process.

[0027] During the entire multi-agent interaction process, all agents can communicate with the Memory module to record or read important information related to their own stage.

[0028] The technical implementation of all intelligent agents follows the "plan-act-reflect" design pattern, and formatted output is used in all three stages for easy parsing.

[0029] The “plan-act-reflect” design pattern enables the agent to dynamically adapt to the task environment, as follows: The "planning" step constrains the intelligent agent to first output thinking content when facing complex problems, and then output more sub-step instructions. Complex problems often need to be broken down through planning. Intelligent agents that lack planning often talk nonsense when faced with complex problems.

[0030] The "action" step provides the agent with the ability to call tools. For example, for the question "Please tell me the current time", without external tools, the agent cannot answer this question. With the "time query tool", the agent's answer history may be "Please tell me the current time -> To answer this question, I need to call the tool check_time() -> check_time call result: 2025 / 1 / 2 -> Now I know the answer, the current time is 2025 / 1 / 2".

[0031] The “reflection” step is used to constrain the agent to check and summarize. When the agent makes a mistake in a previous step and ends up making a mistake, it provides the agent with the ability to “look back” and increase the robustness of the response result.

[0032] by Figure 2 Taking the specific implementation of the monitoring and execution agent Task Management Agent in as an example, Task Management Agent first combines the task (system prompt word) and input information (route planning results) to plan, then selects tools to complete the subtasks in the planning content, and finally observes the context updated after the action, that is, inspects the real-time information, reflects and outputs the next action, whether to continue execution, complete execution, or transfer to other agents. For other agents, the migration can be completed by replacing the tasks, input information, action / reflection context, available tool list, and reflection return type in the Task Management Agent flowchart.

[0033] The prompt words for the preliminary preparation agent are as follows: You are an intelligent assistant responsible for helping users complete the preliminary preparations for drone inspection tasks. You can use the following tools: {tool_descs} You will assist users in completing the following tasks: a. Log in to the drone control platform and confirm user permissions; b. Check the status of the drone’s battery, motor, sensor and other equipment to ensure that the equipment is normal; c. Confirm the availability of the take-off airport and the aircraft nest to ensure that the drone can take off and return smoothly; d. If the user encounters a login failure or device problem, provide corresponding troubleshooting suggestions and assist in reconfiguration; e. If the equipment status is normal and the airport is available, transfer to the mission planning agent for mission planning.

[0034] The prompt words of the task planning agent are as follows: You are a route planning expert, responsible for recommending the most suitable route for the user's drone inspection mission and dynamically adjusting it during execution. You can use the following tools: {tool_descs} Your tasks include: a. Plan the initial route according to the inspection requirements and geographical environment provided by the user; b. If encountering obstacles, no-fly zones or weather changes, adjust the route in real time to ensure the safe flight of the drone; c. During the mission, if the user needs to modify the inspection area or mission objectives, re-plan the route and provide an optimization plan; d. Provide route planning results and make adjustments based on user feedback; e. Assist users to customize route planning when necessary to ensure that all inspection targets are covered; f. After the route planning is confirmed to be correct, it is transferred to the monitoring and execution agent.

[0035] The prompt words for the monitoring and execution agent are as follows: You are a task management assistant, responsible for real-time monitoring of the execution of tasks during drone inspections. You can use the following tools: {tool_descs} Your tasks include: a. Monitor the inspection progress of the drone, including the current inspection point, flight status, battery power, etc.; b. If there is any flight failure, out-of-range, equipment problem or any abnormal situation during the mission, immediately notify the user and provide corresponding solutions; c. Trigger necessary operational tasks, such as adjusting the flight path, restarting the mission, and conducting temporary inspections to ensure the successful completion of the mission; d. Feedback all intermediate outputs of the agent to users in real time to ensure that they understand the progress of the task and potential risks; the intermediate outputs include planning, tool calls, summary reflections, forwarding the right to speak, etc.; e. When a mission is abnormal, it can be transferred to the preparatory agent or mission planning agent for fault repair or route adjustment; f. After the task is completed, it is transferred to the summary agent for return and task archiving.

[0036] The summary agent's prompt words are as follows: You are an inspection report generation expert who is responsible for confirming the task status and generating detailed reports after the task is completed. You can use the following tools: {tool_descs} Your tasks include: a. Confirm whether the mission has been completed, including whether all inspection points have been covered, whether the flight is smooth, and whether the equipment status is normal. b. Summarize all inspection data, including route details, drone status, mission history, exception handling, etc. c. Generate a detailed inspection report based on the collected information. The report content should include task progress, key events and solutions, equipment inspection results, etc. The collected information includes: the execution results of the upstream agent, the memory module retrieval results, the information provided by the user, the results of the intelligent agent calling the tool, etc.

[0037] d. After the report is generated, the user is prompted that the task has been completed and a task summary is provided.

[0038] e. If there is any omission in the report, remind the user to supplement the relevant information to ensure the completeness of the report.

[0039] Based on the multi-agent UAV inspection system and method of the present invention, three experimental tests were conducted as shown in Table 2: Table 2 Experimental test table of multi-agent UAV inspection system and method

[0040] Experimental results show that the system and method of the present invention have achieved outstanding performance in substation inspection tasks, power transmission inspection tasks and water conservancy dam inspection tasks, thereby improving inspection efficiency.

Claims

1. A drone inspection system based on multi-agent collaboration, characterized in that: The system includes a preliminary preparation agent, a task planning agent, a monitoring and execution agent, a summary agent, and a memory module; each agent contains a drone tool set corresponding to its function; The preliminary preparation agent is responsible for helping the user complete the preliminary preparation work of the drone inspection task; The mission planning agent is responsible for recommending the most suitable route for the user's drone inspection mission and dynamically adjusting it during execution; The monitoring and execution agent is responsible for real-time monitoring of the execution of tasks during the drone inspection process; The summarizing agent is responsible for confirming the task status and generating a detailed report after the task is completed; The Memory module is responsible for information storage and information retrieval.

2. The multi-agent collaborative drone inspection system according to claim 1 is characterized in that: The tool set for the preliminary preparation of the intelligent agent includes: logging into the control platform, querying the list of drone nests, and querying drones; The tool set of the mission planning agent includes: route list query, mission status query and mission creation; The tool set of the monitoring and execution agent includes: drone query, nest preparation, drone takeoff, drone pointing flight and drone action execution; The tool set for the summarizing and inductive agent includes: drone return and task archiving.

3. A drone inspection method based on multi-agent collaboration, characterized in that: The method comprises the following steps: S1. The user issues a command, and the preliminary preparation agent parses the user's command and calls the machine nest list query tool and the drone query tool to query the machine nest and drone status. If an abnormality is detected, an abnormality report is returned to the user, prompting the user to troubleshoot the fault and provide new information in an interactive manner. After confirming that the equipment status is normal, the execution right is transferred to the task planning agent; S2. The task planning agent plans the initial route according to the inspection requirements and geographical environment provided by the user, and transmits the planning results to the monitoring and execution agent. If information supplementation or confirmation is required during the planning process, the user is notified to supplement or confirm; S3, the monitoring and execution agent executes each step in the plan in sequence, periodically monitors the execution of the task, and when the task cannot be continued, notifies the preliminary preparation agent to solve the equipment abnormality or notifies the task planning agent to re-plan until the flight is completed; S4. After the inspection is completed, the intelligent agent orders the drone to return and generates an inspection report based on the records of the inspection process.

4. The multi-agent collaborative drone inspection method according to claim 3 is characterized in that: During the entire multi-agent interaction process, all agents can communicate with the Memory module to record or read important information related to their own stage.

5. The method for unmanned aerial vehicle inspection based on multi-agent collaboration according to claim 3 is characterized in that: The technical implementation of all intelligent agents follows the "plan-act-reflect" design pattern, and formatted output is used in all three stages for easy parsing.

6. The multi-agent collaborative drone inspection method according to claim 3 is characterized in that: The prompt words for the tasks performed by the preliminary preparation agent are as follows: a. Log in to the drone control platform and confirm user permissions; b. Check the status of the drone equipment to ensure that it is normal; c. Confirm the availability of the take-off airport and the aircraft nest to ensure that the drone can take off and return smoothly; d. If the user encounters login failure or device abnormality, provide corresponding troubleshooting suggestions and assist in reconfiguration; e. If the equipment status is normal and the airport is available, transfer to the mission planning agent for mission planning.

7. The multi-agent collaborative drone inspection method according to claim 3 is characterized in that: The prompt words for the tasks performed by the task planning agent are as follows: a. Plan the initial route according to the inspection requirements and geographical environment provided by the user; b. If encountering obstacles, no-fly zones or weather changes, adjust the route in real time to ensure the safe flight of the drone; c. During the mission, if the user needs to modify the inspection area or mission objectives, re-plan the route and provide an optimization plan; d. Provide route planning results and make adjustments based on user feedback; e. Assist users to customize route planning when necessary to ensure that all inspection targets are covered; f. After the route planning is confirmed to be correct, it is transferred to the monitoring and execution agent.

8. The multi-agent collaborative drone inspection method according to claim 3 is characterized in that: The prompt words for the tasks performed by the monitoring and execution agent are as follows: a. Monitor the inspection progress of drones; b. If there is any flight failure, out-of-range, equipment problem or any abnormal situation during the mission, immediately notify the user and provide corresponding solutions; c. Trigger necessary operational tasks to ensure successful completion of the task; d. Feedback all intermediate outputs of the agent to the user in real time to ensure that they understand the task progress and potential risks; e. When a mission is abnormal, it can be transferred to the preparatory agent or mission planning agent for fault repair or route adjustment; f. After the task is completed, it is transferred to the summary agent for return and task archiving.

9. The multi-agent collaborative drone inspection method according to claim 3 is characterized in that: The prompt words summarizing the tasks performed by the agent are as follows: a. Confirm whether the mission has been completed, including whether all inspection points have been covered, whether the flight is smooth, and whether the equipment is in normal status; b. Summarize all inspection data; c. Generate a detailed inspection report based on the collected information. The report should include task progress, key events and solutions, equipment inspection results, etc.; d. After the report is generated, the user is prompted that the task has been completed and a task summary is provided; e. If there is any omission in the report, remind the user to supplement the relevant information to ensure the completeness of the report.

Images