Unmanned aerial vehicle inspection task generation method and device based on large language model

Abstract

The invention discloses an unmanned aerial vehicle inspection task generation method and device based on a large language model, and belongs to the technical field of unmanned aerial vehicle inspection. Processing and storing scene information visually perceived in historical data of an unmanned aerial vehicle inspection process, a motion track of an unmanned aerial vehicle body and skill calling generated in a flight process to form an unmanned aerial vehicle inspection memory; designing an unmanned aerial vehicle inspection task analysis agent, and analyzing a natural language instruction input by human into an unmanned aerial vehicle executable inspection task; if the corresponding air route exists in the air route database, the unmanned aerial vehicle inspection task analysis agent is used for matching the inspection air route; and for an inspection task without a pre-planned route, an intelligent route planning agent is constructed based on a large language model, candidate waypoints are automatically matched from the inspection memory, and a final inspection route is formed. The intelligent level and task execution efficiency of unmanned aerial vehicle inspection are improved, and the method is suitable for various unmanned aerial vehicle inspection scenes.

Description

Technical Field

[0001] The present invention relates to a method and device for generating drone inspection tasks based on a large language model, belonging to the technical field of drone inspection. Background Art

[0002] Drone inspection has been widely applied in various industries and scenarios. Traditional drone inspection usually adopts a mode of pre-planning routes and manually selecting routes. Even when using natural language dialogue, it generally only supports route selection and cannot understand and parse complex rules such as key targets to focus on during inspection and response behaviors after discovering targets.

[0003] The inspection instructions input by users are presented in natural language form, covering the inspection area, inspection objects, and behaviors after discovering targets. For example: "Go to the A River for inspection to check for fishing behaviors and illegal buildings. If a fishing behavior is found, immediately give an alarm and shout to drive them away; if an illegal building is found, take pictures as evidence." When users input such complex instructions, the system needs to parse them into structured task information before pushing them to the drone to execute the task. There is also a more complex situation, that is, the user instructions do not contain specific inspection areas and routes. In this case, a route needs to be intelligently planned based on the historical inspection process.

[0004] After being aligned with instructions, the large language model has the ability closest to the current human intelligence level. Through prompt engineering, the large language model can be customized to endow it with different professional capabilities, thereby solving specific tasks. Based on this, the present invention proposes a method and device for generating drone inspection tasks based on a large language model. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a method and device for generating drone inspection tasks based on a large language model, which can...

[0006] The technical solution adopted by the present invention to solve its technical problems is:

[0007] In a first aspect, a method for generating drone inspection tasks based on a large language model provided by an embodiment of the present invention includes the following steps:

[0008] Step S1, based on the inspection memory storage method, process and store the scene information visually perceived in the historical data of the drone inspection process, the movement trajectory of the drone body, and the skill calls that occur during the flight to form a drone inspection memory;

[0009] Step S2: Using the dialogue generation ability of the large language model, design an intelligent agent for parsing UAV inspection tasks. The intelligent agent for parsing UAV inspection tasks can parse the natural language instructions input by humans into inspection tasks executable by the UAV. The inspection tasks cover flight routes or waypoints, inspection content, and specific operation skills after detecting a target.

[0010] Step S3: If there is a corresponding flight route in the flight route database, use the intelligent agent for parsing UAV inspection tasks to match the inspection flight route; for inspection tasks without pre-planned flight routes, adopt an intelligent flight route planning method, and build an intelligent flight route planning agent based on the large language model. The intelligent flight route planning agent automatically matches candidate waypoints from the inspection memory and forms the final inspection flight route.

[0011] As a possible implementation of this embodiment, step S1 includes the following steps:

[0012] Step S11: Input the continuously captured images of the inspection environment by the pan-tilt camera during the inspection flight of the UAV into the neural network model deployed on the computing platform of the UAV body for inference to obtain structured visual perception information related to the inspection business. The structured visual perception information includes the target category and the position of the target in the image, and input the image of the scene into the large visual language model for inference to obtain the text-form scene information of the image content.

[0013] Step S12: Use the positioning system of the UAV itself to obtain the position information of the UAV during the inspection flight, and align the position information with the time of the captured image of the scene to obtain the position information at the time of taking the picture.

[0014] Step S13: Record the skill calls that occur after decision-making based on the inference information or other information of the image of the scene during the flight of the UAV, and record the names of the called skills.

[0015] Step S14: Combine the image, time, position, target, scene, and skill call information obtained after processing steps S11 to S13 into an inspection record, arrange all inspection records in chronological order to obtain a complete inspection memory, and use the database to persistently store the inspection memory.

[0016] As a possible implementation of this embodiment, the neural network model is an object detection model or a semantic segmentation model based on a neural network.

[0017] As a possible implementation of this embodiment, the positioning system is GPS, RTK, or SLAM.

[0018] As a possible implementation of this embodiment, the location information includes longitude, latitude, (x, y, z) coordinates, and altitude.

[0019] As a possible implementation of this embodiment, the name of the skill call adopts a naming method that conforms to the actual content of the skill.

[0020] As a possible implementation of this embodiment, step S1 further includes the following steps:

[0021] Step S15, according to the inspection instruction input by the user, retrieve the inspection memory saved in the database, and filter out the waypoints that meet the inspection instruction to guide the subsequent inspection task route planning.

[0022] As a possible implementation of this embodiment, step S2 includes the following steps:

[0023] Step S21, construct a conversational task parsing agent based on a large language model. The conversational task parsing agent conducts natural language interaction with the user under the drive of the large language model and receives the inspection task described by the user in natural language;

[0024] Step S22, the conversational task parsing agent structures the inspection task into three parts: route, inspection object, and action.

[0025] As a possible implementation of this embodiment, step S22 includes the following steps:

[0026] Through the route matching intelligent agent, according to the user input and the preset route name and route description, match the best route;

[0027] Through the inspection object recognition intelligent agent, according to the user input, recognize the inspection object required by the user;

[0028] Through the behavior recognition intelligent agent, for each inspection object, recognize the behavior expected by the user;

[0029] Through the dialogue agent, summarize the outputs of the route matching intelligent agent, the inspection object recognition intelligent agent, and the behavior recognition intelligent agent, generate an answer to interact with the user;

[0030] Form the matched route and the recognized rules into a valid structured inspection instruction and submit it to the UAV inspection system.

[0031] As a possible implementation of this embodiment, the process of matching the best route includes:

[0032] Set the prompt word template for the large language model;

[0033] Fill the <user instruction> field in the prompt template with the user instruction;

[0034] Traverse the flight routes in the flight route database, and fill the name and description of each flight route into the <flight route name>: <flight route description> field in the prompt template in sequence to obtain the filled prompt template;

[0035] Input the filled prompt template into the large language model to obtain the model output;

[0036] Judge whether the model output contains a flight route in the required format. If so, determine that the currently input flight route is a candidate flight route;

[0037] Update the candidate flight route list in the prompt template, and continue to traverse the unjudged flight routes in the flight route database until all flight routes are traversed, and output the final candidate flight route list.

[0038] As a possible implementation of this embodiment, the process of identifying the inspection object required by the user includes:

[0039] Fill the <user instruction> field in the prompt template with the user input;

[0040] Fill all the objects that can be inspected by the UAV into the <object name> list in the prompt template;

[0041] Input the filled prompt template into the large language model to obtain a list of inspection objects output in the format.

[0042] As a possible implementation of this embodiment, the process of identifying the behavior expected by the user includes:

[0043] Fill the <user instruction> field in the prompt template with the user input;

[0044] Fill the list of inspection objects output by the inspection object recognition intelligent agent into the <object name> list in the prompt template;

[0045] Fill the behavior capabilities of the UAV into the <behavior name> list in the prompt template;

[0046] Input the filled prompt template into the large language model to obtain a list of rules output in the format.

[0047] As a possible implementation of this embodiment, the process of generating an answer for interaction with the user includes:

[0048] Fill the outputs of the flight route matching intelligent agent, the inspection object recognition intelligent agent, and the behavior recognition intelligent agent and the user instruction into the prompt template respectively;

[0049] Input the filled prompt template into the large language model to obtain the content that should be replied to the user.

[0050] As a possible implementation of this embodiment, for the inspection task without a pre-planned route, an intelligent route planning method is adopted. An intelligent route planning agent is constructed based on the large language model. The intelligent route planning agent automatically matches candidate waypoints from the inspection memory and forms the final inspection route, including the following steps:

[0051] Step S31, when there is no pre-planned route, automatically search all trajectory points that meet the instruction from the inspection memory according to the user instruction;

[0052] Step S32, process all trajectory points through the embedding model to generate content vectors, and input the content of the user instruction into the embedding model for processing to obtain the instruction vector;

[0053] Step S33, calculate the similarity between the instruction vector and all content vectors, and use the scenario content represented by the content vector with a higher similarity as the candidate inspection point;

[0054] Step S34, use the large language model to judge each candidate inspection point. If the judgment result of the large language model is affirmative, then use this inspection point as one of the final inspection points;

[0055] Step S35, sort all the final inspection points in a certain order to obtain a planned route.

[0056] As a possible implementation of this embodiment, the embedding model is a pre-trained deep learning model, which is used to convert scenario content, discovered targets, skill call information, and the content of user instructions into vector representations.

[0057] As a possible implementation of this embodiment, the step of sorting all the final inspection points in a certain order includes:

[0058] Sort according to the geographical location relationship, time relationship, or task priority relationship between inspection points.

[0059] As a possible implementation of this embodiment, the inspection memory is all trajectory points and their related information recorded during the historical inspection process, including but not limited to scenario content, discovered targets, and skill call information.

[0060] As a possible implementation of this embodiment, when the intelligent route planning agent forms a route, it also considers the flight performance of the drone, environmental factors, and the priority of the inspection task.

[0061] In a second aspect, a drone inspection task generation device based on a large language model provided by an embodiment of the present invention includes:

[0062] An inspection memory storage module, configured to process and store, based on an inspection memory storage method, the scene information visually perceived in the historical data of the drone inspection process, the movement trajectory of the drone body, and the skill calls occurring during the flight, to form a drone inspection memory;

[0063] An inspection task parsing module, configured to design an inspection task parsing agent for the drone by using the dialogue generation ability of the large language model, where the inspection task parsing agent can parse the natural language instructions input by a human into inspection tasks executable by the drone, and the inspection tasks cover routes or waypoints, inspection contents, and specific operation skills after a target is discovered;

[0064] A route matching and planning module, configured to, if there is a corresponding route in the route database, use the inspection task parsing agent for the drone to match the inspection route; for an inspection task without a pre-planned route, an intelligent route planning method is adopted, and an intelligent route planning agent is constructed based on the large language model, and the intelligent route planning agent automatically matches candidate waypoints from the inspection memory and forms the final inspection route.

[0065] The beneficial effects of the technical solution of the embodiment of the present invention are as follows:

[0066] Aiming at solving the problems of low intelligence level in the generation of existing drone inspection tasks, inability to effectively parse complex user instructions, and low inspection efficiency when there is no pre-planned route, a method for generating a drone inspection task based on a large language model according to the technical solution of the embodiment of the present invention includes the following steps: Step S1, based on an inspection memory storage method, process and store the scene information visually perceived in the historical data of the drone inspection process, the movement trajectory of the drone body, and the skill calls occurring during the flight, to form a drone inspection memory; Step S2, use the dialogue generation ability of the large language model to design an inspection task parsing agent for the drone, where the inspection task parsing agent can parse the natural language instructions input by a human into inspection tasks executable by the drone, and the inspection tasks cover routes or waypoints, inspection contents, and specific operation skills after a target is discovered; Step S3, if there is a corresponding route in the route database, use the inspection task parsing agent for the drone to match the inspection route; for an inspection task without a pre-planned route, an intelligent route planning method is adopted, and an intelligent route planning agent is constructed based on the large language model, and the intelligent route planning agent automatically matches candidate waypoints from the inspection memory and forms the final inspection route. The present invention improves the intelligence level and task execution efficiency of drone inspections and is applicable to various drone inspection scenarios.

[0067] Through the patrol memory storage method, the present invention realizes the effective organization and persistent preservation of information during the UAV patrol process, providing data support for subsequent intelligent route planning and task analysis; through the patrol task analysis method, an intelligent agent for UAV patrol task analysis based on a large language model realizes the accurate analysis of complex user instructions and the generation of structured task information, improving the intelligent level of UAV patrol; through the intelligent route planning method, the automatic generation and efficient execution of patrol tasks without pre-planned routes are realized, improving the patrol efficiency and effect. Description of the Drawings

[0068] Figure 1 It is a flowchart of a method for generating UAV patrol tasks based on a large language model shown according to an exemplary embodiment;

[0069] Figure 2 It is a schematic structural diagram of a device for generating UAV patrol tasks based on a large language model shown according to an exemplary embodiment. Detailed Embodiments

[0070] To more clearly illustrate the technical features of the solution of the present invention, the present invention will be elaborated in detail below through specific embodiments and in conjunction with its drawings.

[0071] As Figure 1 shown, a method for generating UAV patrol tasks based on a large language model provided by an embodiment of the present invention includes the following steps:

[0072] Step S1, based on the patrol memory storage method, process and store the scene information visually perceived in the historical data of the UAV patrol process, the movement trajectory of the UAV body, and the skill calls occurring during the flight to form a UAV patrol memory;

[0073] Step S2, using the dialogue generation ability of the large language model, design an intelligent agent for UAV patrol task analysis, and the intelligent agent for UAV patrol task analysis can parse the natural language instructions input by humans into UAV-executable patrol tasks, where the patrol tasks cover routes or waypoints, patrol content, and specific operation skills after discovering a target;

[0074] Step S3, if there is a corresponding route in the route database, use the intelligent agent for UAV patrol task analysis to match the best patrol route; for patrol tasks without pre-planned routes, adopt the intelligent route planning method, build an intelligent route planning intelligent agent based on the large language model, and the intelligent route planning intelligent agent automatically matches candidate waypoints from the patrol memory and forms the final patrol route.

[0075] As a possible implementation of this embodiment, step S1 includes the following steps:

[0076] Step S11, continuously input the images of the inspection environment captured by the pan-tilt camera during the inspection flight of the UAV into the neural network model deployed on the computing platform of the UAV body for inference to obtain structured visual perception information related to the inspection service. The structured visual perception information includes the target category and the position of the target in the image, and input the image of the scene into the large visual language model for inference to obtain the scene information in text form of the image content;

[0077] Step S12, use the positioning system of the UAV itself to obtain the point position information of the UAV during the inspection flight, and align the point position information with the time of the captured image of the scene to obtain the position information at the time of capturing the image;

[0078] Step S13, record the skill calls that occur after decision-making based on the inference information or other information of the image of the scene during the flight of the UAV, and record the name of the called skill;

[0079] Step S14, form an inspection record with the images, time, position, target, scene, and skill call information obtained after processing steps S11 to S13, arrange all inspection records in chronological order to obtain a complete inspection memory, and use the database to persistently save the inspection memory.

[0080] As a possible implementation of this embodiment, the neural network model is an object detection model or a semantic segmentation model based on a neural network.

[0081] As a possible implementation of this embodiment, the positioning system is GPS, RTK, or SLAM.

[0082] As a possible implementation of this embodiment, the position information includes longitude, latitude, (x, y, z) coordinates, and altitude.

[0083] As a possible implementation of this embodiment, the name of the skill call adopts a naming method that conforms to the actual content of the skill.

[0084] As a possible implementation of this embodiment, step S1 further includes the following steps:

[0085] Step S15, according to the inspection instructions input by the user, retrieve the inspection memory saved in the database, and filter out the waypoints that meet the inspection instructions for guiding the subsequent inspection task route planning.

[0086] As a possible implementation of this embodiment, step S2 includes the following steps:

[0087] Step S21: Build a conversational task parsing agent based on a large language model. The conversational task parsing agent interacts with the user in natural language under the drive of the large language model and receives the inspection task described by the user in natural language.

[0088] Step S22: The conversational task parsing agent structures the inspection task into three parts: flight route, inspection object, and action.

[0089] As a possible implementation of this embodiment, step S22 includes the following steps:

[0090] Through the flight route matching intelligent agent, according to the user input and the preset flight route name and flight route description, match the best flight route;

[0091] Through the inspection object recognition intelligent agent, according to the user input, recognize the inspection object required by the user;

[0092] Through the behavior recognition intelligent agent, for each inspection object, recognize the behavior expected by the user;

[0093] Through the dialogue agent, summarize the outputs of the flight route matching intelligent agent, the inspection object recognition intelligent agent, and the behavior recognition intelligent agent, generate an answer to interact with the user;

[0094] Form an effective structured inspection instruction with the matched flight route and the recognized rules, and submit it to the UAV inspection system.

[0095] As a possible implementation of this embodiment, the process of matching the best flight route includes:

[0096] Set a prompt template for the large language model;

[0097] Fill the user instruction into the <user instruction> field in the prompt template;

[0098] Traverse the flight routes in the flight route database, and fill the name and description of each flight route into the <flight route name>: <flight route description> field in the prompt template in turn to obtain the filled prompt template;

[0099] Input the filled prompt template into the large language model to obtain the model output;

[0100] Judge whether the flight route in the required format is contained in the model output. If so, determine that the currently input flight route is a candidate flight route;

[0101] Update the candidate flight route list in the prompt template, and continue to traverse the unjudged flight routes in the flight route database until all flight routes are traversed, and output the final candidate flight route list.

[0102] As a possible implementation of this embodiment, the process of identifying the inspection objects required by the user includes:

[0103] Fill the user input into the <user instruction> field in the prompt template;

[0104] Fill all the objects that can be inspected by the drone into the <object name> list in the prompt template;

[0105] Input the filled prompt template into the large language model to obtain a list of inspection objects output in a format.

[0106] As a possible implementation of this embodiment, the process of identifying the behavior expected by the user includes:

[0107] Fill the user input into the <user instruction> field in the prompt template;

[0108] Fill the list of inspection objects output by the inspection object recognition intelligent agent into the <object name> list in the prompt template;

[0109] Fill the behavior capabilities of the drone into the <behavior name> list in the prompt template;

[0110] Input the filled prompt template into the large language model to obtain a list of rules output in a format.

[0111] As a possible implementation of this embodiment, the process of generating a response to interact with the user includes:

[0112] Fill the outputs of the route matching intelligent agent, the inspection object recognition intelligent agent, and the behavior recognition intelligent agent, as well as the user instruction, into the prompt template respectively;

[0113] Input the filled prompt template into the large language model to obtain the content that should be replied to the user.

[0114] As a possible implementation of this embodiment, for the inspection task without a pre-planned route, an intelligent route planning method is adopted, and an intelligent route planning agent is constructed based on the large language model. The intelligent route planning agent automatically matches candidate waypoints from the inspection memory and forms the final inspection route, including the following steps:

[0115] Step S31, when there is no pre-planned route, automatically search all the trajectory points that meet the instruction from the inspection memory according to the user instruction;

[0116] Step S32, process all the trajectory points through the embedding model to generate content vectors, and input the content of the user instruction into the embedding model for processing to obtain an instruction vector;

[0117] Step S33: Calculate the similarity between the instruction vector and all content vectors, and use the scenario content represented by the content vectors with higher similarity as candidate inspection points;

[0118] Step S34: Use a large language model to determine each candidate inspection point. If the determination result of the large language model is affirmative, then use this inspection point as one of the final inspection points;

[0119] Step S35: Sort all the final inspection points in a certain order to obtain a planned flight route.

[0120] As a possible implementation of this embodiment, the embedding model is a pre-trained deep learning model, which is used to convert scenario content, discovered targets, skill call information, and the content of user instructions into vector representations.

[0121] As a possible implementation of this embodiment, the step of sorting all the final inspection points in a certain order includes:

[0122] Sort according to the geographical location relationship, time relationship, or task priority relationship between inspection points.

[0123] As a possible implementation of this embodiment, the inspection memory is all trajectory points and their related information recorded during the historical inspection process, including but not limited to scenario content, discovered targets, and skill call information.

[0124] As a possible implementation of this embodiment, when the intelligent flight route planning agent forms a flight route, it also considers the flight performance of the drone, environmental factors, and the priority of the inspection task.

[0125] The present invention can intelligently parse natural language instructions input by humans, convert the inspection intention into an inspection task executable by the drone, and improve the generation efficiency and flexibility of the inspection task; utilize the drone inspection memory, including scenario information, movement trajectories, and skill call data, to provide rich data support for intelligent flight route planning, making the generated inspection task more in line with actual requirements; through the combined use of the embedding model and the large language model, realize the function of automatically matching candidate flight points from the inspection memory and forming a flight route, and improve the intelligent level of flight route planning.

[0126] As Figure 2 shown, a drone inspection task generation device provided by an embodiment of the present invention includes:

[0127] The patrol memory storage module is used to process and store the scene information visually perceived in the historical data of the UAV patrol process, the movement trajectory of the UAV itself, and the skill calls that occur during the flight based on the patrol memory storage method, forming the UAV patrol memory;

[0128] The patrol task parsing module is used to design a UAV patrol task parsing agent by utilizing the dialogue generation ability of a large language model. The UAV patrol task parsing agent can parse the natural language instructions input by humans into UAV-executable patrol tasks, where the patrol tasks cover routes or waypoints, patrol content, and specific operation skills after discovering a target;

[0129] The route matching and planning module is used to, if there is a corresponding route in the route database, utilize the UAV patrol task parsing agent to match the best patrol route; for patrol tasks without a pre-planned route, an intelligent route planning method is adopted, and an intelligent route planning agent is constructed based on a large language model. The intelligent route planning agent automatically matches candidate waypoints from the patrol memory and forms the final patrol route.

[0130] The present invention proposes a method that combines three methods: a patrol memory storage method, a patrol task parsing method, and an intelligent route planning method, to intelligently parse user instructions and generate patrol tasks and routes. For scenarios where the patrol route is clearly included in the instructions, it automatically matches the best route and patrol rules in the preset routes; for scenarios where there is no restriction on a fixed route, it automatically intelligently plans the route based on the previous patrol memory. All human-computer interaction processes are completely carried out through natural language, achieving a simple, intelligent, and efficient human-computer interaction effect. Different perception models, large visual language models, and large language models are used to solve the problem of generating patrol tasks in UAV human-computer interaction. The patrol memory storage method proposes how to process and store the scene information visually perceived during the UAV patrol process, the movement trajectory of the UAV body, and the skill calls that occur during the flight, which is equivalent to the UAV patrol memory. These memories are used to guide subsequent patrol tasks, that is, how to select waypoints that meet the patrol instructions in route planning. The patrol task parsing method is a UAV patrol task parsing agent designed based on the dialogue generation ability of a large language model, responsible for parsing the natural language instructions input by humans into UAV-executable patrol tasks, and converting the patrol intention into a series of flight actions of the UAV. The flight actions mainly involve routes or waypoints, patrol content, and specific skills after discovering a specific target. When there is a corresponding route in the route database, the patrol task parsing method is responsible for matching the best route. For patrol tasks without a pre-planned route, the intelligent route planning method constructs a dedicated agent based on a large language model: the intelligent route planning agent, which is responsible for automatically matching candidate waypoints from the patrol memory in the patrol memory storage method and forming the final patrol route.

[0131] The present invention proposes a method that comprehensively utilizes an inspection memory storage method, an inspection task parsing method, and an intelligent route planning method to intelligently parse user instructions and generate inspection tasks and routes. In the scenario where the instruction clearly includes an inspection route, the best route and corresponding inspection rules are automatically matched from the preset routes; while in the scenario without a fixed route restriction, the route will be intelligently planned automatically based on the previous inspection memory. The entire human-computer interaction process is completely based on natural language, achieving a simple, intelligent, and efficient human-computer interaction effect. With the help of different perception models, large vision-language models, and large language models, the problem of generating inspection tasks for unmanned aerial vehicle (UAV) human-computer interaction is effectively solved.

[0132] Among them, the inspection memory storage method elaborates on how to process and store the scene information visually perceived during the UAV inspection process, the movement trajectory of the UAV itself, and the skill invocation situation during the flight. These information constitute the inspection memory of the UAV and can be used to guide subsequent inspection tasks, especially in route planning to screen waypoints that meet the inspection instructions.

[0133] The inspection task parsing method is an intelligent agent for UAV inspection task parsing designed based on the dialogue generation ability of a large language model. Its responsibility is to parse the natural language instructions input by humans into inspection tasks executable by the UAV, and convert the inspection intention into a series of flight actions of the UAV. These flight actions mainly cover routes or waypoints, inspection content, and specific skill operations after discovering specific targets. When there is a corresponding route in the route database, this method is responsible for matching the best route.

[0134] For inspection tasks without pre-planned routes, the intelligent route planning method constructs a dedicated intelligent agent - the intelligent route planning intelligent agent based on a large language model. This intelligent agent is responsible for automatically matching candidate waypoints from the inspection memory recorded by the inspection memory storage method, and then forming the final inspection route.

[0135] I. Inspection memory storage method.

[0136] This method is applied during the UAV inspection process, that is, the UAV will generate visual inference information, movement trajectories, and skill invocations during the inspection flight, and the inspection memory module is responsible for organizing and persistently storing this information.

[0137] a) Process and save visual reasoning information. The gimbal camera of the drone continuously captures images of the inspection environment, and uses the neural network model required for the inspection service deployed on the computing platform of the drone body to reason about the image. The specific process is as follows: At a certain moment, the on-board camera of the drone captures an image and inputs the image into the service model. This model is generally an object detection or semantic segmentation model based on a neural network. After model reasoning, that is, the neural network model performs forward propagation calculations, structured visual perception information such as the target categories related to the inspection service in the image and the positions of the targets in the image is obtained. At the same time, the image is input into a large visual language model, and after model reasoning, scene information in text form of the content contained in the image is obtained. After being processed by this method, the content seen by the drone during flight is output. When the user requests to inspect a certain equipment object, this content can be used to filter historical moments that meet the inspection conditions.

[0138] b) Process and save the motion trajectory. Use the positioning system of the drone itself, such as GPS, RTK, SLAM, etc. to generate the position points of the drone at each moment. Align with the time of the camera image in step a) to obtain the position information when the image is taken, such as longitude, latitude, (x, y, z) coordinates, altitude, etc.

[0139] c) Process and save skill call information. During the flight of the drone, when a target is found, an action will be taken. The action corresponds to the skills preset in the drone one by one, such as orbiting, precise photographing, tracking, shouting to drive away, etc. If a skill call occurs after decision-making based on the reasoning information or other information of this image, record the name of the called skill. The name of the skill call adopts a naming method that conforms to the actual content of the skill, that is, one can know what action the skill is actually performing through the name. Therefore, only the name of the skill called during flight needs to be saved for subsequent analysis of what targets the drone found and what actions it took at a certain moment and position.

[0140] d) After being processed by steps a), b), and c), the images, time, position, targets, scenes, and skill calls captured by the drone during flight are obtained. Such a set of information forms an inspection record. Arrange all the inspection records in chronological order to obtain a complete inspection memory. Use a database to persistently save these memories for subsequent retrieval.

[0141] The inspection memory storage method of the present invention forms an inspection memory by processing and storing visual perception information, motion trajectory, and skill call information during the drone inspection process, improving the utilization rate of inspection information; persistently saves the inspection memory through a database, facilitating subsequent retrieval and providing strong support for the subsequent inspection task route planning; improving the inspection efficiency and accuracy and reducing the inspection cost.

[0142] II. Patrol task parsing method.

[0143] Build a conversational task parsing agent based on a large language model. Driven by the large language model, this agent conducts natural language interaction with users. Users describe patrol tasks in natural language, and the agent generates drone patrol tasks according to the conversation content. Its ultimate goal is to structure a patrol task into three parts: route, patrol object, and action.

[0144] a) Build four roles based on the large language model, which are responsible for matching routes, identifying the patrol objects required by users, identifying the behaviors expected by users for each patrol object, and an agent responsible for conversing with users. The following is the construction method of the four agents:

[0145] b) Route matching intelligent agent. When manually defining a route, in addition to waypoints and order, set a name with practical meaning and a route description for each route. The route description includes the area covered by the route and the patrol objects. The route matching intelligent agent matches the best route according to the user input, as well as the route name and description.

[0146] First, set prompt words for the large language model, and the template is as follows:

[0147]

[0148] After the user enters an instruction, fill the user instruction into the <user instruction> field in the above prompt word template, and then traverse the routes in the route database. First, fill the name and description of the first route into the above template <route name>: <route description>. After the prompt word template is filled, input it into the large language model to obtain the model output. If the output does not contain a route in the required format, it is determined that the currently input route does not meet the user's needs; otherwise, if the output contains a route in the required format, it is determined that the currently input route is a candidate route.

[0149] After each route is determined, update the candidate route list in the prompt word template, then take an undetermined route and add it to the list, put it into the prompt word template, and continue to input it into the large language model.

[0150] Until all routes are traversed, output the final candidate route list.

[0151] c) Patrol object recognition intelligent agent.

[0152]

[0153] Fill the user input into the <User Instruction> field in the above prompt template, and fill all the objects that can be inspected by the drone into the <Object Name> list in the above prompt template. After filling, input it into the large language model and output the inspection object list according to the format.

[0154] d) Behavior recognition intelligent agent.

[0155]

[0156] Fill the user input into the <User Instruction> field in the above prompt template, fill the inspection object list output in step c) into the <Object Name> list in the above prompt template, and fill the behavior capabilities of the drone into the <Behavior Name> list in the above prompt template. After filling, input it into the large language model and output the rule list according to the format.

[0157] e) Dialogue agent. The dialogue agent needs to summarize the outputs of the three intelligent agents in steps b), c), and d), and generate answers to interact with the user according to the instructions input by the user.

[0158]

[0159] Fill the content of the intelligent agents in steps b), c), and d) and the user instructions of the dialogue agent in this step into the above prompt template respectively, and input it into the large language model to obtain the content that should be replied to the user. At the same time, the flight path and rules matched by the flight path matching intelligent agent and the behavior recognition intelligent agent form a valid structured inspection instruction, which is submitted to the drone inspection system. The drone can determine the flight path trajectory according to the flight path name, start the inspection, and use the rules during the inspection to execute the skills required in the rules when a target is found.

[0160] The inspection task parsing method of the present invention improves the accuracy and flexibility of inspection task parsing, and can handle various complex inspection task descriptions; realizes the structuring of inspection tasks, which is convenient for the execution and monitoring of the drone inspection system; and improves the user experience and satisfaction through conversational interaction.

[0161] III. Intelligent flight path planning method.

[0162] The flight path matching intelligent agent in the inspection task parsing method deals with the scenario where there is a pre-planned flight path. When there is no flight path, that is, in the intelligent flight path planning mode, it is necessary to automatically search all the trajectory points that meet the instructions from the inspection memory according to the user instructions, sort these trajectory points in a certain order, and finally obtain a planned flight path. Since the trajectory points stored in the inspection memory storage method are generated during each inspection process and the data volume is huge, the present invention gives a two-stage screening method.

[0163] a) Process the scene content, discovered targets, and skill call information in all memories obtained by the patrol inspection memory storage method through an embedding model to generate content vectors corresponding to each frame of the image. Similarly, input the content of the user instruction into the embedding model for processing to obtain an instruction vector. Calculate the similarity between the instruction vector and all content vectors, such as cosine similarity. The scene content represented by the content vector with a higher similarity is used as a candidate patrol inspection point.

[0164] b) For each candidate patrol inspection point, use a large language model to determine whether it is used as the final patrol inspection point. The prompt words are as follows:

[0165]

[0166] For a candidate patrol inspection point, if the large language model answers "yes", then determine that this patrol inspection point is one of the final patrol inspection points; otherwise, this patrol inspection point is not within the scope of this patrol inspection. Finally, all patrol inspection points are obtained.

[0167] The intelligent route planning method of the present invention realizes intelligent route planning without a pre-planned route by combining an embedding model and a large language model, improving the automation degree and efficiency of the patrol inspection task. At the same time, the present invention also provides a two-stage screening method, effectively reducing the complexity of data processing and improving the accuracy and reliability of route planning.

[0168] When the present invention is specifically implemented, model deployment is first required. Deploy the business model, large language model, large vision language model, and embedding model to the computer system; then perform manual route planning. In the patrol inspection area, manually delimit the patrol inspection route, name the route name, and provide route description information. Secondly, perform human-computer interaction to generate a patrol inspection task. Use the task parsing method in the patrol inspection task parsing method, and use the large language model and multiple different agents to perform natural language interaction with the user to parse and generate a patrol inspection task. If there is no determined route, use the intelligent route planning method for route planning; finally, generate patrol inspection memories. The unmanned aerial vehicle autonomously flies according to the route, and the images are input into the business model and the large vision language model to obtain the patrol inspection targets and scene content, as well as the skill calls generated for the targets, and store them in the database.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that it is still possible to modify the specific implementation manners of the present invention or make equivalent replacements. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall be covered by the protection scope of the claims of the present invention.

Claims

1. A method for generating UAV inspection tasks based on large language models, characterized in that, The steps include: Step S1: Based on the inspection memory storage method, the visually perceived scene information, the motion trajectory of the drone body, and the skill calls that occurred during the flight in the historical data of the drone inspection process are processed and stored to form the drone inspection memory; Step S2: Leveraging the dialogue generation capabilities of large-scale language models, we design a drone inspection task parsing agent. This agent can parse natural language commands input by humans into inspection tasks executable by drones. These inspection tasks include routes or waypoints, inspection content, and specific operational skills after discovering a target. Step S3: If the corresponding route exists in the route database, the inspection route is matched using the drone inspection task parsing agent; For inspection tasks without pre-planned routes, an intelligent route planning method is adopted. An intelligent route planning agent is constructed based on a large language model. The intelligent route planning agent automatically matches candidate waypoints from the inspection memory and forms the final inspection route.

2. The method for generating an unmanned aerial vehicle inspection task based on a large language model according to claim 1, wherein, The step S1 includes the following steps: Step S11: The images of the inspection environment continuously captured by the drone's gimbal camera during the inspection flight are input into a neural network model deployed on the computing platform of the drone body for inference, thereby obtaining structured visual perception information related to the inspection business. The structured visual perception information includes the target category and the target's position in the image. The image is then input into a large visual language model for inference to obtain scene information in the form of text of the image content. Step S12: using the drone's own positioning system to obtain the drone's position information during the inspection flight, and aligning the position information with the time of the captured image to obtain the position information when the image was captured; Step S13, recording the skill calls that occur during the flight of the drone after decision-making based on the inference information or other information of the screen image, and recording the name of the called skill; Step S14, the image, time, location, target, scene, and skill call information obtained after processing steps S11 to S13 are combined into an inspection record, all inspection records are arranged in chronological order to obtain a complete inspection memory, and the inspection memory is persisted in the database.

3. The method for generating an unmanned aerial vehicle inspection task based on a large language model according to claim 1, wherein The step S2 comprises the following steps: Step S21: construct a conversational task parsing agent based on the large language model. Driven by the large language model, the conversational task parsing agent interacts with the user in natural language and receives the inspection task described by the user in natural language. In step S22, the conversational task parsing agent structures the inspection task into three parts: route, inspection object, and action.

4. The method for generating UAV inspection tasks based on a large language model according to claim 3 is characterized in that: The step S22 includes the following steps: The route matching intelligent agent matches the best route based on user input and preset route names and descriptions; Through the inspection object recognition intelligent agent, the inspection object required by the user is identified based on the user input; Through the behavior recognition intelligent agent, the user's expected behavior is identified for each inspection object; Through the dialogue agent, summarize the outputs of the route matching intelligent agent, the inspection object recognition intelligent agent, and the behavior recognition intelligent agent, generate answers to interact with the user; Formulate the matched route and the recognized rules into valid structured inspection instructions and submit them to the UAV inspection system.

5. The method for generating UAV inspection tasks based on a large language model according to claim 4 is characterized in that: The process of matching the best route includes: Set the prompt template for the large language model; Fill the user instruction into the <user instruction> field in the prompt template; Traverse the routes in the route database, and fill the name and description of each route into the <route name>: <route description> field in the prompt template in turn to obtain the filled prompt template; Input the filled prompt template into the large language model to obtain the model output; Judge whether the route in the required format is contained in the model output. If so, determine the currently input route as a candidate route; Update the candidate route list in the prompt template, and continue to traverse the unjudged routes in the route database until all routes are traversed, and output the final candidate route list.

6. The method for generating an unmanned aerial vehicle inspection task based on a large language model according to claim 4, wherein, The process of recognizing the inspection objects required by the user includes: Fill the user input into the <user instruction> field in the prompt template; Fill all the objects that can be inspected by the UAV into the <object name> list in the prompt template; Input the filled prompt template into the large language model to obtain the inspection object list output in the format.

7. The method for generating an unmanned aerial vehicle inspection task based on a large language model according to claim 4, wherein The process of recognizing the behavior expected by the user includes: Fill the user input into the <user instruction> field in the prompt template; Fill the inspection object list output by the inspection object recognition intelligent agent into the <object name> list in the prompt template; Fill the behavior capabilities of the UAV into the <behavior name> list in the prompt template; Input the filled prompt template into the large language model to obtain the rule list output in the format.

8. The method for generating an unmanned aerial vehicle inspection task based on a large language model according to claim 4, wherein, The process of generating answers to interact with the user includes: Fill the outputs of the route matching intelligent agent, the inspection object recognition intelligent agent, the behavior recognition intelligent agent, and the user instruction into the prompt template respectively; Input the filled prompt template into the large language model to obtain the content that should be replied to the user.

9. The method for generating an unmanned aerial vehicle inspection task based on a large language model according to any one of claims 1-8, characterized in that, For the inspection task without a pre-planned route, an intelligent route planning method is adopted. An intelligent route planning agent is constructed based on the large language model. The intelligent route planning agent automatically matches candidate waypoints from the inspection memory and forms the final inspection route, including the following steps: Step S31, when there is no pre-planned route, automatically search all track points that meet the instruction from the inspection memory according to the user instruction; Step S32, process all track points through the embedding model to generate content vectors, and input the content of the user instruction into the embedding model for processing to obtain the instruction vector; Step S33, calculate the similarity between the instruction vector and all content vectors, and use the scenario content represented by the content vector with a higher similarity as the candidate inspection point; Step S34, use the large language model to judge each candidate inspection point. If the judgment result of the large language model is affirmative, then use this inspection point as one of the final inspection points; Step S35: Sort all the final inspection points in a certain order to obtain a planned flight path.

10. An unmanned aerial vehicle inspection task generation device based on a large language model, characterized in that, Including: An inspection memory storage module, which is used to process and store the scene information visually perceived in the historical data of the UAV inspection process, the movement trajectory of the UAV itself, and the skill calls that occur during the flight based on the inspection memory storage method, and form the UAV inspection memory. An inspection task parsing module, which is used to design an inspection task parsing agent for UAVs by using the dialogue generation ability of large language models. The inspection task parsing agent for UAVs can parse the natural language instructions input by humans into inspection tasks that can be executed by UAVs, where the inspection tasks cover flight paths or waypoints, inspection contents, and specific operation skills after discovering targets. A flight path matching and planning module, which is used to match the inspection flight path by using the inspection task parsing agent for UAVs if there is a corresponding flight path in the flight path database. For inspection tasks without a pre-planned flight path, an intelligent flight path planning method is adopted, and an intelligent flight path planning agent is constructed based on large language models. The intelligent flight path planning agent automatically matches candidate waypoints from the inspection memory and forms the final inspection flight path.

Images