Unmanned aerial vehicle instruction semantic self-evolution method and system based on execution feedback loop

Abstract

The application provides a kind of unmanned aerial vehicle instruction semantic self-evolution method and system based on executing feedback closed loop, method includes: receiving natural language instruction, it is converted into domain-specific language DSL by large language model;The DSL is parsed into unmanned aerial vehicle control instruction and executes task;Based on the success or failure evaluation of task execution result, automatically build training sample set, wherein the instruction and DSL mapping pair marked as negative sample with model high confidence output but execution failure;Using contrast learning strategy, the large language model is fine-tuned using the sample set;System includes task planning, decision execution, effect evaluation, sample mining and model fine-tuning module, form " execution-evaluation-learning " closed loop mechanism.The application introduces execution feedback and closed loop learning mechanism, effectively solves the ambiguity problem of natural language instruction, realizes the autonomous continuous optimization of unmanned aerial vehicle semantic understanding ability.

Description

Technical Field

[0001] This invention relates to the fields of intelligent control of unmanned aerial vehicles (UAVs), natural language processing, and deep learning, specifically to a method and system for semantic self-evolution of UAV commands based on execution feedback loop. Background Technology

[0002] With the continuous improvement of the intelligence level of drones, they have been widely used in many fields such as power line inspection, agricultural monitoring, facility inspection, and emergency rescue. In these complex business scenarios, how to achieve efficient, natural, and reliable human-machine interaction to reduce the operating threshold and improve task execution efficiency has become a key issue.

[0003] Currently, the mainstream interaction and control methods for drones mainly include the following: Manual remote control: The operator directly controls the drone's flight using a specialized handle. This method offers high control precision, but it is highly dependent on the operator's professional skills and experience, has a high learning curve, and is difficult to execute complex automated inspection tasks.

[0004] Ground station pre-planned tasks: Flight waypoints and maneuvers are pre-planned in the ground station software, and a structured task script is generated and uploaded for execution. This method is suitable for repetitive tasks, but it lacks flexibility and cannot cope with temporary, dynamic command changes.

[0005] Basic voice command control: Some studies attempt to introduce speech recognition technology to trigger preset flight actions by recognizing limited keywords (such as "take off," "land," and "turn left"). However, this method is essentially an extension of a static "keyword-command" mapping table, which has significant limitations: Weak semantic understanding: Unable to parse natural language instructions with multiple modifiers, contextual dependencies, or ambiguous objectives (e.g., "Fly closer and check that red device," "Go back to where we found the problem last time"). This is known as the "semantic gap"—the huge difference between the ambiguity of natural language and the precise parameters required by the flight control system.

[0006] The model is static and cannot evolve: its instruction recognition model (whether rule-based or a simple machine learning model) is usually trained offline before deployment. Once deployed, it becomes a static system. When the system misparses instructions and causes task failure, it cannot efficiently correct the error, the same error may occur repeatedly, and it lacks adaptive optimization capabilities.

[0007] Open-loop systems lack feedback: Existing solutions are mostly one-way "instruction input - action execution" open-loop processes. After the system executes the instruction, it does not care about or can not automatically evaluate whether the task completion effect matches the user's intention. The knowledge of whether the execution was successful or not cannot be fed back to the understanding model, and an optimization loop cannot be formed.

[0008] In summary, existing technologies generally suffer from core problems such as static models lacking evolutionary capabilities, open-loop systems lacking feedback optimization, and semantic gaps leading to ambiguous understanding. In recent years, large-scale language models have demonstrated powerful capabilities in natural language understanding, offering new possibilities for UAV natural language interaction. However, directly applying general-purpose large-scale language models to UAV control still faces challenges: the commands generated by the model may not conform to domain safety specifications or specific flight control protocols, and unpredictable risks exist. Furthermore, simply fine-tuning the model using successful mission data is inefficient and fails to correct persistent misunderstandings already formed by the model. Therefore, existing technologies lack an intelligent UAV interaction system capable of continuous self-optimization during real-world mission execution, understanding complex contextual commands, and learning from errors. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for self-evolution of UAV command semantics based on execution feedback closed loop, which solves the problem that existing UAV intelligent interaction systems lack continuous self-optimization capabilities and the ability to learn from errors.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: A self-evolutionary method for UAV command semantics based on execution feedback closed loop includes the following steps: S1. Receive natural language instructions, use a large language model to convert the natural language instructions into a domain-specific language (DSL), and parse the DSL into control code for the UAV to perform the corresponding inspection task; S2. Evaluate the execution results of the inspection task and generate a success or failure result label; S3. Based on the result labels, mark the corresponding (natural language instructions, DSL) mapping pairs as training samples; S4. Using the labeled training samples, the large language model is fine-tuned through a contrastive learning strategy to optimize the mapping relationship between natural language instructions and DSL, and the fine-tuned model is used for subsequent instruction parsing. Furthermore, parsing the DSL into UAV control code specifically includes: combining the DSL instructions with the current UAV state information according to preset state machine logic to generate a task execution logic sequence; and translating the task execution logic sequence into executable code of the underlying flight control protocol.

[0011] Furthermore, the step of marking the corresponding (natural language instruction, DSL) mapping pair as training samples specifically includes: if the execution result label is successful, then the mapping pair is marked as a positive sample; if the execution result label is failure, then the mapping pair is marked as a negative sample, wherein the negative sample is a mapping pair whose confidence level is higher than a preset threshold when the large language model generates the corresponding DSL but fails to execute, and is used as a difficult negative sample for fine-tuning the large language model.

[0012] Furthermore, the fine-tuning of the large language model through the contrastive learning strategy specifically includes: using a contrastive loss function in the semantic embedding space to reduce the distance between the natural language instruction representation and the corresponding DSL representation in the positive samples, and to increase the distance between the natural language instruction representation and the corresponding DSL representation in the difficult negative samples.

[0013] Furthermore, the method also includes: constructing and maintaining a domain knowledge graph for structured storage of historical inspection task data, the data including natural language instructions, converted DSLs, task execution environment information, and execution result tags; when converting natural language instructions into DSLs, auxiliary parsing is also performed based on relevant context information retrieved from the domain knowledge graph.

[0014] Furthermore, the domain knowledge graph is dynamically updated based on the new inspection task execution results and tags: parsing the new inspection task data, automatically adding or updating entity nodes representing natural language instructions, DSL instructions, and environmental objects in the domain knowledge graph; establishing new semantic relationship edges between the natural language instruction entities, DSL entities, and environmental context entities based on the result tags, or updating the attribute values ​​of existing relationship edges, and solidifying the "success" or "failure" status of the execution result into relationship attributes in the graph; dynamically optimizing the retrieval weight based on the result tags, reducing the association weight or adding risk penalty items for mapping relationships marked as "failure" in subsequent searches of the same context, and increasing the association weight for mapping relationships marked as "success".

[0015] A self-evolving semantic system for UAV commands based on an execution feedback loop is disclosed to implement the aforementioned self-evolving method for UAV command semantics based on an execution feedback loop. The system includes a task planning module, a decision execution module, an effect evaluation module, a sample mining and storage module, and a model fine-tuning module. The task planning module receives natural language commands and converts them into a DSL (Directional Language Module) using a large language model. The decision execution module parses the DSL into UAV control code and drives the UAV to perform corresponding inspection tasks. The effect evaluation module evaluates the execution results of the inspection tasks and generates success or failure labels. The sample mining and storage module marks the corresponding (natural language command, DSL) mapping pairs as training samples based on the result labels. The model fine-tuning module uses the training samples to fine-tune the large language model through a contrastive learning strategy. The fine-tuned parameters of the large language model are updated in the task planning module, forming a closed-loop optimization system.

[0016] Furthermore, the system also includes a knowledge graph management module; the knowledge graph management module is used to construct, store, and dynamically update a domain knowledge graph, which is used to record structured data of historical inspection tasks; when the task planning module converts instructions, it calls the knowledge graph management module to obtain relevant context information to assist in parsing.

[0017] Furthermore, the decision execution module includes a state machine logic unit and a protocol translation unit; the state machine logic unit is used to generate a task execution logic sequence based on the input DSL command and the current UAV state; the protocol translation unit is used to translate the task execution logic sequence into underlying flight control protocol code.

[0018] As can be seen from the above technical solutions, the present invention has the following technical advantages compared with the prior art: 1. This invention constructs a feedback closed-loop mechanism of "execution-evaluation-learning," which can automatically utilize task execution results, especially failure results, as supervisory signals to drive iterative updates of large language models. It eliminates the dependence on static rule bases or fixed models, possessing the ability to continuously learn and self-correct in practical applications, effectively solving the performance degradation problem of traditional systems due to their inability to adapt to new instructions or scenarios.

[0019] 2. This invention marks instructions that fail to execute and have high model confidence, along with DSL mapping pairs, as hard negative samples, and uses contrastive learning techniques for fine-tuning. This enables the model to significantly separate erroneous mappings in the semantic space, effectively correcting their inherent cognitive biases. It can improve the model's ability to distinguish and understand ambiguous and unclear instructions with higher data efficiency and faster convergence speed.

[0020] 3. This invention uses DSL as a standardized intermediate representation, effectively decoupling the ambiguity of natural language from the precision requirements of low-level control commands. This not only significantly reduces the difficulty and risk of generating safe and compliant control commands from large language models, but also decouples the system's core semantic understanding module from specific flight control protocols, enhancing the system's compatibility and scalability across different hardware platforms and mission types.

[0021] 4. This invention employs a fusion of dynamic knowledge graphs to achieve context-based intelligent decision-making. It constructs and continuously updates a domain knowledge graph to structurally store multi-dimensional information such as the environment, instructions, and results of historical interactions. When parsing new instructions, the large language model can retrieve and utilize relevant context from the graph to accurately interpret referential and implicit instructions. Furthermore, it can proactively adjust behavioral strategies or initiate confirmation interactions based on historical experience, thereby significantly improving the system's decision-making intelligence and task execution success rate in dynamic and complex real-world environments. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating the steps of the UAV command semantic self-evolution method based on execution feedback closed loop of the present invention; Figure 2 This is a flowchart illustrating the self-evolutionary semantic system of UAV commands based on execution feedback closed loop according to the present invention. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 2 As shown, this embodiment provides a self-evolving UAV command semantic system based on an execution feedback closed loop. Its core lies in constructing an automated closed loop that includes forward command execution and feedback learning optimization. The system mainly includes the following functional modules: The input layer receives natural language instructions from the drone operator (e.g., "Check the insulators on tower 1").

[0025] The task planning module integrates a finely tuned large language model to convert unstructured natural language commands input by the drone operator into a standardized domain-specific language (DSL) with the assistance of a domain knowledge graph. For example, the above command can be converted into {Action: INSPECT, Target: TOWER_1, Component: INSULATOR}.

[0026] The finely tuned large language model is based on a general, large-scale pre-trained language model (such as GPT, Qwen, LLM, etc.), and is further trained using professional data in the field of UAV inspection (such as mission instructions, flight protocols, equipment names, operation logs, etc.) to make its output more in line with the technical specifications, terminology system and mission logic of the field.

[0027] The decision execution module, which includes a state machine logic unit and a protocol translation unit, is used to receive standardized DSL commands, combine them with the real-time status of the UAV (such as GPS coordinates, battery level, and sensor data), generate precise flight control code sequences (such as MAVLink and ROS messages), and drive the UAV to perform physical actions.

[0028] The performance evaluation module uses preset logical rules (such as whether the target area was reached, whether the captured image is clear) or direct feedback from the operator to make a binary judgment of success or failure for each task execution, and generates corresponding result labels. In specific operations, the evaluation criteria not only include logical rules, but can also be a combination of multimodal feedback such as sensor data (image clarity analysis) and operator scores.

[0029] The sample mining and storage module automatically categorizes and stores the (instruction, DSL) mapping pairs for each interaction based on the results labels from the performance evaluation module. Successful cases are marked as positive samples, while failed cases (especially those with high model confidence output but execution failure, such as confidence > 0.9) are marked as hard-to-bear samples.

[0030] The model fine-tuning module periodically utilizes accumulated positive and difficult negative samples and employs a contrastive learning strategy to incrementally fine-tune the large language model in the task planning module in order to correct erroneous semantic mapping relationships.

[0031] The knowledge graph management module (optional enhancement module) is used to structure and store historical task data (natural language instructions, DSL, environment, results), and supports dynamic updates and intelligent retrieval, providing rich contextual information for task planning.

[0032] The above modules work together through software and hardware interfaces to form a complete intelligent evolution loop from perception, decision-making, execution to evaluation, learning, and re-decision-making.

[0033] like Figure 1 The UAV command semantic self-evolution method based on execution feedback closed loop is shown below. Taking a typical substation equipment inspection scenario as an example, the specific implementation steps of the method of the present invention will be explained in detail, and its internal logic and technical effects will be described.

[0034] Step 1: System Initialization and Command Reception After the system starts, it first loads the pre-trained basic large language model, the DSL rule base specifically for substation inspection, and the existing domain knowledge graph. The operator issues natural language commands through a voice or text interface, such as: "Fly higher, take clearer pictures." This command has inherent ambiguity; what constitutes "a little higher" and what the standard for "clearer" is depends on context and historical experience for understanding.

[0035] Step 2: Context-enhanced instruction parsing Upon receiving the instruction, the task planning module does not translate it in isolation. First, it calls the knowledge graph management module to retrieve information such as the current drone location, historical inspection records regarding altitude adjustments, and local height restrictions as context. Then, the large language model integrates the current instruction with the retrieved context, performs an initial semantic mapping, and outputs a standardized DSL instruction. For example, the initial model might parse it as: DSL_1: {ACTION: ADJUST_ALTITUDE, PARAM: {DELTA: +2.0, UNIT: m}};{ACTION: SET_CAMERA, PARAM: {ZOOM: 2x}}.

[0036] This step decouples the ambiguity of natural language from the precision of control codes by introducing a DSL as an intermediate layer. This allows the model to learn only relatively regular semantic mappings, significantly reducing the difficulty and risk of directly generating complex control codes.

[0037] Step 3: Code Generation and Task Execution (Decision Execution) The decision execution module receives DSL_1. The state machine logic unit calculates the target's absolute altitude based on the ADJUST_ALTITUDE action and the current altitude. The protocol translation unit converts this logic into a series of specific MAVLink commands, such as CMD_NAV_TAKEOFF and CMD_CONDITION_CHANGE_ALT, and sends them to the UAV flight control system. The UAV then begins to execute climb and zoom maneuvers.

[0038] Step 4: Monitoring and Evaluation of Implementation Results The performance evaluation module is activated during and after the execution process. Assuming that during this execution, the drone enters the temporary height restriction zone of the substation after ascending 2 meters, triggering the obstacle avoidance system to hover urgently, the mission is judged as a failure by the system. Simultaneously, the system records that the model has a high confidence level (e.g., 0.95) for the output +2.0m when analyzing "flying a little higher".

[0039] By monitoring and evaluating the execution results, an objective and quantitative assessment of the execution outcome is achieved, transforming the abstract concept of "misunderstanding error" into a concrete and recordable "task failure" event, providing a clear signal for subsequent learning.

[0040] Step 5: Mining and Storage of Hard-Bearing Samples The sample mining and storage module marks the key-value pair of this interaction ("Fly higher, take a clearer picture", DSL_1) as a difficult negative sample based on the "failure" label. Its "difficulty" lies in the fact that the model is highly confident (with a high degree of confidence) that its parsing +2.0m is correct, but the actual result is incorrect. This sample is stored in the training sample library along with the positive samples generated from successful tasks.

[0041] Step Six: Fine-tuning the Closed-Loop Model Based on Comparative Learning During system downtime (e.g., at night), the model fine-tuning module is activated. This module extracts a batch of data containing both positive and negative samples from the sample library. For the aforementioned difficult negative samples, the contrastive learning strategy begins to take effect. Specifically, the loss function is designed as follows: For positive samples, the model is encouraged to shorten the distance between its natural language instruction embedding vector and the correct DSL embedding vector in the semantic space.

[0042] For the aforementioned difficult-to-bear samples, the distance between the semantic embedding of "flying higher" and the embedding of the erroneous DSL +2.0m is forcibly increased (in specific operations, triple loss or cosine similarity loss function can be used to achieve this), while possibly guiding it to move closer to a safer DSL (such as +0.5m) or a DSL that requests confirmation.

[0043] Optimizing model parameters through backpropagation essentially forces the model to "forget" or correct the erroneous perception that "flying a little higher" is strongly associated with "+2.0m".

[0044] Step 7: Dynamic Update of Knowledge Graph and Deployment of Model In specific operations, the dynamic updating of the knowledge graph includes: parsing the new inspection task data, automatically adding or updating entity nodes representing natural language instructions, DSL instructions, and environmental objects in the domain knowledge graph; establishing new semantic relationship edges between the natural language instruction entities, DSL entities, and environmental context entities based on the result tags, or updating the attribute values ​​of existing relationship edges, and solidifying the "success" or "failure" status of the execution result into relationship attributes in the graph; dynamically optimizing the retrieval weight based on the result tags, reducing the association weight or adding risk penalty items for mapping relationships marked as "failure" in subsequent searches of the same context, and increasing the association weight for mapping relationships marked as "success".

[0045] Specifically, the complete data of this failed task, including instructions, DSL, environment, and results, is structured and updated to the domain knowledge graph: for example, under the "Current Region" node, a record is associated with "A significant increase in instructions previously caused failure." The fine-tuned new model parameters are seamlessly updated to the online task planning module.

[0046] Step 8: Further interaction after evolution (demonstrating the effect of self-evolution) The following day, in the same area, the operator issued the instruction again: "Fly higher." The evolved system workflow is as follows: The task planning module retrieves (through Cypher query language or vector retrieval in the graph database) the knowledge graph and learns that "historical execution of the 'Fly higher' instruction in this area failed due to excessive altitude." Because the fine-tuned large language model has undergone contrastive learning, the direct mapping between "Fly higher" and "+2.0m" has been weakened.

[0047] Therefore, considering the context of past failures, the model may output a more conservative and safer DSL this time: DSL_2:{ACTION: ADJUST_ALTITUDE, PARAM: {DELTA: +0.5, UNIT: m}}; Alternatively, output a DSL that includes an acknowledgment mechanism: DSL_3:{ACTION: REQUEST_CONFIRMATION, MSG: “Historical records indicate that rising 2 meters may cause an anomaly; a rise of 0.5 meters is recommended. Execute?”}

[0048] In summary, the complete "execution-evaluation-learning" closed-loop effect of this embodiment has been verified. Through efficient correction of a single failure, autonomous evolution of semantic understanding capabilities has been achieved. This not only avoids repeating errors but also, through the utilization of knowledge graphs, makes decision-making more intelligent and contextualized, significantly improving the flexibility of human-computer interaction and the security and accuracy of task execution.

[0049] The above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A self-evolutionary method for UAV command semantics based on execution feedback closed loop, characterized in that, Includes the following steps: S1. Receive natural language instructions, use a large language model to convert the natural language instructions into a domain-specific language (DSL), and parse the DSL into control code for the UAV to perform the corresponding inspection task; S2. Evaluate the execution results of the inspection task and generate a success or failure result label; S3. Based on the result labels, mark the corresponding (natural language instructions, DSL) mapping pairs as training samples; S4. Using the labeled training samples, the large language model is fine-tuned through a contrastive learning strategy to optimize the mapping relationship between natural language instructions and DSL, and the fine-tuned model is used for subsequent instruction parsing.

2. The UAV command semantic self-evolution method based on execution feedback closed loop according to claim 1, characterized in that, The specific steps of parsing the DSL into UAV control code include: Based on the preset state machine logic, the DSL command is combined with the current state information of the UAV to generate a task execution logic sequence; The task execution logic sequence is translated into executable code of the underlying flight control protocol.

3. The UAV command semantic self-evolution method based on execution feedback closed loop according to claim 1, characterized in that, The step of labeling the corresponding (natural language instruction, DSL) mapping pairs as training samples specifically includes: If the execution result label is success, then the mapping pair is marked as a positive sample; If the execution result label is failure, the mapping pair is marked as a negative sample. The negative sample is a mapping pair in which the large language model has a confidence level higher than a preset threshold when generating the corresponding DSL but fails to execute, and is used as a difficult negative sample for fine-tuning the large language model.

4. The UAV command semantic self-evolution method based on execution feedback closed loop according to claim 1, characterized in that, The fine-tuning of the large language model through a contrastive learning strategy specifically includes: The contrastive loss function is used in the semantic embedding space to reduce the distance between the natural language instruction representation and the corresponding DSL representation in the positive samples, and to increase the distance between the natural language instruction representation and the corresponding DSL representation in the hard negative samples.

5. The UAV command semantic self-evolution method based on execution feedback closed loop according to claim 1, characterized in that, The method further includes: Construct and maintain a domain knowledge graph for structured storage of historical inspection task data, including natural language instructions, converted DSLs, task execution environment information, and execution result tags; When converting natural language instructions into a DSL, auxiliary parsing is also performed based on relevant contextual information retrieved from the domain knowledge graph.

6. The UAV command semantic self-evolution method based on execution feedback closed loop according to claim 5, characterized in that, The domain knowledge graph is dynamically updated based on the new inspection task execution results and tags: The new inspection task data is parsed, and entity nodes representing natural language commands, DSL commands, and environmental objects are automatically added or updated in the domain knowledge graph. Based on the result label, establish new semantic relationship edges between the natural language instruction entity, DSL entity and environment context entity, or update the attribute values ​​of existing relationship edges, and solidify the "success" or "failure" status of the execution result into the relationship attributes in the graph. The retrieval weight is dynamically optimized based on the result tags. For mapping relationships marked as "failed", the association weight is reduced or a risk penalty item is added in subsequent searches with the same context. For mapping relationships marked as "successful", the association weight is increased.

7. A self-evolving semantic system for UAV commands based on an execution feedback loop, used to implement the self-evolving semantic system for UAV commands based on an execution feedback loop as described in any one of claims 1-6, characterized in that, The system includes: The task planning module is used to receive natural language instructions and convert them into a DSL using a large language model. The decision execution module is used to parse the DSL into control code for the UAV and drive the UAV to perform corresponding inspection tasks. The effect evaluation module is used to evaluate the execution results of the inspection task and generate success or failure result labels. A sample mining and storage module, which is used to mark the corresponding (natural language instruction, DSL) mapping pairs as training samples based on the result labels; A model fine-tuning module is used to fine-tune the large language model using the training samples through a contrastive learning strategy. The finely tuned parameters of the large language model are updated to the task planning module, forming a closed-loop optimization system.

8. The UAV command semantic self-evolution system based on execution feedback closed loop according to claim 7, characterized in that, The system also includes: A knowledge graph management module is used to build, store, and dynamically update a domain knowledge graph, which is used to record structured data of historical inspection tasks. When converting instructions, the task planning module calls the knowledge graph management module to obtain relevant context information to assist in parsing.

9. The UAV command semantic self-evolution system based on execution feedback closed loop according to claim 8, characterized in that, The decision execution module includes: A state machine logic unit is used to generate a task execution logic sequence based on the input DSL command and the current UAV state. A protocol translation unit is used to translate the task execution logic sequence into underlying flight control protocol code.

Images