An unmanned aerial vehicle natural language control method and system based on a structured domain-specific language and time-aware target binding

Abstract

The application relates to the technical field of unmanned aerial vehicle natural language control and discloses an unmanned aerial vehicle natural language control method and system based on a structured field special language and time-aware target binding. A text or voice task instruction is acquired, an observation snapshot and an instance-level relationship scene graph are generated according to a target detection result with time information, a text task instruction, a scene graph abstract, available robot skills and format constraints are input into a large language model to generate and verify a structured field special language plan, time-aware target binding is performed when a step involving a target object is executed to obtain a target instance identifier, a small-step movement is adopted when approaching the target, a new observation snapshot is waited for, and a closed-loop execution mode of re-binding or verifying the same target instance is adopted, and when the target is lost, target detection information is expired, a target instance changes, detection confidence is insufficient, a candidate target is ambiguous or there is no continuous progress, a failure reason is generated and bounded re-planning is triggered, and the verifiability and execution stability are improved.

Description

Technical Field

[0001] This invention relates to the fields of robot natural language control, UAV visual perception, large language model planning and closed-loop execution technology, and in particular to a UAV natural language control method and system based on structured domain-specific language and time-aware target binding. Background Technology

[0002] With the development of large language models, natural language is gradually becoming an important entry point for robot task planning and interactive control. Users can express their task intentions through natural language commands such as "find target object", "approach an object", and "scan the surrounding environment". The planning model then converts the commands into executable actions for the robot, thereby lowering the barrier to entry for using robot systems.

[0003] Existing natural language robot control systems typically input visual detection results, available motor skills, and user tasks into a large language model, which then generates an executable plan. This approach improves the flexibility of task expression, but several problems still exist in real-world drone or mobile robot scenarios.

[0004] First, some systems directly generate script code or free-form plans from large language models, which are then interpreted and executed by the executor. This approach lacks strict constraints on operation sets and parameter types, making it difficult to detect illegal operations, missing parameters, or dangerous logic before execution, thus reducing the verifiability and interpretability of robot control.

[0005] Second, existing systems often use flat object detection lists to describe the environment, such as only providing the planning end with the object category name, location, and size. This representation method is difficult to distinguish multiple instances of the same type of object, and it is also difficult to express the orientation, proximity, or depth relationships between objects, resulting in unstable object binding in the system when users describe related objects such as "the chair on the left" or "the cup near the table".

[0006] Third, there is a time lag between target detection and robot movement. If the detection results before the movement are still used to make decisions after the drone rotates or moves forward, problems may occur such as the target having already left the field of view while the system continues to move, the detection box being a tracking prediction rather than a new observation, or the target instance switching.

[0007] Fourth, common target-approaching actions often employ open-loop control, which involves rotating towards the target and then advancing a fixed distance. This method lacks the ability to check for new visual feedback after movement, and cannot determine whether the target has truly been approached based on changes in target area, depth, or visibility. It also struggles to automatically recover when the target is lost, the detection expires, or there is continuous lack of progress.

[0008] Therefore, a technical solution for natural language control of UAVs is needed, which can improve the safety, verifiability and environmental adaptability of task execution by retaining the planning capabilities of large language models, through structured plan representation, instance-level scene modeling, time-aware target binding and closed-loop execution mechanism. Summary of the Invention

[0009] The technical problem to be solved by this invention is: how to constrain the natural language robot control plan generated by a large language model into a verifiable and interpretable structured plan, and to achieve stable binding, closed-loop approach and failure replanning of the natural language target by the UAV or robot platform in the case of visual perception delay, multiple instance targets and target loss.

[0010] To address the aforementioned technical problems, this invention provides a natural language control method for unmanned aerial vehicles (UAVs) based on structured domain-specific language and time-aware target binding, comprising the following steps: Step 1: Obtain the natural language task instruction input by the user. When the natural language task instruction is a voice task instruction, convert the voice task instruction into a text task instruction. Step 2: Collect environmental images through a drone or robot platform, and use the edge-side perception module deployed on the drone or robot platform, or on an edge computing device that is connected to the drone or robot platform, to perform target detection or target tracking on the environmental images to obtain target detection results with time information; Step 3: Generate an observation snapshot based on the environmental image, target detection results, and robot status; Step 4: Construct an instance-level relationship scene graph based on the observation snapshot. The instance-level relationship scene graph includes nodes representing target instances and edges representing spatial relationships between target instances. Step 5: Input the text task instructions, instance-level relational scene graph summary, available robot skills, and preset structured domain-specific language format constraints into the large language model to obtain a structured domain-specific language plan; wherein, the instance-level relational scene graph summary is a text description formed by the node attributes and edge relationships in the instance-level relational scene graph, and the available robot skills are the operations that the UAV or robot platform is pre-registered and can execute, as well as the parameter constraints corresponding to each operation; Step 6: Parse the structured domain-specific language plan and verify whether the operation type of each plan step belongs to the preset operation set, whether the operation parameters are complete, and whether the value type of the operation parameters is correct; after the verification is passed, execute the structured domain-specific language plan in the order of steps. Step 7: When performing steps involving target objects, perform time-aware target binding based on semantic target, instance-level relationship scene graph, target trajectory age, detection confidence and score difference between candidate targets, establish a binding relationship between the semantic target and a candidate instance, and use the target instance identifier of the bound candidate instance as the binding result; When performing steps to approach the target, a closed-loop execution method is adopted. After each single movement displacement does not exceed the preset single-step distance threshold, a new observation snapshot is waited for. Target instances with the same target instance identifier are re-bound or verified, and the execution progress is judged based on target visibility, target trajectory age, target image area change, or target depth change. Step 8: When a target is lost, the target detection information is outdated, the target instance changes, the detection confidence is insufficient, the candidate target is ambiguous, or there is no progress, generate a failure reason and trigger bounded reprogramming.

[0011] Furthermore, in step one, the natural language task instructions include text task instructions and voice task instructions. The voice task instructions are converted into text task instructions by the voice recognition module before being input into the planning process. The voice recognition module is used to recognize the voice content as text and does not directly generate or execute robot control instructions, so that the voice input only serves as the task entry point and does not bypass the structured domain-specific language plan verification.

[0012] Further, in step three, the observation snapshot includes an observation snapshot identifier, an image frame identifier, a millisecond-level timestamp, an image acquisition time, a perception processing completion time, image data, a target detection list, robot pose, and perception processing latency. Each target detection result includes a target category, target position, normalized center coordinates, normalized width, normalized height, target depth, detection confidence, target tracking identifier, target instance identifier, most recent detection time, and target trajectory age. The perception processing latency is the time interval between the perception processing completion time and the image acquisition time, and the target trajectory age is the time interval between the perception processing completion time and the most recent detection time.

[0013] Further, in step four, the construction of the instance-level relationship scene graph based on the observation snapshot includes: generating scene graph nodes according to the target detection results, wherein the scene graph nodes include target instance identifier, target category, target position, normalized center coordinates, normalized width, normalized height, target depth, detection confidence, image frame identifier of the corresponding observation snapshot, and target trajectory age; generating left, right, top, bottom, adjacent, or front spatial relationship edges according to the horizontal distance, vertical distance, center distance, or depth difference between any two scene graph nodes.

[0014] Furthermore, in step five, the instance-level relational scene graph summary is provided to the large language model in text form. The text form includes the node attributes of the scene graph nodes and the spatial relation edges between the scene graph nodes, which enables the large language model to generate a structured domain-specific language plan based on specific target instances and spatial relations.

[0015] Further, in step five, the structured domain-specific language plan consists of one or more plan steps arranged in execution order. Each plan step includes an operation type and operation parameters corresponding to that operation type. The operation type is limited to one operation from a preset operation set. In step six, the preset operation set includes left rotation, right rotation, scene scanning, scanning according to target description, scene exploration, target binding, facing the target, approaching the target, target visibility verification, status recording, and replanning. For each operation, the allowed parameter names, value types and value ranges of each parameter, parameters that cannot be omitted, and parameters that can be omitted due to preset values ​​are predefined. Before execution, plan steps that contain undefined operations, lack parameters that cannot be omitted, have incorrect parameter value types, have parameter values ​​exceeding the specified range, or contain unspecified parameters are rejected.

[0016] Further, in step seven, the time-aware target binding includes: parsing the target category, target instance identifier, or location cues in the semantic target; filtering candidate instances in the instance-level relationship scene graph nodes; calculating candidate scores based on the target category matching degree, detection confidence, target trajectory age, and location cues matching degree of the candidate instances; determining the candidate instance with the highest score and target trajectory age not greater than a preset maximum age threshold and detection confidence not lower than a preset confidence threshold as the target instance, and using the target instance identifier of the target instance as the binding result.

[0017] When the target trajectory age of the candidate instance with the highest score exceeds the preset maximum age threshold, output the failure reason that the target detection information has expired; when the detection confidence of the candidate instance with the highest score is lower than the preset confidence threshold, output the failure reason that the detection confidence is insufficient; when the score difference between the two candidate instances with the highest scores is less than the preset ambiguity threshold and no target instance identifier or location prompt is provided, output the failure reason that the candidate target is ambiguous.

[0018] The closed-loop execution method includes: performing time-aware target binding and recording the target instance identifier at the beginning of the step approaching the target; calculating the steering angle based on the center deviation of the target instance in the image, and performing steering and forward movement with a single forward displacement not exceeding a preset single-step distance threshold; taking an observation snapshot after the movement and waiting for the image frame identifier to be updated; re-performing time-aware target binding in the new observation snapshot; and determining that progress has been made when the re-obtained target instance identifier is consistent with the recorded target instance identifier and the target image area increases or the target depth decreases.

[0019] When the target instance is temporarily not visible, target reacquisition is performed based on the target instance identifier and estimated target orientation recorded in the execution memory. The target reacquisition includes rotating according to the estimated target orientation or a preset rotation angle, waiting for a new observation snapshot, and re-executing time-aware target binding. The number of times the target reacquisition is repeated does not exceed a preset reacquisition limit. If a target instance with the same target instance identifier is not reacquisitioned after reaching the preset reacquisition limit, bounded replanning is triggered.

[0020] Furthermore, in step eight, the bounded replanning includes: when the execution result of the structured domain-specific language plan contains a failure state or meets the preset replanning trigger condition, collecting the original text task instructions, the executed plan steps, the failure reason, the failure situation that triggers replanning, and the latest instance-level relational scene graph summary, generating a replanning context and re-inputting it into the large language model to obtain a new structured domain-specific language plan; wherein, the number of replanning is constrained by the preset upper limit of the number of replanning.

[0021] This invention also provides a natural language control system for unmanned aerial vehicles (UAVs) based on structured domain-specific language and time-aware target binding, including a natural language task instruction input module, an edge-side perception module, an observation snapshot generation module, an instance-level relational scene graph construction module, a large language model planning module, a domain-specific language parsing and verification module, a time-aware target binding module, a closed-loop execution module, and a bounded replanning module; each module is configured to execute the above methods.

[0022] Furthermore, the closed-loop execution module also includes a collaborative cancellation unit, which is used to set a cancellation flag when a stop request is received, so that the structured domain-specific language plan execution loop, scanning process or closed loop stops at the step boundary of the target process, and outputs the failure reason or cancellation status of the user cancellation.

[0023] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described natural language control method for unmanned aerial vehicles.

[0024] Compared with the prior art, the present invention has the following beneficial effects: (1) By replacing the free code plan with a structured domain-specific language, the output of the large language model can be parsed, verified and rejected before execution, reducing the risk of arbitrary code execution and uncontrollable actions.

[0025] (2) By observing the snapshot record image frame identifier, timestamp, target trajectory age and perception processing delay, the execution end can distinguish between fresh detection information and expired detection information.

[0026] (3) Provide target instances and spatial relationships to the planning end through instance-level relationship scene diagrams to improve the planning quality under multi-target scenarios and relationship target descriptions.

[0027] (4) By using the time-aware target binding mechanism, semantic targets are bound to specific target instance identifiers, and the reasons for failure are clearly defined for expired target detection information, insufficient detection confidence, or ambiguous output of candidate targets.

[0028] (5) By performing closed-loop approach to the target and verifying new observation snapshots after movement, the UAV or robot platform is prevented from continuing to move based on outdated perception results, thus improving the execution stability in real-world scenarios.

[0029] (6) Through bounded replanning driven by failure reasons, the system can generate new structured domain-specific language plans when the target is lost, the target instance changes, or there is no continuous progress, thus avoiding the repeated execution of actions that have already failed.

[0030] (7) Through the collaborative cancellation mechanism, users can stop the scanning process or close the loop approaching the target during task execution, thereby improving the safety of drone or robot platform operation. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the overall system architecture in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of generating a structured domain-specific language plan from a natural language task in an embodiment of the present invention. Figure 3 This is a schematic diagram of the observation snapshot and relationship scene graph construction process in an embodiment of the present invention; Figure 4 This is a schematic diagram of the time-aware target binding process in an embodiment of the present invention; Figure 5 This is a schematic diagram of the closed-loop target approach execution and failure replanning process in an embodiment of the present invention. Detailed Implementation

[0032] The technical solution of the present invention will be further described below with reference to specific embodiments and accompanying drawings. It should be understood that the following embodiments are used to illustrate the present invention and are not intended to limit the scope of protection of the present invention; in the absence of conflict, the technical features of the various embodiments can be combined with each other.

[0033] Implementation method 1: System and method implementation based on the technical solution described in the claims.

[0034] like Figure 1 As shown, the UAV natural language control system of this embodiment includes a natural language task command input module, an edge-side perception module, an observation snapshot generation module, an instance-level relational scene graph construction module, a large language model planning module, a domain-specific language parsing and verification module, a time-aware target binding module, a closed-loop execution module, and a bounded replanning module. These modules can be located on the same computing device, or they can be distributed across the UAV or robot platform, an edge computing device communicating with the platform, and local or remote computing devices.

[0035] This embodiment provides a natural language control method for unmanned aerial vehicles (UAVs) based on structured domain-specific language and time-aware target binding, including the following steps: Step 1: Obtain the natural language task instruction input by the user. When the natural language task instruction is a voice task instruction, convert the voice task instruction into a text task instruction. Step 2: Collect environmental images through a drone or robot platform, and use the edge-side perception module deployed on the drone or robot platform, or on an edge computing device that is connected to the drone or robot platform, to perform target detection or target tracking on the environmental images to obtain target detection results with time information; Step 3: Generate an observation snapshot based on the environmental image, target detection results, and robot status; Step 4: Construct an instance-level relationship scene graph based on the observation snapshot. The instance-level relationship scene graph includes nodes representing target instances and edges representing spatial relationships between target instances. Step 5: Input the text task instructions, instance-level relational scene graph summary, available robot skills, and preset structured domain-specific language format constraints into the large language model to obtain a structured domain-specific language plan; wherein, the instance-level relational scene graph summary is a text description formed by the node attributes and edge relationships in the instance-level relational scene graph, and the available robot skills are the operations that the UAV or robot platform is pre-registered and can execute, as well as the parameter constraints corresponding to each operation; Step 6: Parse the structured domain-specific language plan and verify whether the operation type of each plan step belongs to the preset operation set, whether the operation parameters are complete, and whether the value type of the operation parameters is correct; after the verification is passed, execute the structured domain-specific language plan in the order of steps. Step 7: When performing steps involving target objects, perform time-aware target binding based on semantic target, instance-level relationship scene graph, target trajectory age, detection confidence and score difference between candidate targets, establish a binding relationship between the semantic target and a candidate instance, and use the target instance identifier of the bound candidate instance as the binding result; When performing steps to approach the target, a closed-loop execution method is adopted. After each single movement displacement does not exceed the preset single-step distance threshold, a new observation snapshot is waited for. Target instances with the same target instance identifier are re-bound or verified, and the execution progress is judged based on target visibility, target trajectory age, target image area change, or target depth change. Step 8: When a target is lost, the target detection information is outdated, the target instance changes, the detection confidence is insufficient, the candidate target is ambiguous, or there is no progress, generate a failure reason and trigger bounded reprogramming.

[0036] In step one, the natural language task instructions include text task instructions and voice task instructions. The voice recognition module converts the voice content into text task instructions before inputting them into the planning process, without directly generating or executing robot control instructions. This ensures that voice input serves only as a task entry point and does not bypass the structured domain-specific language plan verification.

[0037] In step two, the edge-side perception module can be directly deployed on a drone or robot platform, or it can be deployed on an edge computing device that is communicatively connected to the platform. The edge-side perception module receives continuous environmental images, performs target detection or target tracking on each environmental image, and associates the target detection results with image frame identifiers and time information for use in subsequent observation snapshot generation and target trajectory age calculation.

[0038] In step three, the observation snapshot includes an observation snapshot identifier, an image frame identifier, a millisecond-level timestamp, the image acquisition time, the perception processing completion time, image data, a target detection list, robot pose, and perception processing latency. Each target detection result includes the target category, target position, normalized center coordinates, normalized width, normalized height, target depth, detection confidence, target tracking identifier, target instance identifier, most recent detection time, and target trajectory age. The perception processing latency is the time interval between the perception processing completion time and the image acquisition time, and the target trajectory age is the time interval between the perception processing completion time and the most recent detection time.

[0039] When generating observation snapshots, the observation snapshot generation module encapsulates the environmental image, target detection result, and robot state corresponding to the same perception processing into a non-interchangeable data set at the same moment. The image frame identifier is used to determine whether a new environmental image has been obtained after the motion, the target trajectory age is used to determine whether the target detection information still meets the execution requirements, and the perception processing latency is used to record the time elapsed from image acquisition to the completion of perception processing.

[0040] like Figure 3 As shown, in step four, scene graph nodes are generated based on the target detection results. The scene graph nodes include target instance identifier, target category, target location, normalized center coordinates, normalized width, normalized height, target depth, detection confidence, image frame identifier of the corresponding observation snapshot, and target trajectory age. Spatial relationship edges such as left, right, top, bottom, adjacent, or front are generated based on the horizontal distance, vertical distance, center distance, or depth difference between any two scene graph nodes.

[0041] When constructing an instance-level relationship scene graph, each scene graph node corresponds to a target instance, rather than just a target category. Therefore, when multiple targets of the same category exist in the same environment image, each target is distinguished by a different target instance identifier. Spatial relationship edges are generated by the relative positions between two target instances and, together with their corresponding scene graph nodes, constitute the instance-level relationship scene graph.

[0042] like Figure 2 As shown, in step five, the instance-level relational scene graph summary is provided to the large language model in text form. This text form includes the node attributes of the scene graph nodes and the spatial relation edges between the scene graph nodes, which enables the large language model to generate a structured domain-specific language plan based on specific target instances and spatial relations. The structured domain-specific language plan consists of one or more plan steps arranged in execution order, and each plan step includes an operation type and operation parameters corresponding to that operation type.

[0043] In step five, available robot skills are pre-registered by the drone or robot platform. Each available robot skill corresponds to at least one executable operation, specifying the allowed parameter names, parameter value types, and parameter value ranges for that operation. The large language model planning module generates planning steps only within the constraints of available robot skills and structured domain-specific language format.

[0044] In step six, the preset operation set includes left rotation, right rotation, scene scanning, scanning according to target description, scene exploration, target binding, facing the target, approaching the target, target visibility verification, status recording, and replanning. For each operation, the allowed parameter names, value types and ranges of each parameter, parameters that cannot be omitted, and parameters that can be omitted due to preset values ​​are predefined. The domain-specific language parsing and verification module rejects planned steps before execution that contain undefined operations, lack parameters that cannot be omitted, have incorrect parameter value types, have parameter values ​​outside the specified range, or contain unspecified parameters.

[0045] like Figure 4 As shown, in step seven, time-aware target binding includes: parsing the target category, target instance identifier, or location cues in the semantic target; filtering candidate instances in the instance-level relationship scene graph nodes; calculating candidate scores based on the target category matching degree, detection confidence, target trajectory age, and location cues matching degree of the candidate instances; determining the candidate instance with the highest score and target trajectory age not greater than a preset maximum age threshold and detection confidence not lower than a preset confidence threshold as the target instance, and using the target instance identifier of the target instance as the binding result.

[0046] When the target trajectory age of the candidate instance with the highest score exceeds the preset maximum age threshold, the time-aware target binding module outputs the failure reason that the target detection information has expired; when the detection confidence of the candidate instance with the highest score is lower than the preset confidence threshold, it outputs the failure reason that the detection confidence is insufficient; when the score difference between the two candidate instances with the highest scores is less than the preset ambiguity threshold and no target instance identifier or location prompt is provided, it outputs the failure reason that the candidate target is ambiguous.

[0047] like Figure 5 As shown, the closed-loop execution method includes: performing time-aware target binding and recording the target instance identifier at the beginning of the step approaching the target; calculating the steering angle based on the center deviation of the target instance in the image, and performing steering and forward movement with a single forward displacement not exceeding a preset single-step distance threshold; taking an observation snapshot after the movement and waiting for the image frame identifier to be updated; re-performing time-aware target binding in the new observation snapshot; and determining that progress has been made when the re-obtained target instance identifier is consistent with the recorded target instance identifier and the target image area increases or the target depth decreases.

[0048] The closed-loop execution module records the image frame identifier of the current observation snapshot before each movement and waits for the image frame identifier to be updated after the movement ends. If no new observation snapshot is obtained within the preset waiting time limit, the target detection results before the movement are not used to continue, and instead, the failure reason of expired target detection information is output. If a new observation snapshot is obtained, the instance-level relationship scene graph is reconstructed using the observation snapshot and time-aware target binding is re-executed.

[0049] When a target instance is temporarily unavailable, target reacquisition is performed based on the target instance identifier and estimated target orientation recorded in the execution memory. Target reacquisition includes rotating the target according to the estimated target orientation or a preset rotation angle, waiting for a new observation snapshot, and re-executing time-aware target binding. The number of times target reacquisition is repeated does not exceed a preset reacquisition limit, and bounded replanning is triggered if a target instance with the same target instance identifier is not reacquisitioned after reaching the limit.

[0050] In step eight, bounded replanning includes: when the execution result of the structured domain-specific language plan contains a failure state or meets the preset replanning trigger conditions, the original text task instructions, executed plan steps, failure reasons, failure scenarios that trigger replanning, and the latest instance-level relational scene graph summary are collected. A replanning context is generated and re-input into the large language model to obtain a new structured domain-specific language plan. The number of replanning iterations is constrained by a preset upper limit. The new plan must still be verified by the domain-specific language parsing and verification module before it can be executed.

[0051] The closed-loop execution module may further include a collaborative cancellation unit. Upon receiving a stop request, the collaborative cancellation unit sets a cancellation flag, causing the Structured Domain Language (SDL) execution loop, scanning process, or closed loop to stop near the target process step boundary, and outputs the user's cancellation failure reason or cancellation status to prevent the continuation of subsequent actions after a cancellation request arrives.

[0052] The above method can also be implemented by a computer program stored in a computer-readable storage medium. When the processor executes the computer program, it controls each module to sequentially complete the following tasks: natural language task instruction acquisition, environmental image perception, observation snapshot generation, instance-level relational scene graph construction, structured domain-specific language plan generation and verification, time-aware target binding, closed-loop execution, and bounded replanning.

[0053] Implementation Method 2: Experimental Procedure.

[0054] This implementation is used to verify the complete execution process of structured domain-specific language (DNS) plan verification, instance-level relational scene graph construction, time-aware target binding, closed-loop approach to target, target re-acquisition, and bounded replanning. The experiment can be conducted on a virtual robot platform, a physical drone platform, or a physical mobile robot platform; the platform used is equipped with an environmental image acquisition device and is communicatively connected to a computing device capable of running the edge-side perception module and the aforementioned control module.

[0055] Before the experiment began, the installation position and field of view of the environmental image acquisition device were calibrated to convert the target position in the image into normalized center coordinates, normalized width, and normalized height; the time of the UAV or robot platform and the edge computing device were synchronized; operations such as left rotation, right rotation, scene scanning, scanning according to target description, scene exploration, target binding, facing the target, approaching the target, target visibility verification, status recording, and replanning were registered, and the allowed parameter names, value types, and value ranges were set for each operation.

[0056] To ensure reproducibility, preset maximum age threshold, preset reliability threshold, preset ambiguity threshold, preset single-step distance threshold, preset maximum number of reacquisitions, and preset maximum number of replannings were fixed within the same experimental group. The platform status, perception module version, planning model version, available robot skills set, and threshold configurations were recorded. Different threshold configurations were recorded separately for different experimental groups and were not mixed within the same experimental group.

[0057] The experimental scenario includes at least one target object, and multiple targets of the same category, reference objects that are to the left, right, above, below, adjacent to, or in front of the target, and obstacles that can temporarily obscure the target, depending on the function to be verified. Natural language tasks include at least single-target tasks, multiple similar-target tasks, target tasks with location cues, target tasks with spatial relationship descriptions, and tasks for recovering a target after it has been temporarily obscured.

[0058] The experiment was conducted as follows: First, text or voice task instructions were input. For voice task instructions, the speech recognition module converted them into text task instructions, and it was verified that the robot action was not triggered directly during the conversion stage. Subsequently, environmental images were collected by a drone or robot platform. The edge perception module output target detection results with time information, the observation snapshot generation module generated observation snapshots, and the instance-level relationship scene graph construction module generated scene graph nodes, spatial relationship edges, and an instance-level relationship scene graph summary.

[0059] The text task instructions, instance-level relational scene graph summary, available robot skills, and structured domain-specific language format constraints are input into the large language model planning module, which records the generated structured domain-specific language plan. The domain-specific language parsing and verification module sequentially checks the operation type, operation parameter completeness, parameter value type, parameter value range, and additional parameters. For manually constructed undefined operations, missing parameters, error types, out-of-bounds parameters, or unspecified parameters, the verification system rejects the corresponding plan step before the action is executed and outputs a verification failure message.

[0060] For the validated plan steps involving target objects, record the target category, detection confidence, target trajectory age, location cue matching degree, and candidate score of the candidate instance. Verify whether the target instance identifier of the binding result is consistent with the target instance pre-labeled in the experiment. When the scores of multiple targets of the same category are close, provide and do not provide target instance identifiers or location cuees respectively to verify whether the system can complete the binding when the information is sufficient, and output the reason for the failure of candidate target ambiguity when the information is insufficient.

[0061] For each planned step approaching the target, record the observation snapshot identifier, image frame identifier, target instance identifier, target image area, and target depth before the movement. After performing a turning motion and a forward movement not exceeding a preset single-step distance threshold, wait for a new observation snapshot and record the above information again. Only when the target instance identifier is consistent and the target image area increases or the target depth decreases is the movement recorded as progress made. The approaching-target process ends when the target reaches a preset stopping condition or the planned step reaches a preset termination condition.

[0062] In the target reacquisition experiment, the target is temporarily removed from the field of view during the closed-loop approach process through occlusion, viewpoint changes, or platform movement. Based on the target instance identifier and estimated target location stored in the execution memory, the system performs rotation, waits for a new observation snapshot, and rebinds within a preset maximum number of reacquisition attempts. If the same target instance identifier is reacquired, the reacquisition is recorded as successful, and the closed-loop execution continues; if reacquisition fails, the reason for failure—target loss or expired target detection information—is recorded, and the system enters a bounded replanning process.

[0063] In the failed replanning experiment, scenarios were constructed including target loss, expired target detection information, target instance change, insufficient detection confidence, ambiguity of candidate targets, and continuous lack of progress. The system records the original text task instructions, executed plan steps, failure reasons, failure scenarios that trigger replanning, and the latest instance-level relationship scene graph summary, generates a replanning context, and requests a new structured domain-specific language plan; it verifies that the number of replanning attempts does not exceed the preset replanning attempt limit, and that each new plan is parsed and verified again before execution.

[0064] In the collaborative cancellation experiment, stop requests were sent during the scene scanning process, the structured domain-specific language plan execution loop, and the closed-loop approach to the target. The setting time of the cancellation flag, the end time of the current step, and the status of subsequent actions were recorded. The verification system stopped at the step boundary and output the failure reason or cancellation status of the user's cancellation.

[0065] During the experiment, at least the following should be recorded: whether the task was completed, whether the plan verification passed, whether the target instance binding was correct, whether the target instance identifier was consistent before and after the movement, the number of times the target detection information expired and was rejected, the detection status of ambiguity in candidate targets, the number of closed-loop movements, the number of times the target was reacquired, the number of times the bounded replanning was performed, the task execution time, and the cancellation response status. Each type of task was repeatedly executed under the same configuration, and different experimental samples were formed by adjusting the target position, the number of similar targets, the occlusion timing, and the perception delay.

[0066] The experimental results were statistically analyzed according to pre-set judgment rules: A successful task was defined as one where the structured domain-specific language plan passed all format and parameter checks before action execution, and the bound target matched the pre-marked target upon task completion; a valid closed-loop progress was defined as one where the system completed post-motion verification based on new observation snapshots and continued execution even when the target image area increased or the target depth decreased; a valid anomaly identification was defined as one where the system rejected outdated target detection information, insufficient detection confidence, or ambiguous candidate targets, and provided corresponding reasons for failure; and a valid recovery or replanning was defined as one where the system completed target reacquisition or generated and verified a new structured domain-specific language plan within the maximum number of attempts.

[0067] Through the above experimental process, the inputs, outputs, execution status, and failure reasons of each technical link of the present invention can be verified step by step. The specific platform model, perception algorithm, planning model, and threshold can be selected according to the application scenario. As long as its execution process conforms to the above observation snapshot, instance-level relationship scenario graph, structured domain-specific language plan verification, time-aware target binding, closed-loop execution, and bounded replanning relationship, it can be used to implement the present invention.

Claims

1. A natural language control method for unmanned aerial vehicles (UAVs) based on structured domain-specific language and time-aware target binding, characterized in that, Includes the following steps: Step 1: Obtain the natural language task instruction input by the user. When the natural language task instruction is a voice task instruction, convert the voice task instruction into a text task instruction. Step 2: Collect environmental images through a drone or robot platform, and use the edge-side perception module deployed on the drone or robot platform, or on an edge computing device that is connected to the drone or robot platform, to perform target detection or target tracking on the environmental images to obtain target detection results with time information; Step 3: Generate an observation snapshot based on the environmental image, target detection results, and robot status. Step 4: Construct an instance-level relationship scene graph based on the observation snapshot. The instance-level relationship scene graph includes nodes representing target instances and edges representing spatial relationships between target instances. Step 5: Input the text task instructions, instance-level relational scene graph summary, available robot skills, and preset DSL format constraints into the large language model to obtain the DSL plan; wherein, the instance-level relational scene graph summary is a text description formed by the node attributes and edge relationships in the instance-level relational scene graph, and the available robot skills are the operations that the UAV or robot platform is pre-registered and can perform, as well as the parameter constraints corresponding to each operation; Step 6: Parse the DSL plan and verify whether the operation type of each plan step belongs to the preset operation set, whether the operation parameters are complete, and whether the value type of the operation parameters is correct; after the verification is passed, execute the DSL plan in the order of steps. Step 7: When performing steps involving target objects, perform time-aware target binding based on semantic target, instance-level relationship scene graph, target trajectory age, detection confidence and score difference between candidate targets, establish a binding relationship between the semantic target and a candidate instance, and use the target instance identifier of the bound candidate instance as the binding result; When performing steps to approach the target, a closed-loop execution method is adopted. After each single movement displacement does not exceed the preset single-step distance threshold, a new observation snapshot is waited for. Target instances with the same target instance identifier are re-bound or verified, and the execution progress is judged based on target visibility, target trajectory age, target image area change, or target depth change. Step 8: When a target is lost, the target detection information is outdated, the target instance changes, the detection confidence is insufficient, the candidate target is ambiguous, or there is no progress, generate a failure reason and trigger bounded reprogramming.

2. The UAV natural language control method according to claim 1, characterized in that, In step one, the natural language task instructions include text task instructions and voice task instructions. The voice task instructions are converted into text task instructions by the voice recognition module before being input into the planning process. The voice recognition module is used to recognize the voice content as text and does not directly generate or execute robot control instructions, so that the voice input only serves as the task entry point and does not bypass the DSL plan verification. In step three, the observation snapshot includes an observation snapshot identifier, an image frame identifier, a millisecond-level timestamp, an image acquisition time, a perception processing completion time, image data, a target detection list, robot pose, and a perception processing delay. Each target detection list includes a target category, target position, normalized center coordinates, normalized width, normalized height, target depth, detection confidence, target tracking identifier, target instance identifier, most recent detection time, and target trajectory age. The perception processing delay is the time interval between the perception processing completion time and the image acquisition time, and the target trajectory age is the time interval between the perception processing completion time and the most recent detection time.

3. The natural language control method for unmanned aerial vehicles according to claim 1, characterized in that, In step four, the construction of instance-level relationship scene graph based on observation snapshots includes: generating scene graph nodes based on target detection results, wherein the scene graph nodes include target instance identifier, target category, target location, normalized center coordinates, normalized width, normalized height, target depth, detection confidence, and target trajectory age; and generating left, right, top, bottom, adjacent, or front spatial relationship edges based on the horizontal distance, vertical distance, center distance, or depth difference between any two scene graph nodes.

4. The UAV natural language control method according to claim 1, characterized in that, In step five, the instance-level relational scene graph summary is provided to the large language model in text form. The text form contains scene graph nodes and spatial relation edges between scene graph nodes, which is used to enable the large language model to generate a DSL plan based on specific target instances and spatial relations. The DSL plan consists of one or more plan steps arranged in execution order. Each plan step includes an operation type and operation parameters corresponding to that operation type. The operation type is limited to an operation in a preset set of operations.

5. The natural language control method for unmanned aerial vehicles according to claim 1, characterized in that, In step six, the preset operation set includes left rotation, right rotation, scene scanning, scanning according to target description, scene exploration, target binding, facing the target, approaching the target, target visibility verification, status recording, and replanning; For each operation, the allowed parameter names, value types and ranges of each parameter, parameters that cannot be omitted, and parameters that can be omitted due to preset values ​​are specified in advance. Before execution, planned steps that contain undefined operations, missing parameters that cannot be omitted, incorrect parameter value types, parameter values ​​that exceed the specified range, or contain unspecified parameters are rejected.

6. The natural language control method for unmanned aerial vehicles according to claim 1, characterized in that, In step seven, the time-aware target binding includes: parsing the target category, target instance identifier, or location cues in the semantic target; filtering candidate instances in the instance-level relationship scene graph nodes; calculating candidate scores based on the target category matching degree, detection confidence, target trajectory age, and location cues matching degree of the candidate instances; determining the candidate instance with the highest score and target trajectory age not greater than a preset maximum age threshold and detection confidence not lower than a preset confidence threshold as the target instance, and using the target instance identifier of the target instance as the binding result; When the target trajectory age of the candidate instance with the highest score exceeds the preset maximum age threshold, output the failure reason that the target detection information has expired; when the detection confidence of the candidate instance with the highest score is lower than the preset confidence threshold, output the failure reason that the detection confidence is insufficient; when the score difference between the two candidate instances with the highest scores is less than the preset ambiguity threshold and no target instance identifier or location hint is provided, output the failure reason that the candidate target is ambiguous. The closed-loop execution method includes: performing time-aware target binding and recording the target instance identifier at the beginning of the step approaching the target; calculating the steering angle based on the center deviation of the target instance in the image, and performing a steering and forward movement with a single forward displacement not exceeding a preset single-step distance threshold; taking an observation snapshot after the movement and waiting for the image frame identifier to be updated; re-performing time-aware target binding in the new observation snapshot; and determining that progress has been made when the re-obtained target instance identifier is consistent with the recorded target instance identifier and the target image area increases or the target depth decreases. When the target instance is temporarily not visible, target reacquisition is performed based on the target instance identifier and estimated target orientation recorded in the execution memory. The target reacquisition includes rotating according to the estimated target orientation or a preset rotation angle, waiting for a new observation snapshot, and re-executing time-aware target binding. The number of times the target reacquisition is repeated does not exceed a preset reacquisition limit. If a target instance with the same target instance identifier is not reacquisitioned after reaching the preset reacquisition limit, bounded replanning is triggered.

7. The natural language control method for unmanned aerial vehicles according to claim 1, characterized in that, In step eight, the bounded replanning includes: when the execution result of the DSL plan contains a failure state or meets the preset replanning triggering condition, collecting the original text task instructions, the executed plan steps, the failure reason, the failure situation that triggers replanning, and the latest instance-level relationship scene graph summary, generating a replanning context and re-inputting it into the large language model to obtain a new DSL plan; wherein, the number of replanning is constrained by the preset upper limit of the number of replanning.

8. A natural language control system for unmanned aerial vehicles (UAVs) based on DSL and time-aware target binding, characterized in that, The method includes a natural language task instruction input module, an edge-side perception module, an observation snapshot generation module, an instance-level relational scene graph construction module, a large language model planning module, a domain-specific language parsing and verification module, a time-aware target binding and closed-loop execution module, and a bounded replanning module. Each module is configured to perform steps one to eight of the UAV natural language control method based on structured domain-specific language and time-aware target binding as described in any one of claims 1 to 7.

9. The UAV natural language control system according to claim 8, characterized in that, The closed-loop execution module also includes a cooperative cancellation unit, which is used to set a cancellation flag when a stop request is received, so that the DSL plan execution loop, scanning process or closed loop stops at the step boundary of the target process, and outputs the failure reason or cancellation status of the user cancellation.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the natural language control method for unmanned aerial vehicles (UAVs) based on structured domain-specific language and time-aware target binding as described in any one of claims 1 to 7.

Images