A safety-constrained feedback planning system and method for unmanned aerial vehicle multi-modal tasks

Abstract

The present application belongs to the technical field of unmanned aerial vehicle multi-modal task planning and flight safety control, and particularly relates to a safety constraint feedback planning system and method for unmanned aerial vehicle multi-modal tasks, which comprises a multi-modal acquisition module for acquiring data and instructions; a time-space synchronization and coordinate registration module for generating time-space registration data packets; a multi-modal task reasoning module for generating candidate actions; a candidate flight path generation module for converting candidate actions into candidate flight paths and establishing a corresponding relationship between action step numbers and flight path segment numbers; a segment risk assessment module for outputting risk attribution results; a safety constraint verification module for verification and output of safety verification results; a feedback constraint module for generating re-planning constraints and sending them to the multi-modal task reasoning module; and a flight control execution module for receiving candidate flight paths that pass the verification. The system can output structured rejection reasons, which can prompt the upstream link to make targeted corrections.

Description

Technical Field

[0001] This application belongs to the field of UAV multimodal mission planning and flight safety control technology, specifically involving a safety constraint feedback planning system and method for UAV multimodal missions. Background Technology

[0002] Existing UAV mission planning methods based on language models or multimodal models can generate mission actions or action plans based on natural language commands, images, maps, and UAV status. However, these methods typically remain at the level of mission intent parsing and action sequence generation, with outputs mostly consisting of mission-level actions such as approaching the target, low-altitude observation, circling and filming, or continuing the search. They lack pre-execution verification conditions that are tied to the current flight path, energy reserves, obstacle distribution, airspace boundaries, and communication link status, making them difficult to directly use as the basis for flight control execution.

[0003] Existing UAV flight control systems, path planning systems, and multi-sensor fusion systems already possess functions such as time synchronization, coordinate registration, collision detection, low-battery return-to-home, no-fly zone restrictions, and trajectory risk analysis. However, these functions mostly operate as independent protection mechanisms, and safety verification results are typically expressed as pass, failure, or alarm, without compiling the reasons for failure, risk locations, unexecutable parameters, and alternative actions into structured feedback that can be read by the mission reasoning process. Therefore, while safety protection can block dangerous flight paths, it is difficult to drive targeted corrections in the upstream planning process.

[0004] In low-altitude patrol, road emergency inspection, and target search missions, risks are often not evenly distributed throughout the entire mission, but rather concentrated in a specific sub-step or segment of the candidate action. For example, the approach segment in a close-range observation mission might be too close to the safety boundary of power lines; the observation segment in a low-altitude observation mission might enter restricted airspace above areas with high population density; and the return segment in a search mission might fail to meet return margin requirements due to insufficient remaining battery power. If the system only outputs the overall risk level, the multimodal mission inference module will find it difficult to determine the action parameters that need adjustment, the risk areas to be avoided, and the corresponding alternative safe actions.

[0005] In summary, the existing technology mainly suffers from the following technical problems:

[0006] (1) The output results are mostly mission-level actions such as approaching the target, low-altitude observation, circling and shooting, or continuing to search. They lack pre-execution verification conditions that are bound to the current flight path, energy reserves, obstacle distribution, airspace boundary and communication link status, making it difficult to directly use as the basis for flight control execution.

[0007] (2) When the link is abnormal or the security time limit is insufficient, the cloud results directly affect the flight control execution, which is a high risk.

[0008] (3) The safety verification results are usually expressed as pass, failure or alarm. The reasons for failure, risk location, non-executable parameters and alternative actions are not organized into a structured feedback that can be read by the task reasoning stage, making it difficult to drive the upstream planning stage to make targeted corrections.

[0009] Therefore, a new method is urgently needed, which first converts the candidate actions generated by the multimodal task reasoning module into verifiable candidate tracks, then performs risk attribution on a per-action-step and track-segment basis, and converts rejection reasons into replanning constraints, thereby using the safety verification results to inversely constrain the generation of subsequent candidate actions. Summary of the Invention

[0010] To address the aforementioned technical problems in the existing technology, the purpose of this invention is to establish a closed-loop data path between multimodal mission reasoning and flight control execution, consisting of candidate actions, candidate trajectories, segmented risk attribution, structured rejection reasons, and replanning constraints, thereby driving targeted corrections in the upstream planning stage. The technical solution is as follows:

[0011] A safety constraint feedback planning system for multimodal UAV missions includes the following modules:

[0012] Multimodal acquisition module: Acquires images, videos, point clouds, maps, telemetry data, and natural language commands;

[0013] Spatiotemporal synchronization and coordinate registration module: Generates a spatiotemporal registration data package based on the images, videos, point clouds, maps, telemetry data, and natural language commands acquired by the multimodal acquisition module;

[0014] Multimodal task reasoning module: Generates candidate actions based on the spatiotemporal registration data packet generated by the spatiotemporal synchronization and coordinate registration module;

[0015] Candidate track generation module: Converts the candidate actions generated by the multimodal task reasoning module into candidate tracks with action step numbers and track segment numbers, and establishes the correspondence between action step numbers and track segment numbers;

[0016] Segmented Risk Assessment Module: Outputs risk attribution results for each segment of the trajectory based on the correspondence established by the candidate trajectory generation module;

[0017] Safety constraint verification module: Based on candidate tracks, risk attribution results, and UAV telemetry status, perform reachability, collision, boundary, energy, and confidence verification, and output safety verification results;

[0018] Feedback constraint module: Based on the full verification results output by the security constraint verification module, it generates replanning constraints according to different cases and sends them to the multimodal task inference module;

[0019] Flight control execution module: Receives candidate tracks verified by the safety constraint verification module.

[0020] Furthermore, the spatiotemporal synchronization and coordinate registration module converts image targets, point cloud obstacles, map regions, and natural language entities into the same low-altitude coordinate system; the spatiotemporal registration data package includes: timestamp, UAV pose, current position, speed, remaining battery power, range capability, power parameters, link status, sensor type, data quality indicators, geographic coordinates, mission context identifier, and original data reference.

[0021] Furthermore, the multimodal task reasoning module queries historical rejection reasons, historical replanning constraints, and operator preference records according to the task context identifier. The multimodal task reasoning module includes the following units:

[0022] (1) Visual encoding unit: outputs target category, target bounding box, region features and target confidence;

[0023] (2) Map coding unit: Outputs roads, routes, buildings, no-fly zones and geographic reference points;

[0024] (3) Telemetry coding unit: outputs current location, speed, remaining battery power and link quality;

[0025] (4) Language parsing unit: Outputs task entities and action intentions;

[0026] (5) Cross-modal alignment unit: Determine the matching relationship between language entities and spatial targets based on visual feature similarity, map spatial proximity, temporal consistency and task context consistency;

[0027] (6) Action generation unit: Outputs candidate actions;

[0028] The action generation condition of the multimodal task reasoning module is that all parameters of the candidate action are aligned across modalities and the alignment confidence is not lower than the preset alignment threshold.

[0029] Furthermore, the cross-modal alignment unit determines the matching relationship according to the following priority order:

[0030] First priority: Spatial consistency rule. If the target entity T in the natural language instruction contains an absolute location description, it will be matched with the location in the spatial location M output by the map coding unit whose geographic coordinate deviation does not exceed a preset spatial threshold.

[0031] Second priority: visual feature rules. If T contains visual attribute descriptions, it will be prioritized to match the target in the candidate target set V output by the visual encoding unit with a confidence level not lower than the preset visual threshold and matching attributes.

[0032] Third priority: Task context rules. If T references the output of a previous action, then search for the same target reference in the completion criteria of the previous action according to the task context identifier. The matching condition is that the task context identifier is consistent and the spatial coordinate deviation does not exceed the preset context matching threshold.

[0033] Fourth priority: Temporal consistency rule. If T is mentioned in multiple consecutive frames, the candidate target with the highest trajectory smoothness is selected first.

[0034] In the action generation unit, the candidate action includes: skill name, spatial target, action parameters, termination condition, action steps, action parameter confidence level, and safety constraint fields.

[0035] Furthermore, in the candidate trajectory generation module, the candidate trajectory includes waypoint coordinates, estimated arrival time, flight altitude, target speed, sensor attitude, backup waypoints, action step number, and trajectory segment number;

[0036] The candidate trajectory generation module establishes a correspondence between action step numbers and trajectory segment numbers by using the skill names and action steps in the candidate actions.

[0037] Furthermore, the risk attribution results output by the segmented risk assessment module include the track segment number, risk type, risk level, trigger layer, trigger data source, risk area coordinates, and handling measures;

[0038] The risk types include at least one of obstacle collision risk, energy risk, regulatory boundary risk, weather risk, and model uncertainty risk.

[0039] Furthermore, the security constraint verification module includes:

[0040] a. Reachability verification unit: Determines whether candidate tracks are reachable based on current location, range capability, and power parameters;

[0041] b. Collision verification unit: Determines whether candidate tracks avoid obstacles and dynamic targets based on risk attribution results;

[0042] c. Boundary verification unit: Determines whether candidate flight paths are compliant based on no-fly zones, height-restricted zones, and privacy-sensitive areas;

[0043] d. Energy verification unit: Determines whether the candidate trajectory meets the energy reserve requirement for return based on the remaining battery power;

[0044] e. Confidence verification unit: Determines whether a candidate action has reached the confidence threshold based on the confidence level of the spatial target and the confidence level of the action parameters;

[0045] The security constraint verification module determines the verification status according to the following rules:

[0046] <1> The output is "pass" if there are no violations in any of the verification units.

[0047] <2> When there is a violation and it involves areas with large populations, temporary airspace restrictions, privacy-sensitive areas, model uncertainty, or the confidence level of alternative safety actions is lower than the preset safety action threshold, and a decision is required from the ground control terminal, manual confirmation will be output.

[0048] <3> If a violation exists and the manual confirmation trigger condition is not met, output "Reject";

[0049] <4> When manual confirmation is triggered by model uncertainty or insufficient confidence and the ground control terminal issues a confirmation execution command, the confidence verification unit will use the operator confirmation flag as an external confidence supplement input during the re-verification, and will take the larger value with the model confidence before making a confidence threshold judgment.

[0050] In the security constraint verification module, the security verification result includes verification status, structured rejection reason, manual confirmation flag, corresponding risk track segment and alternative safety action;

[0051] The verification status includes pass, reject, and manual confirmation;

[0052] The structured denial reason includes at least one of the following: denial reason type, risk track segment number, risk area coordinates, non-executable parameters, data source that triggered the denial, minimum safe distance, prohibited area, and alternative safety action.

[0053] Furthermore, the feedback constraint module includes:

[0054] 1) Rejection Reason Compilation Unit: Receives structured rejection reasons and risk attribution results;

[0055] 2) Replanning constraint generation unit: Generates replanning constraints including risk area coordinates, non-executable parameters, prohibited entry areas, minimum safe distance, alternative safe actions, supplementary task information, supplementary perception information, and replanning termination conditions;

[0056] 3) Alternative safety action generation unit: Generates at least one alternative safety action among long-distance observation, supplementary shooting, detour observation, hovering, return to base, or safe landing;

[0057] In the feedback constraint module, the process of generating replanning constraints and sending them to the multimodal task inference module for different scenarios includes the following cases:

[0058] A. When the security verification result is a rejection, the feedback constraint module converts the structured rejection reason into a replanning constraint and sends it to the multimodal task inference module;

[0059] B. When the candidate action field is incomplete, the parameter alignment confidence of the candidate action does not meet the preset alignment threshold, or the candidate trajectory generation module does not generate a candidate trajectory, the feedback constraint module generates supplementary task information or supplementary perception information replanning constraints and sends them to the multimodal task inference module.

[0060] C. When the security verification result requires manual confirmation, the system waits for confirmation instructions from the ground control terminal. These confirmation instructions can fall into one of three categories:

[0061] 1> If the instruction is confirmed to be a replanning, the feedback constraint module generates replanning constraints and sends them to the multimodal task inference module;

[0062] 2> If the confirmation command is to confirm execution, the safety constraint verification module will write the operator confirmation flag into the task context and use it as an external confidence input in the confidence verification of the re-verification. After the candidate track passes the re-verification, it will be sent to the flight control execution module. If the verification fails, the safety verification result will be converted into a rejection and the feedback constraint module will generate replanning constraints.

[0063] 3> If no confirmation instruction is received within the time limit, then hover, return to base, or land safely.

[0064] Furthermore, the system also includes a semantic map module, a cloud-edge collaborative scheduling module, a memory module, and a task skill library module;

[0065] The semantic map module includes:

[0066] ① Static layer: Writes the task area map, no-fly zone, altitude-restricted zone, and geographic reference information;

[0067] ② Dynamic layer: Writes information on moving targets, temporary obstacles, and weather changes;

[0068] ③ Mission layer: Write inspection points, search areas, spatial targets, and return points;

[0069] ④ Risk layer: Includes risk attribution results, security verification results, weak communication coverage areas, privacy-sensitive areas, and low-confidence areas;

[0070] The cloud-edge collaborative scheduling module selects inference nodes based on task type, link status, latency requirements, computing load, privacy level, model capability matching degree, and energy consumption constraints.

[0071] The memory module saves reasoning records, candidate action records, candidate trajectory records, risk attribution records, safety verification result records, replanning constraint records, execution result records, and operator preference records according to the task context identifier;

[0072] Each skill in the task skill library module includes a skill name, input parameter mode, output result mode, prerequisites, termination conditions, and security constraints.

[0073] The skills include at least one of the following: target search, target confirmation, close-range observation, long-range observation, obstacle avoidance, area coverage, tracking and shooting, anomaly reporting, return to base, and safe landing.

[0074] A safety constraint feedback planning method for a safety constraint feedback planning system for multimodal UAV missions includes the following steps:

[0075] Step 1: Receive images, videos, point clouds, maps, telemetry data, and natural language commands;

[0076] Step 2: Generate a spatiotemporal registration data package based on the images, videos, point clouds, maps, telemetry data, and natural language commands;

[0077] Step 3: Generate candidate actions based on the spatiotemporal registration data packet, and convert the candidate actions into candidate tracks with action step numbers and track segment numbers;

[0078] Step 4: Establish the correspondence between action step numbers and track segment numbers, output the risk type, risk level, trigger data source, and risk area coordinates for each track segment; and verify the candidate tracks.

[0079] Step 5: When the verification result is rejection, generate a structured rejection reason. When manual confirmation is required, output a manual confirmation flag, a structured rejection reason, and send a confirmation request. When the verification result is rejection or the manual confirmation instruction is replanning, convert the structured rejection reason into a replanning constraint.

[0080] Step 6: Regenerate candidate actions based on the replanning constraints and return to step 3, then repeat steps 3 to 4;

[0081] Step 7: When the manual confirmation command is confirmed to be executed, write the confirmation flag and verify the candidate track again. If the verification fails again, the verification result is changed to rejection, and steps 5 to 7 are repeated until the verification is successful or the replanning termination condition is met.

[0082] Step 8: When the candidate trajectory passes verification, the candidate trajectory is sent to the flight control execution module. When manual confirmation times out or the replanning termination condition is met but verification still fails, the UAV is hovered, returned to home, or landed safely.

[0083] Beneficial effects: (1) After the multimodal task reasoning module generates candidate actions, the candidate actions need to be converted into candidate tracks before they can enter the risk assessment and safety verification, avoiding the direct entry of abstract task actions into the flight control interface. (2) The candidate tracks have action step numbers and track segment numbers, so that the risk assessment results can be attributed to specific action steps and specific track segments. (3) The safety constraint verification module outputs structured rejection reasons, so that the safety verification results can be converted into replanning constraints by the feedback constraint module. (4) The multimodal task reasoning module regenerates candidate actions based on replanning constraints, so that the replanning has clear risk areas, non-executable parameters and alternative safe action sources. (5) The three-dimensional risk violation vector and segmented risk aggregation method retain the three types of risk sources: space collision, ground impact and airspace compliance, which facilitates the generation of rejection reasons by dimension. (6) The cloud-edge collaborative scheduling module restricts the flight control permissions of cloud reasoning results, reducing the risk that cloud results will directly affect flight control execution when the link is abnormal or the safety time limit is insufficient. (7) The feedback constraint module generates an independent operator preference record after the manual confirmation process is completed, which provides auxiliary reference for the selection of candidate actions in subsequent similar tasks. It runs independently of the safety verification and does not replace each other. Attached Figure Description

[0084] Figure 1 This is a structural diagram of the safety constraint feedback planning system for multimodal unmanned aerial vehicle (UAV) missions according to the present invention.

[0085] Figure 2 This is a flowchart of the spatiotemporal registration data packet and semantic map writing process of the present invention;

[0086] Figure 3 This is a flowchart of the candidate action to candidate trajectory and segmented risk attribution data of the present invention;

[0087] Figure 4 The closed-loop diagram for safety constraint verification and feedback in this invention has been redesigned.

[0088] Figure 5 This is a schematic diagram illustrating the binding of the action steps and track segments of the present invention;

[0089] Figure 6 This is a flowchart illustrating the manual confirmation and timeout handling process of the present invention;

[0090] Figure 7 This is a schematic diagram of a power transmission line inspection scenario according to Embodiment 1 of the present invention;

[0091] Figure 8This is a schematic diagram of a road emergency patrol scenario, which is an embodiment 2 of the present invention. Detailed Implementation

[0092] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0093] like Figure 1 As shown, the safety constraint feedback planning system for UAV multimodal missions of the present invention includes a multimodal acquisition module, a spatiotemporal synchronization and coordinate registration module, a multimodal mission inference module, a candidate trajectory generation module, a segmented risk assessment module, a safety constraint verification module, a feedback constraint module, and a flight control execution module. The modules exchange data through structured messages with mission context identifiers, timestamps, and data bodies, forming a closed loop of candidate actions, candidate trajectories, risk attribution, safety verification, and constraint feedback. The specific contents of each module are as follows:

[0094] 1. Multimodal acquisition module

[0095] like Figure 2 As shown, the multimodal acquisition module collects images, videos, point clouds, maps, telemetry data, and natural language commands; among them, the telemetry data includes the UAV's current position, attitude, speed, remaining battery power, range capability, power parameters, and link status.

[0096] 2. Spatiotemporal synchronization and coordinate registration module

[0097] The spatiotemporal synchronization and coordinate registration module converts image targets, point cloud obstacles, map areas, and natural language entities to the same low-altitude coordinate system based on the images, videos, point clouds, maps, telemetry data, and natural language commands acquired by the multimodal acquisition module, and generates a spatiotemporal registration data package.

[0098] The spatiotemporal registration data package includes: timestamp, UAV pose, current location, speed, remaining battery power, range capability, power parameters, link status, sensor type, data quality indicators, geographic coordinates, mission context identifier, and original data references.

[0099] 3. Multimodal Task Reasoning Module

[0100] The multimodal mission inference module generates candidate actions based on the spatiotemporal registration data package generated by the spatiotemporal synchronization and coordinate registration module. Candidate actions include a skill name, spatial target, action parameters, termination condition, action steps, action parameter confidence level, and safety constraint fields. Candidate actions are not directly sent to the flight control execution module but are instead used as input to the candidate trajectory generation module. If a candidate action lacks a skill name, spatial target, action parameters, or termination condition, the candidate trajectory generation module does not generate a candidate trajectory, and the feedback constraint module generates replanning constraints to supplement mission information. If the parameter alignment confidence level of a candidate action does not meet a preset alignment threshold, the feedback constraint module generates replanning constraints to supplement perception information.

[0101] The candidate action generation conditions for the multimodal task inference module are: all parameters of the candidate action are aligned across modalities and the alignment confidence is not lower than the preset alignment threshold.

[0102] The multimodal task reasoning module queries historical rejection reasons, historical replanning constraints, and operator preference records according to the task context identifier, including: (1) Visual encoding unit: outputs target category, target box, regional features, and target confidence; (2) Map encoding unit: outputs roads, routes, buildings, no-fly boundaries, and geographic reference points; (3) Telemetry encoding unit: outputs current location, speed, remaining battery power, and link quality; (4) Language parsing unit: outputs task entities and action intentions; (5) Cross-modal alignment unit: determines the matching relationship between language entities and spatial targets based on visual feature similarity, map spatial proximity, temporal consistency, and task context consistency; (6) Action generation unit: outputs candidate actions.

[0103] The cross-modal alignment unit determines the matching relationship according to the following priority order:

[0104] First priority: Spatial consistency rule. If the target entity T in the natural language instruction contains an absolute location description, it will be matched with the location in the spatial location M output by the map coding unit whose geographic coordinate deviation does not exceed a preset spatial threshold.

[0105] Second priority: visual feature rules. If T contains visual attribute descriptions, it will be prioritized to match the target in the candidate target set V output by the visual encoding unit with a confidence level not lower than the preset visual threshold and matching attributes.

[0106] Third priority: Task context rules. If T references the output of a previous action, then search for the same target reference in the completion criteria of the previous action according to the task context identifier. The matching condition is that the task context identifier is consistent and the spatial coordinate deviation does not exceed the preset context matching threshold.

[0107] Fourth priority: Temporal consistency rule. If T is mentioned in multiple consecutive frames, the candidate target with the highest trajectory smoothness is selected first.

[0108] 4. Candidate Track Generation Module

[0109] like Figure 3 As shown, the candidate trajectory generation module converts the candidate actions into candidate trajectories with action step numbers and trajectory segment numbers based on the skill names and action steps in the candidate actions generated by the multimodal task reasoning module. It also establishes a correspondence between action step numbers and trajectory segment numbers through the skill names and action steps in the candidate actions and sends them to the segment risk assessment module.

[0110] The candidate track includes waypoint coordinates, estimated arrival time, flight altitude, target speed, sensor attitude, backup waypoints, action step number, and track segment number.

[0111] To formally describe the binding relationship between action steps in candidate actions and candidate track segments, the mapping function is defined as follows:

[0112]

[0113] in, Indicates the first One candidate action; Indicates candidate actions A set of action steps; Indicates candidate actions The generated candidate tracks; This represents the set of track segments for the candidate track; This indicates the mapping relationship between action step numbers and track segment numbers.

[0114] This mapping relationship forms the basis for subsequent segmented risk attribution and structured rejection reasons to be linked to specific track segments. For example, a candidate close-up observation action includes three steps: approach, hovering and taking pictures, and exiting. After mapping, these steps correspond to three track segments, enabling risk assessment to accurately pinpoint the insufficient distance between the approach segment and the obstacle, rather than simply reporting a risk in the close-up observation action.

[0115] 5. Segmented Risk Assessment Module

[0116] like Figure 5As shown, the segmented risk assessment module performs segmented risk attribution analysis on each track segment based on the correspondence established by the candidate track generation module, and outputs the risk attribution results. The risk attribution results include the track segment number, risk type, risk level, trigger layer, trigger data source, risk area coordinates, and mitigation measures. The risk type includes at least one of obstacle collision risk, energy risk, regulatory boundary risk, meteorological risk, and model uncertainty risk. If the candidate action includes three steps: approach, hovering and shooting, and exit, the segmented risk assessment module outputs obstacle collision risk, energy risk, regulatory boundary risk, meteorological risk, or model uncertainty risk for the approach segment, hovering and shooting segment, and exit segment, respectively, and records the corresponding risk area coordinates, trigger data source, and mitigation measures.

[0117] 6. Security Constraint Verification Module

[0118] like Figure 4 As shown, the safety constraint verification module includes: a. Accessibility verification unit: determines whether the candidate trajectory is reachable based on the current location, range capability, and power parameters; b. Collision verification unit: determines whether the candidate trajectory avoids obstacles and dynamic targets based on risk attribution results; c. Boundary verification unit: determines whether the candidate trajectory complies with no-fly zones, height-restricted zones, and privacy-sensitive areas; d. Energy verification unit: determines whether the candidate trajectory meets the return energy margin after execution based on the remaining battery power; e. Confidence verification unit: determines whether the candidate action reaches the confidence threshold based on the confidence level of the spatial target and the confidence level of the action parameters.

[0119] The safety constraint verification module performs reachability, collision, boundary, energy, and confidence verifications based on candidate tracks, risk attribution results, and UAV telemetry status, and outputs safety verification results. The safety verification results include verification status, structured rejection reasons, manual confirmation flags, corresponding risk track segments, and alternative safety actions.

[0120] The structured denial reasons include at least one of the following: denial reason type, risk track segment number, risk area coordinates, non-executable parameters, data source that triggered the denial, minimum safe distance, prohibited area, and alternative safety actions;

[0121] The verification status includes pass, rejection, and manual confirmation. When the verification status is pass, the flight control execution module receives candidate tracks. When the verification status is rejection, the feedback constraint module generates replanning constraints. When the verification status is manual confirmation, the system enters the ground control terminal confirmation process. Only after receiving the confirmation execution command and passing the safety constraint verification again will the candidate track be sent to the flight control execution module. If the safety constraint verification still fails, the result is automatically converted to a rejection status, and the feedback constraint module generates replanning constraints.

[0122] The security constraint verification module determines the verification status according to the following rules: <1> The output is "pass" if there are no violations in any of the verification units. <2> When there is a violation and it involves areas with large populations, temporary airspace restrictions, privacy-sensitive areas, model uncertainty, or the confidence level of alternative safety actions is lower than the preset safety action threshold, and a decision is required from the ground control terminal, manual confirmation will be output. <3> If a violation exists and the manual confirmation trigger condition is not met, output "Reject"; <4> When manual confirmation is triggered by model uncertainty or insufficient confidence and the ground control terminal issues a confirmation execution command, the confidence verification unit will use the operator confirmation flag as an external confidence supplement input during the re-verification, and will take the larger value with the model confidence before making a confidence threshold judgment.

[0123] 7. Feedback Constraint Module

[0124] The feedback constraint module includes: 1) a rejection reason processing unit: receiving structured rejection reasons and risk attribution results; 2) a replanning constraint generation unit: generating replanning constraints including risk area coordinates, non-executable parameters, prohibited entry areas, minimum safe distance, alternative safety actions, supplementary task information, supplementary perception information, and replanning termination conditions; 3) an alternative safety action generation unit: generating at least one alternative safety action among long-distance observation, supplementary shooting, detour observation, hovering, return to base, or safe landing;

[0125] Based on the full verification result output by the security constraint verification module, the feedback constraint module generates replanning constraints and sends them to the multimodal task inference module in the following cases:

[0126] A. When the security verification result is rejection, the feedback constraint module converts the structured rejection reason into a replanning constraint and sends it to the multimodal task inference module;

[0127] B. When the candidate action field is incomplete, the parameter alignment confidence of the candidate action does not meet the preset alignment threshold, or the candidate trajectory generation module does not generate a candidate trajectory, the feedback constraint module generates supplementary task information or supplementary perception information replanning constraints and sends them to the multimodal task inference module.

[0128] C. When the security verification result requires manual confirmation, the system waits for confirmation instructions from the ground control terminal. These confirmation instructions can fall into one of three categories:

[0129] 1> If the instruction is confirmed to be a replanning, the feedback constraint module generates replanning constraints and sends them to the multimodal task inference module;

[0130] 2> If the confirmation command is to confirm execution, the safety constraint verification module will write the operator confirmation flag into the task context and use it as an external confidence input in the confidence verification of the re-verification. After the candidate track passes the re-verification, it will be sent to the flight control execution module. If the verification fails, the safety verification result will be converted into a rejection and the feedback constraint module will generate replanning constraints.

[0131] 3> If no confirmation instruction is received within the time limit, then hover, return to base, or land safely.

[0132] The replanning constraints include risk area coordinates, non-executable parameters, prohibited areas, minimum safe distance, alternative safe actions, supplementary task information, supplementary perception information, and replanning termination conditions.

[0133] The feedback constraint module generates an operator preference record after the manual confirmation process is completed. This record is stored independently of the structured rejection reasons. To enable the safety constraint verification module to quantify the reasons for verification failure and output them as structured rejection reasons, a three-dimensional risk violation vector is used to describe the safety constraint violations of candidate waypoints or waypoint segments. The specific definitions are as follows:

[0134]

[0135] in, Indicates a waypoint or waypoint segment in a candidate track; Indicates waypoints or waypoint segments The three-dimensional risk violation vector; This indicates the amount of space collision risk that may be violated. Indicates the amount of ground impact risk violation; This indicates the amount of airspace compliance risk violations.

[0136] For meteorological risks and model uncertainty risks, the segmented risk assessment module can map their impact on collision distance, flight altitude, energy margin, or airspace compliance to the aforementioned three-dimensional violation vector. When model uncertainty risks cause the confidence level of spatial targets or action parameters to fall below a preset threshold, and the risk involves areas with high population density, temporary airspace restrictions, privacy-sensitive areas, or alternative safe actions with confidence levels below a preset safe action threshold, the safety constraint verification module outputs a manual confirmation flag. When the confidence level is insufficient but does not meet the manual confirmation triggering conditions, the feedback constraint module generates replanning constraints with supplementary perception information. In other cases where verification fails, a rejection is output.

[0137] When manual confirmation is triggered by model uncertainty, insufficient confidence in space targets, or insufficient confidence in motion parameters, and the ground control terminal returns a confirmation execution command, the system writes the operator confirmation flag into the task context. During subsequent safety constraint verification, the confidence verification unit uses the operator confirmation flag as an external confidence supplement input, compares it with the larger value of the model confidence, and then compares it with a preset alignment threshold or confidence threshold. The operator confirmation flag only applies to confidence verification and does not change the physical constraint violations such as space collision, airspace boundary, energy, accessibility, and altitude.

[0138] In addition to model uncertainties, when safety constraint verification detects that a candidate trajectory enters a populated area, a temporary airspace restriction area, or a privacy-sensitive area, and the operator needs to make a final decision based on on-site visual information, mission objectives, or alternative safety actions, the safety constraint verification module also outputs a manual confirmation flag. For manual confirmations triggered by violations of physical constraints such as altitude, no-entry boundaries, collision distance, or return-to-home energy, the ground control terminal provides replanning or timeout protection options by default. Confirmation execution is not a condition for bypassing physical constraints; if the operator chooses to confirm execution, the candidate trajectory must still pass the safety constraint verification again. If it fails, it automatically enters the replanning process of the feedback constraint module.

[0139] The preset safety action threshold is configurable, preferably 0.5, and can be adjusted according to mission time pressure, complexity of alternative actions, UAV platform capabilities, and on-site environmental risk level. Based on the above three-dimensional risk violation vector, the permission criteria for whether a waypoint or waypoint segment can enter the flight control execution process are defined as follows:

[0140]

[0141] in, Indicates waypoints or waypoint segments The word "perhaps" can be used; when When true, the waypoint or waypoint segment is allowed to enter the flight control execution module; when When the result is false, the safety constraint verification module generates a structured rejection reason based on the non-zero violation dimension and outputs rejection or manual confirmation according to the manual confirmation trigger criterion; the feedback constraint module only generates replanning constraints when the verification status is rejection or the manual confirmation instruction is replanning. Compared with a single comprehensive risk score, the three-dimensional violation vector retains the distinguishability of risk sources, making it easier to distinguish between collision risk, ground impact risk, and airspace compliance risk, and generate corresponding rejection reasons accordingly. To merge waypoint-level risks into track segment-level risks, the segment risk assessment module uses a dimension-wise maximum value aggregation method to calculate the segment risk violation vector:

[0142]

[0143] in, Indicates the first Each flight path segment; Indicates the first segment within the track. One waypoint or waypoint segment; Indicates waypoints or waypoint segments The three-dimensional risk violation vector; Indicates track segments The segmented risk violation vector; This means taking the maximum violation amount for each waypoint risk violation vector within the segment, dimension by dimension.

[0144] 8. Flight control execution module

[0145] The flight control execution module only executes candidate tracks that have passed safety constraint verification.

[0146] In addition, the safety constraint feedback planning system for UAV multimodal missions of the present invention also includes: a semantic map module, a cloud-edge collaborative scheduling module, a memory module, and a mission skill library module;

[0147] The semantic map module includes: ① Static layer: writing the mission area map, no-fly zone, altitude-restricted zone and geographic reference information; ② Dynamic layer: writing moving targets, temporary obstacles and weather changes; ③ Mission layer: writing inspection points, search areas, spatial targets and return points; ④ Risk layer: writing risk attribution results, security verification results, weak communication coverage areas, privacy-sensitive areas and low-confidence areas.

[0148] The cloud-edge collaborative scheduling module selects inference nodes based on task type, link status, latency requirements, computing load, privacy level, model capability matching degree, and energy consumption constraints. Specifically, the selection of inference nodes is divided into the following two-stage decision-making logic:

[0149] The first stage generates a candidate node set based on model capability and risk thresholds. The second stage selects the node with the lowest overall cost from the candidate node set. When the link quality is below a preset threshold, the remaining battery power is below the return-to-home threshold, or the task is a runaway protection task, the cloud inference result is only used as the semantic input of the task and does not directly generate candidate tracks or flight control actions. Safety-related actions such as obstacle avoidance, hovering, return-to-home, and safe landing are executed by the UAV or edge station after passing safety constraint verification. When the link is disconnected, the edge station is unavailable, or the latency exceeds a preset threshold, the UAV executes the action independently based on its local safety policy. When the link is normal and the edge station meets the low-latency control conditions, the edge station issues safety action commands verified by safety constraints. When the cloud is unavailable but the edge station is available, the edge station takes over the selection of inference nodes or the issuance of safety action commands, without directly triggering local degradation on the UAV.

[0150] The node selection execution rules for the cloud-edge collaborative scheduling module are as follows: The first stage is candidate set screening. Nodes selected for the candidate set must simultaneously meet the model capability threshold and the risk threshold, where the risk threshold function is:

[0151]

[0152] in, Represents a node Corresponding link risk level Indicates the remaining battery power. This indicates the amount of electricity required for the return trip. This indicates the time pressure of the task. The larger the value, the less sufficient the safety time margin of the task, and the less suitable it is to choose high link risk or remote inference nodes. This is a weighting coefficient. When the link quality is lower than a preset link threshold or the remaining battery power is lower than a preset multiple of the return-to-home threshold, It is set to an over-limit state. If the task requirement only indicates high computing power priority rather than safety time pressure, then this priority is not included in the risk threshold function. The time delay weight or node capability threshold can be adjusted.

[0153] The second stage is cost ranking, in the candidate node set. Select the node with the minimum overall cost. As an execution node:

[0154]

[0155]

[0156] in, The sum of inference delay and communication transmission delay. This is the sum of inference energy consumption and transmission energy consumption. and Weighting is applied at a cost. When the link quality is below a preset threshold, the remaining battery power is below the return-to-home threshold, or the mission is a runaway protection mission, the cloud inference results are only used as semantic input for the mission and are not directly used to generate candidate tracks or flight control actions.

[0157] The memory module saves inference records, candidate action records, candidate trajectory records, risk attribution records, safety verification result records, replanning constraint records, execution result records, and operator preference records according to the task context identifier.

[0158] Each skill in the mission skill library module includes a skill name, input parameter mode, output result mode, prerequisites, termination conditions, and safety constraints; wherein, the skill includes at least one of target search, target confirmation, close-range observation, long-range observation, obstacle avoidance, area coverage, tracking and shooting, anomaly reporting, return to base, and safe landing.

[0159] like Figures 2-4 As shown, the safety constraint feedback planning method of the safety constraint feedback planning system for UAV multimodal missions of the present invention includes the following steps:

[0160] Step 1: The system receives natural language task instructions and simultaneously collects images, videos, point clouds, maps, and telemetry data.

[0161] Step 2: The spatiotemporal synchronization and coordinate registration module generates a spatiotemporal registration data package and converts multi-source data to the same task context identifier and the same low-altitude coordinate system.

[0162] Step 3: The multimodal task reasoning module generates candidate actions based on the spatiotemporal registration data package. The candidate actions include skill name, spatial target, action parameters and termination conditions.

[0163] Step 4: The candidate track generation module converts the candidate actions into candidate tracks with action step numbers and track segment numbers.

[0164] Step 5: The segmented risk assessment module outputs the risk type, risk level, trigger data source, and risk area coordinates for candidate tracks based on the correspondence between action step numbers and track segment numbers.

[0165] Step 6: The safety constraint verification module performs reachability, collision, boundary, energy, and confidence verification on the candidate tracks.

[0166] Step 7: If the safety verification result is passed, the flight control execution module receives the candidate trajectory and executes the corresponding task skill according to the waypoint coordinates, flight altitude, target speed and sensor attitude in the candidate trajectory.

[0167] Step 8: If the security verification result is rejection, the security constraint verification module outputs a structured rejection reason, and the feedback constraint module converts the structured rejection reason into a redesign constraint; if the security verification result is manual confirmation, the security constraint verification module outputs a manual confirmation flag and a structured rejection reason, and enters the ground control terminal confirmation process.

[0168] Step 9: When the safety verification result is rejection or the manual confirmation instruction is replanning, the multimodal task inference module reads the risk area coordinates, non-executable parameters, minimum safe distance and alternative safe actions in the replanning constraints, regenerates candidate actions, and returns to step 4; when the manual confirmation instruction is confirmation to execute, the system writes the operator confirmation flag into the task context, the candidate track re-enters the safety constraint verification module, and if the verification fails again, it is converted to a rejection state and the feedback constraint module generates replanning constraints before returning to step 4.

[0169] Step 10: After the task is completed, rejected, replanned, or transferred to manual confirmation, the memory module writes the reasoning record, candidate action record, candidate trajectory record, risk attribution record, safety verification result record, replanning constraint record, execution result record, and operator preference record.

[0170] like Figure 6 As shown, when the safety constraint verification module outputs a manual confirmation flag, the system sends candidate actions, candidate tracks, risk levels, risk track segments, structured rejection reasons, and alternative safety actions to the ground control terminal. If a confirmation execution command is received within the preset confirmation time, the system writes an operator confirmation flag, the candidate track undergoes safety constraint verification again, and is sent to the flight control execution module after successful verification. If the verification fails again, the system is converted to a rejection state, and the feedback constraint module generates replanning constraints. If the confirmation command is replanning, the feedback constraint module generates replanning constraints. If no confirmation command is received within the time limit, hovering, returning to home, or safe landing are executed.

[0171] In one optional embodiment, the multimodal task reasoning module supports an auxiliary decision-making function based on operator preference records: when the accumulated number of operator preference records reaches a preset threshold, the historical records of operators confirming, selecting alternative safety actions, or manually specifying alternative actions in the manual confirmation process are used as auxiliary references for candidate action generation. Candidate actions consistent with the operator's historical preferences are prioritized in the same or similar task contexts. This auxiliary decision-making function does not affect the independent verification of the safety constraint verification module, nor does it replace the constraint generation process of the feedback constraint module.

[0172] Example 1:

[0173] like Figure 7 As shown, in the transmission line inspection task, the user inputs "Observe and photograph the insulator on the right side of the transmission line tower at close range, and return to long-range observation if necessary." The system execution flow is as follows:

[0174] Step 1: The multimodal acquisition module acquires RGB 4K images, 30fps video, LiDAR point cloud with an accuracy of ±2cm around the tower, and telemetry data at a horizontal distance of 45 meters from the tower. The spatiotemporal synchronization and coordinate registration module generates a spatiotemporal registration data package.

[0175] Step 2: The multimodal task inference module detects the third string of insulators on the right side of the tower in the image through the visual encoding unit, with a target confidence of 0.87. The phrase "right-side insulator" in natural language is matched to this spatial target, with a coordinate deviation of 4.3 meters, which is lower than the preset spatial threshold. Based on this, the multimodal task inference module generates candidate actions for close-range observation, with action parameters including an observation distance of 3 meters, a pan-tilt angle of -15°, and at least 5 shots taken.

[0176] Step 3: The candidate trajectory generation module generates an approach waypoint that flies 42 meters from the current position toward the tower, an observation waypoint that hovers 3 meters away from the insulator, a shooting waypoint for circling and shooting at 5 positions, and an exit waypoint that retreats to a safe distance of 15 meters, based on the close-range observation skill parameters. The module also writes the action step number and trajectory segment number for the above waypoints.

[0177] Step 4: The segmented risk assessment module found that the horizontal distance between the third waypoint near the segment and the 220kV transmission line was only 1.8 meters, which is less than the minimum safe distance of 5 meters. Therefore, it outputs the normalized violation amount of obstacle collision risk. The data includes the segment number, the cylindrical safety boundary area extending 5 meters outward from the line center, the point cloud data source that triggered the risk, and the record of a safety distance of less than 3.2 meters in the structured engineering quantity.

[0178] Step 5: The safety constraint verification module generates structured rejection reasons. The rejection reason type is space collision risk. The safe distance is less than 3.2 meters; the risk track segment number is approaching the 3rd waypoint of the segment; the risk area coordinates are the safety boundary outside the line with a radius of 5 meters and the center line of the conductor as the axis; the unexecutable parameter is that the approaching waypoint is 1.8 meters away from the conductor, which does not meet the minimum safe distance of 5 meters; the alternative safe action is to take a long-distance zoom photo with an observation distance of 15 meters and optical zoom of 10x.

[0179] Step 6: The feedback constraint module converts the structured rejection reasons into replanning constraints. The replanning constraints include prohibiting candidate tracks from entering the line safety boundary formed by extending 5 meters outside the conductor centerline, ensuring that the target observation distance is not less than 5 meters, adjusting the sensor attitude to 10x optical zoom observation, and changing the candidate action termination condition to completing long-distance zoom shooting and ensuring that the image resolution of the insulator area is not less than 0.5 mm / pixel.

[0180] Step 7: The multimodal task reasoning module generates candidate actions for long-range zoom shooting based on the replanning constraints, and the candidate trajectory generation module regenerates the long-range observation trajectory; after the safety verification is passed, the flight control execution module completes the shooting according to the long-range observation trajectory and returns to base.

[0181] Example 2:

[0182] like Figure 8 As shown, in the road emergency patrol task, the user enters "Patrol the accident site ahead, observe the distribution of vehicles and people, and find a safe observation position".

[0183] The system execution flow is as follows:

[0184] Step 1: The drone is currently at an altitude of 50 meters, approximately 300 meters from the accident site, and has 71% battery remaining. The multimodal acquisition module collects images, videos, point clouds, and telemetry data from the area surrounding the accident site. The spatiotemporal synchronization and coordinate registration module encapsulates the data under the same task context identifier.

[0185] Step 2: The multimodal task inference module identifies the accident site through the visual encoding unit. The scene shows a rear-end collision between two vehicles, with debris scattered over an area of ​​approximately 15 meters × 8 meters. Simultaneously, it detects a gathering of approximately 12 people on the sidewalk east of the accident site. Based on this, the multimodal task inference module generates candidate low-altitude observation actions, with parameters including an observation height of 20 meters and a coverage area of ​​50 meters × 30 meters.

[0186] Step 3: The candidate trajectory generation module generates three trajectory segments: a 280-meter flight segment to the accident site, a low-altitude observation segment that descends to 20 meters and hovers to take pictures, and a segment that moves westward to find an observation position.

[0187] Step 4: The segmented risk assessment module found that the second waypoint of the low-altitude observation segment was located directly above a densely populated area, and its observation altitude of 20 meters was lower than the minimum safe observation altitude of 50 meters for that area. The ground impact risk violation, normalized to the minimum safe observation altitude, was 0.60. The same waypoint also entered the strictly prohibited airspace below 30 meters directly above the densely populated area, resulting in an airspace compliance risk violation of 0.33 normalized to the prohibited altitude boundary. The final waypoint of the search for observation location segment was only 8 meters from the restricted flight boundary of the urban expressway, lower than the 10-meter default regulatory buffer distance set by the semantic map static layer for the urban expressway restricted flight zone. The airspace compliance risk violation, normalized to the regulatory buffer distance, was 0.20. The risk attribution results accurately pinpointed the second waypoint of the low-altitude observation segment and the final waypoint of the search for observation location segment.

[0188] Step 5: The safety constraint verification module outputs a manual confirmation flag and a structured rejection reason. The triggering reason is that the candidate flight path involves flying above a densely populated area and there is a hard no-entry airspace boundary. The structured rejection reason includes the low-altitude observation segment number, the coordinates of the densely populated area within 20 meters east of the incident point, ground impact risk reasons, airspace compliance risk reasons, the minimum safe observation altitude of 50 meters, and alternative safety actions. The ground impact risk reason indicates that the minimum safe observation altitude for the densely populated area is 50 meters, and the current altitude is 20 meters. The airspace compliance risk reason indicates that the airspace below 30 meters directly above the densely populated area is a prohibited area, entry is not allowed within the 10-meter regulatory buffer zone of the urban expressway flight restriction boundary, and the terminal waypoint is 8 meters away from the boundary. The alternative safety action is high-altitude coverage observation at an altitude of 80 meters, using a zoom lens.

[0189] Step 6: The feedback constraint module converts the structured rejection reasons into replanning constraints: the observation altitude is not less than 50 meters, the candidate flight path does not cross the airspace below 30 meters directly above the population gathering area, the candidate flight path maintains a monitoring buffer distance of not less than 10 meters from the restricted flight boundary of the urban expressway, and the observation is carried out along the west side of the road where there are no people, or it is changed to 80-meter high-altitude coverage observation.

[0190] Step 7: If the ground control terminal confirms the adoption of the high-altitude coverage observation scheme within 15 seconds, that is, the confirmation instruction is to replan, the system will regenerate an 80-meter altitude coverage track and execute it after verification; if the manual confirmation timeout occurs, the flight control execution module will hover at the current 50-meter altitude and wait for further instructions.

[0191] The above embodiments are used to explain and illustrate the present invention, and not to limit it. Any modifications and changes made to the present invention within the spirit and scope of the claims fall within the protection scope of the present invention.

Claims

1. A safety constraint feedback planning system for multimodal unmanned aerial vehicle (UAV) missions, characterized in that, Includes the following modules: Multimodal acquisition module: Acquires images, videos, point clouds, maps, telemetry data, and natural language commands; Spatiotemporal synchronization and coordinate registration module: Generates a spatiotemporal registration data package based on the images, videos, point clouds, maps, telemetry data, and natural language commands acquired by the multimodal acquisition module; Multimodal task reasoning module: Generates candidate actions based on the spatiotemporal registration data packet generated by the spatiotemporal synchronization and coordinate registration module; Candidate track generation module: Converts the candidate actions generated by the multimodal task reasoning module into candidate tracks with action step numbers and track segment numbers, and establishes the correspondence between action step numbers and track segment numbers; Segmented Risk Assessment Module: Outputs risk attribution results for each track segment based on the correspondence established by the candidate track generation module; Safety constraint verification module: Based on candidate tracks, risk attribution results, and UAV telemetry status, perform reachability, collision, boundary, energy, and confidence verification, and output safety verification results; Feedback constraint module: Based on the full verification results output by the security constraint verification module, it generates replanning constraints according to different cases and sends them to the multimodal task inference module; Flight control execution module: Receives candidate tracks verified by the safety constraint verification module.

2. The safety constraint feedback planning system for multimodal UAV missions according to claim 1, characterized in that, The spatiotemporal synchronization and coordinate registration module converts image targets, point cloud obstacles, map regions, and natural language entities into the same low-altitude coordinate system; the spatiotemporal registration data package includes: timestamp, UAV pose, current position, speed, remaining battery power, range capability, power parameters, link status, sensor type, data quality indicators, geographic coordinates, mission context identifier, and original data reference.

3. The safety constraint feedback planning system for multimodal UAV missions according to claim 1, characterized in that, The multimodal task reasoning module queries historical rejection reasons, historical replanning constraints, and operator preference records according to the task context identifier. The multimodal task reasoning module includes the following units: (1) Visual encoding unit: outputs target category, target bounding box, region features and target confidence; (2) Map coding unit: Outputs roads, routes, buildings, no-fly zones and geographic reference points; (3) Telemetry coding unit: outputs current location, speed, remaining power and link quality; (4) Language parsing unit: Outputs task entities and action intentions; (5) Cross-modal alignment unit: Determine the matching relationship between language entities and spatial targets based on visual feature similarity, map spatial proximity, temporal consistency and task context consistency; (6) Action generation unit: Outputs candidate actions; The action generation condition of the multimodal task reasoning module is that all parameters of the candidate action are aligned across modalities and the alignment confidence is not lower than the preset alignment threshold.

4. A safety constraint feedback planning system for multimodal UAV missions according to claim 3, characterized in that, The cross-modal alignment unit determines the matching relationship according to the following priority order: First priority: Spatial consistency rule. If the target entity T in the natural language instruction contains an absolute location description, it will be matched with the location in the spatial location M output by the map coding unit whose geographic coordinate deviation does not exceed a preset spatial threshold. Second priority: visual feature rules. If T contains visual attribute descriptions, it will be prioritized to match the target in the candidate target set V output by the visual encoding unit with a confidence level not lower than the preset visual threshold and matching attributes. Third priority: Task context rules. If T references the output of a previous action, then search for the same target reference in the completion criteria of the previous action according to the task context identifier. The matching condition is that the task context identifier is consistent and the spatial coordinate deviation does not exceed the preset context matching threshold. Fourth priority: Temporal consistency rule. If T is mentioned in multiple consecutive frames, the candidate target with the highest trajectory smoothness is selected first. In the action generation unit, the candidate action includes: skill name, spatial target, action parameters, termination condition, action steps, action parameter confidence level, and safety constraint fields.

5. A safety constraint feedback planning system for multimodal UAV missions according to claim 1, characterized in that, In the candidate trajectory generation module, the candidate trajectory includes waypoint coordinates, estimated arrival time, flight altitude, target speed, sensor attitude, backup waypoints, action step number, and trajectory segment number; The candidate trajectory generation module establishes a correspondence between action step numbers and trajectory segment numbers by using the skill names and action steps in the candidate actions.

6. A safety constraint feedback planning system for multimodal UAV missions according to claim 1, characterized in that, The risk attribution results output by the segmented risk assessment module include the track segment number, risk type, risk level, trigger layer, trigger data source, risk area coordinates, and handling measures. The risk types include at least one of obstacle collision risk, energy risk, regulatory boundary risk, weather risk, and model uncertainty risk.

7. A safety constraint feedback planning system for multimodal UAV missions according to claim 1, characterized in that, The security constraint verification module includes: a. Reachability verification unit: Determines whether candidate tracks are reachable based on current location, range capability, and power parameters; b. Collision verification unit: Determines whether candidate tracks avoid obstacles and dynamic targets based on risk attribution results; c. Boundary verification unit: Determines whether candidate flight paths are compliant based on no-fly zones, height-restricted zones, and privacy-sensitive areas; d. Energy verification unit: Determines whether the candidate trajectory meets the energy reserve for return based on the remaining battery power; e. Confidence verification unit: Determines whether a candidate action has reached the confidence threshold based on the confidence level of the spatial target and the confidence level of the action parameters; The security constraint verification module determines the verification status according to the following rules: <1> The output is "pass" if there are no violations in any of the verification units. <2> When there is a violation and it involves areas with large populations, temporary airspace restrictions, privacy-sensitive areas, model uncertainty, or the confidence level of alternative safety actions is lower than the preset safety action threshold, and a decision is required from the ground control terminal, manual confirmation will be output. <3> If a violation exists and the manual confirmation trigger condition is not met, output "Reject"; <4> When manual confirmation is triggered by model uncertainty or insufficient confidence and the ground control terminal issues a confirmation execution command, the confidence verification unit will use the operator confirmation flag as an external confidence supplement input during the re-verification, and will take the larger value with the model confidence before making a confidence threshold judgment. In the security constraint verification module, the security verification result includes verification status, structured rejection reason, manual confirmation flag, corresponding risk track segment and alternative safety action; The verification status includes pass, reject, and manual confirmation; The structured denial reason includes at least one of the following: denial reason type, risk track segment number, risk area coordinates, non-executable parameters, data source that triggered the denial, minimum safe distance, prohibited area, and alternative safety action.

8. A safety constraint feedback planning system for multimodal UAV missions according to claim 1, characterized in that, The feedback constraint module includes: 1) Rejection Reason Compilation Unit: Receives structured rejection reasons and risk attribution results; 2) Replanning constraint generation unit: Generates replanning constraints including risk area coordinates, non-executable parameters, prohibited areas, minimum safe distance, alternative safe actions, supplementary task information, supplementary perception information, and replanning termination conditions; 3) Alternative safety action generation unit: Generates at least one alternative safety action among long-distance observation, supplementary shooting, detour observation, hovering, return to base, or safe landing; In the feedback constraint module, the process of generating replanning constraints and sending them to the multimodal task inference module for different scenarios includes the following cases: A. When the security verification result is a rejection, the feedback constraint module converts the structured rejection reason into a replanning constraint and sends it to the multimodal task inference module; B. When the candidate action field is incomplete, the parameter alignment confidence of the candidate action does not meet the preset alignment threshold, or the candidate trajectory generation module does not generate a candidate trajectory, the feedback constraint module generates supplementary task information or supplementary perception information replanning constraints and sends them to the multimodal task inference module. C. When the security verification result requires manual confirmation, the system waits for confirmation instructions from the ground control terminal. These confirmation instructions can fall into one of three categories: 1> If the instruction is confirmed to be a replanning, the feedback constraint module generates replanning constraints and sends them to the multimodal task inference module; 2> If the confirmation command is to confirm execution, the safety constraint verification module will write the operator confirmation flag into the task context and use it as an external confidence input in the confidence verification of the re-verification. After the candidate track passes the re-verification, it will be sent to the flight control execution module. If the verification fails, the safety verification result will be converted into a rejection and the feedback constraint module will generate replanning constraints. 3> If no confirmation instruction is received within the time limit, then hover, return to base, or land safely.

9. A safety constraint feedback planning system for multimodal UAV missions according to claim 1, characterized in that, The system also includes a semantic map module, a cloud-edge collaborative scheduling module, a memory module, and a task skill library module; The semantic map module includes: ① Static layer: Writes the task area map, no-fly zone, altitude-restricted zone, and geographic reference information; ② Dynamic layer: Writes information on moving targets, temporary obstacles, and weather changes; ③ Mission layer: Write inspection points, search areas, spatial targets, and return points; ④ Risk layer: Includes risk attribution results, security verification results, weak communication coverage areas, privacy-sensitive areas, and low-confidence areas; The cloud-edge collaborative scheduling module selects inference nodes based on task type, link status, latency requirements, computing load, privacy level, model capability matching degree, and energy consumption constraints. The memory module stores reasoning records, candidate action records, candidate trajectory records, risk attribution records, safety verification result records, replanning constraint records, execution result records, and operator preference records according to the task context identifier; Each skill in the task skill library module includes a skill name, input parameter mode, output result mode, prerequisites, termination conditions, and security constraints. The skills include at least one of the following: target search, target confirmation, close-range observation, long-range observation, obstacle avoidance, area coverage, tracking and shooting, anomaly reporting, return to base, and safe landing.

10. A safety constraint feedback planning method based on the safety constraint feedback planning system for multimodal UAV missions as described in claim 1, characterized in that, Includes the following steps: Step 1: Receive images, videos, point clouds, maps, telemetry data, and natural language commands; Step 2: Generate a spatiotemporal registration data package based on the images, videos, point clouds, maps, telemetry data, and natural language commands; Step 3: Generate candidate actions based on the spatiotemporal registration data packet, and convert the candidate actions into candidate tracks with action step numbers and track segment numbers; Step 4: Establish the correspondence between action step numbers and track segment numbers, output the risk type, risk level, trigger data source, and risk area coordinates for each track segment; and verify the candidate tracks. Step 5: When the verification result is rejection, generate a structured rejection reason. When manual confirmation is required, output a manual confirmation flag, a structured rejection reason, and send a confirmation request. When the verification result is rejection or the manual confirmation instruction is replanning, convert the structured rejection reason into a replanning constraint. Step 6: Regenerate candidate actions based on the replanning constraints and return to step 3, then repeat steps 3 to 4; Step 7: When the manual confirmation command is confirmed to be executed, write the confirmation flag and verify the candidate track again. If the verification fails again, the verification result is changed to rejection, and steps 5 to 7 are repeated until the verification is successful or the replanning termination condition is met. Step 8: When the candidate trajectory passes verification, the candidate trajectory is sent to the flight control execution module. When manual confirmation times out or the replanning termination condition is met but verification still fails, the UAV is hovered, returned to home, or landed safely.

Images