Unmanned aerial vehicle adaptive shooting method and system based on multiple sensors and AI

By combining multiple sensors with AI technology, real-time path planning and image optimization for drones in complex environments are achieved, solving the problems of insufficient perception and safety of drones in complex environments, and improving shooting quality and battery life.

CN121541459AInactive Publication Date: 2026-02-17广东财贸职业学院
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202511399776.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-28
Publication Date
2026-02-17
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing drone photography technology lacks sufficient perception capabilities in complex environments, cannot avoid obstacles in real time, has low shooting efficiency and is prone to collisions, lacks intelligent parameter adjustment, resulting in blurry images or inaccurate exposure, and has limited battery life.

Method used

Employing multi-sensor fusion and AI technology, a dynamic environmental model is constructed using the Transformer-GNN deep learning architecture. Combining RRT global planning and DWA local obstacle avoidance, real-time path optimization is achieved, and parameters are automatically adjusted by evaluating image quality using a lightweight CNN.

Benefits of technology

It significantly improves the safety, adaptability, and operational efficiency of drones in complex environments, ensures stable image quality, enhances robustness and endurance, and avoids collisions and sensor system failures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121541459A_ABST
    Figure CN121541459A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of unmanned aerial vehicle control, in particular to an unmanned aerial vehicle adaptive shooting method and system based on multiple sensors and AI, and the method comprises the steps: multi-sensor data collection, deep learning fusion processing, multi-modal exception processing, decision-making layer path planning reinforcement, adaptive shooting precision improvement, and a re-planning mechanism. Compared with the defects that environmental perception is incomplete and the shooting quality is suddenly reduced under complex illumination or severe weather due to dependence on a single sensor in the prior art, according to the scheme of the invention, a multi-mode sensor array (laser radar / millimeter wave radar / illumination sensor and the like) and Transform-GNN deep learning fusion architecture is adopted; joint modeling of obstacle positions, dynamic tracks and illumination distribution is realized; therefore, the dimensionality and precision of environmental perception are remarkably improved, and the unmanned aerial vehicle can still output clear and stable high-quality pictures under extreme conditions of strong light, rain and fog and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of drone control technology, and in particular to a drone adaptive shooting method and system based on multiple sensors and AI. Background Technology

[0002] Drone photography technology has become a key tool in fields such as aerial photography, surveying, and inspection, efficiently acquiring image and video data from an aerial perspective. Current mainstream solutions rely on preset fixed flight paths and a single sensor (such as a visible light camera) for operation, which can meet basic needs in static scenarios.

[0003] However, existing technologies have significant limitations: fixed-route flights cannot avoid sudden obstacles (such as birds and kites) in real time, which can easily lead to collisions; the perception capability of a single sensor drops sharply in complex environments such as strong light and heavy fog, resulting in blurry images or inaccurate exposure; at the same time, the lack of intelligent parameter adjustment mechanisms and reliance on manual intervention result in delayed responses, low shooting efficiency and easy omission of key areas when facing large-scale scenes.

[0004] To address the aforementioned shortcomings, this invention proposes an adaptive shooting scheme based on multi-sensor fusion and intelligent path planning: It fuses multi-source data such as LiDAR and millimeter-wave radar using a Transformer-GNN deep learning architecture to construct a dynamic environmental model with confidence; and employs RRT (Real-Time Tracking)... A layered strategy combining global planning and DWA (Dynamic Object Assist) for local obstacle avoidance enables real-time path optimization. Combined with lightweight CNN image quality assessment and semantic recognition, it automatically switches shooting modes and adjusts parameters accordingly. This overcomes technical bottlenecks in environmental perception, dynamic obstacle avoidance, image quality control, and energy efficiency management, comprehensively improving the safety, adaptability, and operational efficiency of drones in complex scenarios. Summary of the Invention

[0005] To overcome the problems mentioned in the background art, the present invention proposes an adaptive shooting method and system for drones based on multiple sensors and AI.

[0006] The technical solution of this invention is: an adaptive shooting method for drones based on multi-sensor and AI, comprising the following steps: S11: Multi-sensor data acquisition, which collects environmental data in real time through a multi-modal sensor array; S12: Deep learning fusion processing, using Transformer layers to process cross-sensor time series data, establishing temporal correlations, and constructing a dynamic environment map through graph neural networks, ultimately outputting a 3D environment model with confidence. S13: Multimodal anomaly handling, real-time monitoring of sensor data reliability, triggering multi-level response mechanisms; S14: Enhanced path planning for decision-making levels, through RRT A layered strategy of global planning and DWA local obstacle avoidance is used to optimize the trajectory and range by combining a cost function that includes battery capacity and wind resistance. S15: Improved adaptive shooting accuracy, based on lightweight CNN to evaluate image quality in real time and adjust parameters accordingly, and automatically switch shooting modes by semantically recognizing target categories; S16: Replanning mechanism, which sets four thresholds to trigger second-level dynamic path adjustment: obstacle intrusion, target speeding, sudden change in lighting, and low battery.

[0007] As a preferred embodiment, when using Transformer layers to process cross-sensor time-series data, establishing temporal correlations, and constructing a dynamic environmental atlas through graph neural networks to ultimately output a confidence-based 3D environmental model, the specific steps include: S21: Spatiotemporal data preprocessing, timestamp alignment and coordinate system unification of raw data from multiple sensors, generating a synchronized time-series data stream; S22: Transformer temporal coding, which inputs synchronous data into the Transformer coding layer, extracts cross-sensor temporal correlation features through a multi-head attention mechanism, and outputs a fused temporal feature vector; S23: Graph structure modeling, using each obstacle and landmark in the environment as a node, injecting attribute data, calculating the relative distance and velocity direction angle between nodes, and generating weighted directed edges; S24: Graph Neural Network Inference. The dynamic graph is input into the message passing layer of the GNN. The features are updated by aggregating the information of neighboring nodes. After three iterations, the node state containing spatial topological relationships is obtained. S25: Trajectory prediction and confidence generation, decode the updated node features, predict the movement trajectory of key obstacles in the next 2 seconds, and calculate the model confidence based on feature reconstruction error; S26: Output of 3D environment model, which integrates Transformer temporal features and GNN spatial features to generate a 3D environment model with confidence labels.

[0008] Preferably, when real-time monitoring of sensor data reliability triggers a multi-level response mechanism, the specific components include: S31: Real-time reliability monitoring, dynamically calculating the reliability score of each sensor through a quantitative evaluation model, with dedicated criteria designed for different sensors; S32: Multi-level response mechanism is triggered by dynamically calculating the confidence score of each sensor according to the quantitative evaluation model. The multi-level response mechanism includes main sensor switching, data weighted fusion, and abnormal alarm and tag encapsulation. S33: Real-time performance guarantee mechanism, using hardware acceleration scheme to control end-to-end latency: FPGA handles reliability calculation, preset switching path table realizes hot backup switching, GPU accelerates fusion computing, and combines 5G dedicated channel transmission.

[0009] As a preferred option, when designing specific criteria for different sensors, the criteria specifically include: A11: For lidar, the confidence level is calculated based on the point cloud density reduction rate; A12: For cameras, confidence is calculated using image blur and signal-to-noise ratio; A13: For millimeter-wave radar, confidence data is calculated by detecting multipath effect errors. A14: For other sensors, if three consecutive frames of data exceed the physical constraints, the sensor is directly deemed to have failed.

[0010] As a preferred option, the specific rules for the multi-level response mechanism are as follows: A21: Master Sensor Switching: When the confidence level of a sensor is less than or equal to 0.3, the sensor is judged to be faulty. The master dependency source is switched according to the preset priority, and coordinate transformation compensation is performed. A22: Data weighting fusion dynamically assigns weights to sensors with confidence levels greater than 0.3 and less than 0.6, according to the formula: The output sensor fusion data, among which, Let be the measurement value of the i-th sensor. The dynamic weight of the i-th sensor. This is the final output fusion value; A23: Abnormal alarm and tag encapsulation. For sensors with a confidence level greater than 0.6 and less than 0.95, encapsulate a JSON message containing a timestamp, anomaly code, confidence level, and backup sensor information, and send it to the ground terminal.

[0011] As a preferred approach, when enhancing the decision-making level path planning, the specific steps include: S41: Global path planning, based on the 3D model of the target area, uses the RRT algorithm to generate the initial path: a node tree is constructed by spatial random sampling, and when a new node falls within 5 meters of a key point, it is marked as a coverage. The path length and safety distance cost function are combined for iterative optimization, and finally the global route is output after smoothing by B-spline curve. S42: Local dynamic adjustment, detecting sudden obstacles every 15ms, triggering dynamic window method real-time planning: calculating the feasible speed space under the current speed and steering constraints, selecting the optimal direction through a cost function, and outputting response commands. The cost function is: ; S43: Energy consumption optimization mechanism, the path cost function integrates four types of parameters: path length, headwind resistance, altitude change and battery capacity, and dynamically adjusts the weights; S44: Dual planning layer collaborative control, based on the global path, with local obstacle avoidance dynamically covering waypoints: DWA adjusts the path deviation to no more than 15%, and energy consumption constraints are immediately injected during replanning.

[0012] As a preferred option, the path cost function in the energy consumption optimization mechanism is specifically as follows: ; in, For path length, This is the headwind coefficient. The coefficient for height variation. This represents the remaining battery capacity.

[0013] As a preferred embodiment, when evaluating image quality in real time based on a lightweight CNN and adjusting parameters accordingly, and simultaneously switching shooting modes automatically through semantic recognition of target categories, the specific features include: S51: Physical closed-loop control, which adjusts shooting parameters based on real-time ambient brightness data from the light sensor and IMU attitude feedback; S52: Intelligent closed-loop image optimization, which analyzes the image at 30fps through a lightweight CNN model deployed on the drone, outputs a sharpness score and the proportion of overexposed areas, and simultaneously performs semantic recognition to identify the content in the image and drive the shooting mode to change according to the content of the image. S53: Dual closed-loop collaborative strategy. Sudden targets are prioritized by intelligent closed-loop image optimization, while gradual environmental changes are dominated by physical closed-loop control. When the confidence level of the lightweight CNN model deployed on the drone is less than 0.6, it automatically degrades to physical closed-loop control.

[0014] As a preferred option, the replanning mechanism specifically includes: S61: Threshold monitoring, real-time detection of four types of environmental changes, including obstacle intrusion radius, key target movement speed, light intensity changes, and battery remaining. S62: Replanning trigger. When triggered by any threshold, obstacle and target movement classes call the DWA algorithm to generate obstacle avoidance paths, while lighting and battery classes are linked with the shooting module to adjust parameters. S63: DWA local planning, based on the current speed and obstacle position, calculates the safe window for steering angle and acceleration, selects the optimal trajectory with a deviation of less than 12% from the original path, and outputs heading and speed commands; S64: Global strategy adjustment, automatically adjusts ISO and shutter speed when sudden changes in light exceed the preset limit, shortens the total flight range by 40% and shuts down non-core sensors when the battery level is less than or equal to 15%; S65: Multi-module linkage, path replanning and synchronous compensation of gimbal angle, automatic switching to wide-angle lens when encountering obstacles and climbing; emergency hovering and sound and light alarm when DWA fails, forced return to home when battery is less than or equal to 10%.

[0015] An adaptive drone shooting system based on multi-sensor and AI includes: The intelligent sensing module is used to collect environmental data through a multi-modal sensor array, and outputs a 3D environmental model with confidence using a Transformer-GNN fusion architecture, and monitors sensor anomalies to trigger multi-level responses in real time. Decision planning module, used for RRT-based The system generates an initial path through global planning, and combines it with the DWA local obstacle avoidance algorithm to achieve dynamic replanning. It also incorporates the cost function of battery remaining capacity and wind resistance to optimize the trajectory and range. The adaptive shooting module is used to evaluate image quality in real time based on a lightweight CNN and adjust parameters accordingly. It automatically switches shooting modes through semantic recognition and uses dual closed-loop control to deal with unexpected scenarios. The collaborative communication module is responsible for real-time data transmission between the UAV and the ground terminal under the 5G link, synchronously receiving user commands and feeding back system status.

[0016] The beneficial effects of this invention are: 1. Compared to existing technologies that rely on a single sensor (such as a camera), resulting in incomplete environmental perception and a sharp drop in image quality under complex lighting or severe weather conditions, this solution utilizes a multi-modal sensor array (LiDAR / millimeter-wave radar / light sensor, etc.) and a Transformer-GNN deep learning fusion architecture to achieve joint modeling of obstacle positions, dynamic trajectories, and light distribution. This significantly improves the dimensionality and accuracy of environmental perception, enabling drones to output clear, stable, and high-quality images even under extreme conditions such as strong light, rain, and fog. 2. Compared to existing technologies that use fixed flight paths and cannot dynamically avoid collisions caused by sudden obstacles (such as birds or kites), this solution innovatively designs a layered path planning strategy: RRT. The algorithm generates a global safe flight path, and the DWA dynamic window method achieves millisecond-level real-time obstacle avoidance. This mechanism gives the UAV the ability to autonomously adjust its path during flight, effectively eliminating safety hazards caused by sudden obstacles and greatly improving operational safety and mission continuity. 3. Compared to existing technologies that rely on manual adjustment of shooting parameters, resulting in delayed response and inaccurate composition, this solution constructs a physical-intelligent dual closed-loop control system: the physical closed loop compensates for attitude and exposure parameters in real time through IMU and light sensor, while the intelligent closed loop uses lightweight CNN to evaluate image quality and automatically switches shooting modes (such as people tracking / building panoramas) based on semantic recognition; thereby achieving precise matching of shooting parameters with scene requirements and completely solving core pain points such as overexposure, blurring, and target loss. 4. Compared with the shortcomings of existing technologies, which lack effective fault tolerance mechanisms when facing sensor failures (such as rain and fog interfering with lidar), causing the perception system to be paralyzed, this solution establishes a multi-level anomaly response mechanism: triggering hot switching of the main sensor based on dynamic reliability assessment, weighted fusion of multi-source data, and real-time alarm; this design ensures that the system can still maintain high-precision environmental perception when a single sensor fails, greatly enhancing robustness and reliability under extreme conditions. 5. In contrast to the shortcomings of existing technologies that ignore energy consumption factors and thus limit the range, this solution incorporates the reciprocal of the remaining battery capacity and the wind resistance coefficient into the path cost function to dynamically optimize the trajectory planning (such as reducing speed by 20% in headwinds and shortening the range by 30% when the battery is low). This mechanism enables the drone to actively adapt to weather conditions and energy status, significantly extending the duration of a single operation and providing sustainable power for large-scale complex scene shooting missions. Attached Figure Description

[0017] Figure 1 The diagram shown is a flowchart of the drone adaptive shooting method based on multiple sensors and AI of the present invention. Figure 2 The diagram shown is a schematic representation of the structure of the drone adaptive shooting system based on multiple sensors and AI according to the present invention. Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0019] Please see Figures 1-2 This invention provides an embodiment: an adaptive shooting method for drones based on multiple sensors and AI, comprising the following steps: S11: Multi-sensor data acquisition, which collects environmental data in real time through a multi-modal sensor array; S12: Deep learning fusion processing, using Transformer layers to process cross-sensor time series data, establishing temporal correlations, and constructing a dynamic environment map through graph neural networks, ultimately outputting a 3D environment model with confidence. S13: Multimodal anomaly handling, real-time monitoring of sensor data reliability, triggering multi-level response mechanisms; S14: Enhanced path planning for decision-making levels, through RRT A layered strategy of global planning and DWA local obstacle avoidance is used to optimize the trajectory and range by combining a cost function that includes battery capacity and wind resistance. S15: Improved adaptive shooting accuracy, based on lightweight CNN to evaluate image quality in real time and adjust parameters accordingly, and automatically switch shooting modes by semantically recognizing target categories; S16: Replanning mechanism, which sets four thresholds to trigger second-level dynamic path adjustment: obstacle intrusion, target speeding, sudden change in lighting, and low battery.

[0020] In this embodiment, the present invention achieves full-dimensional perception of complex environments and dynamic obstacle trajectory prediction through multi-modal sensor collaborative acquisition and deep learning fusion processing (Transformer-GNN architecture); it improves flight path safety and endurance efficiency by combining hierarchical path planning (RRT global + DWA local) and energy consumption optimization cost function; and it simultaneously utilizes lightweight CNN dual closed-loop control to achieve image quality self-optimization and intelligent switching of shooting modes, and sets four types of mutation thresholds to trigger second-level replanning. This significantly enhances the shooting quality, operational safety, and energy efficiency management of UAVs in dynamic environments.

[0021] As a preferred embodiment, when using Transformer layers to process cross-sensor time-series data, establishing temporal correlations, and constructing a dynamic environmental atlas through graph neural networks to ultimately output a confidence-based 3D environmental model, the specific steps include: S21: Spatiotemporal data preprocessing, timestamp alignment and coordinate system unification of raw data from multiple sensors, generating a synchronized time-series data stream; S22: Transformer temporal coding, which inputs synchronous data into the Transformer coding layer, extracts cross-sensor temporal correlation features through a multi-head attention mechanism, and outputs a fused temporal feature vector; S23: Graph structure modeling, using each obstacle and landmark in the environment as a node, injecting attribute data, calculating the relative distance and velocity direction angle between nodes, and generating weighted directed edges; S24: Graph Neural Network Inference. The dynamic graph is input into the message passing layer of the GNN. The features are updated by aggregating the information of neighboring nodes. After three iterations, the node state containing spatial topological relationships is obtained. S25: Trajectory prediction and confidence generation, decode the updated node features, predict the movement trajectory of key obstacles in the next 2 seconds, and calculate the model confidence based on feature reconstruction error; S26: Output of 3D environment model, which integrates Transformer temporal features and GNN spatial features to generate a 3D environment model with confidence labels.

[0022] In this embodiment, the present invention eliminates multi-sensor spatiotemporal bias through spatiotemporal data preprocessing, captures cross-modal temporal correlation features using Transformer layers, and analyzes the spatial topological relationships of obstacles using a GNN message passing mechanism. Through iterative node feature updates and trajectory prediction decoding, a 3D environment model with confidence labels is generated. This significantly improves the accuracy and reliability of environment modeling, providing highly reliable input for path planning.

[0023] Preferably, when real-time monitoring of sensor data reliability triggers a multi-level response mechanism, the specific components include: S31: Real-time reliability monitoring, dynamically calculating the reliability score of each sensor through a quantitative evaluation model, with dedicated criteria designed for different sensors; S32: Multi-level response mechanism is triggered by dynamically calculating the confidence score of each sensor according to the quantitative evaluation model. The multi-level response mechanism includes main sensor switching, data weighted fusion, and abnormal alarm and tag encapsulation. S33: Real-time performance guarantee mechanism, using hardware acceleration scheme to control end-to-end latency: FPGA handles reliability calculation, preset switching path table realizes hot backup switching, GPU accelerates fusion computing, and combines 5G dedicated channel transmission.

[0024] In this embodiment, the invention designs sensor-specific criteria to quantify reliability (such as lidar point cloud density and camera signal-to-noise ratio), triggering a three-level response mechanism of main sensor switching, weighted fusion, and anomaly alarm. Real-time performance across the entire link is ensured through FPGA hardware acceleration of reliability calculation (≤10ms), GPU fusion computation (≤15ms), and 5G low-latency transmission (≤100ms). This achieves robust perception performance (92% reduction in sensor failure rate) and immediate response under extreme environments.

[0025] As a preferred option, when designing specific criteria for different sensors, the criteria specifically include: A11: For lidar, the confidence level is calculated based on the point cloud density reduction rate; A12: For cameras, confidence is calculated using image blur and signal-to-noise ratio; A13: For millimeter-wave radar, confidence data is calculated by detecting multipath effect errors. A14: For other sensors, if three consecutive frames of data exceed the physical constraints, the sensor is directly deemed to have failed.

[0026] In this embodiment, the present invention uses specific quantitative indicators such as point cloud density reduction rate for lidar, ambiguity and signal-to-noise ratio for cameras, and multipath effect error for millimeter-wave radar to calculate confidence levels. Other sensors are directly judged as faulty through physical constraints (such as IMU angular velocity thresholds). This ensures the scene adaptability and scientific rigor of anomaly detection.

[0027] As a preferred option, the specific rules for the multi-level response mechanism are as follows: A21: Master Sensor Switching: When the confidence level of a sensor is less than or equal to 0.3, the sensor is judged to be faulty. The master dependency source is switched according to the preset priority, and coordinate transformation compensation is performed. A22: Data weighting fusion dynamically assigns weights to sensors with confidence levels greater than 0.3 and less than 0.6, according to the formula: The output sensor fusion data, among which, Let be the measurement value of the i-th sensor. The dynamic weight of the i-th sensor. This is the final output fusion value; A23: Abnormal alarm and tag encapsulation. For sensors with a confidence level greater than 0.6 and less than 0.95, encapsulate a JSON message containing a timestamp, anomaly code, confidence level, and backup sensor information, and send it to the ground terminal.

[0028] In this embodiment, the present invention triggers differentiated responses based on confidence thresholds: ≤0.3, the primary sensor is switched and coordinate deviation is compensated (e.g., correcting a 0.2m ranging error when millimeter-wave radar replaces lidar); 0.3-0.6, dynamic weighted fusion is performed (e.g., lidar weight W1=0.6); >0.6, a JSON alarm message is encapsulated (including anomaly code and backup sensor information). This achieves graded fault handling and optimal resource allocation.

[0029] As a preferred approach, when enhancing the decision-making level path planning, the specific steps include: S41: Global path planning, based on the 3D model of the target area, uses the RRT algorithm to generate the initial path: a node tree is constructed by spatial random sampling, and when a new node falls within 5 meters of a key point, it is marked as a coverage. The path length and safety distance cost function are combined for iterative optimization, and finally the global route is output after smoothing by B-spline curve. S42: Local dynamic adjustment, detecting sudden obstacles every 15ms, triggering dynamic window method real-time planning: calculating the feasible speed space under the current speed and steering constraints, selecting the optimal direction through a cost function, and outputting response commands. The cost function is: ; S43: Energy consumption optimization mechanism, the path cost function integrates four types of parameters: path length, headwind resistance, altitude change and battery capacity, and dynamically adjusts the weights; S44: Dual planning layer collaborative control, based on the global path, with local obstacle avoidance dynamically covering waypoints: DWA adjusts the path deviation to no more than 15%, and energy consumption constraints are immediately injected during replanning.

[0030] In this embodiment, the present invention uses the RRT algorithm to generate a global path with a coverage of ≥98% (key points are marked within 5m). Combined with DWA local obstacle avoidance, it outputs the optimal heading command within 15ms (the cost function balances path deviation, obstacle distance, and target proximity). Through dual-planning layer collaborative control (local deviation <15%) and real-time energy consumption constraint injection, path efficiency and dynamic safety are improved.

[0031] As a preferred option, the path cost function in the energy consumption optimization mechanism is specifically as follows: ; in, For path length, This is the headwind coefficient. The coefficient for height variation. This represents the remaining battery capacity.

[0032] In this embodiment, the present invention innovatively introduces the reciprocal of battery remaining capacity and wind resistance coefficient into the path cost function. When the headwind is >5m / s, the weight of the wind resistance coefficient is increased to 0.3 and the speed is reduced by 20%. When the battery level is <20%, the weight of altitude change is increased to 0.6 and the flight range is shortened by 30%. This significantly optimizes the energy adaptive capability.

[0033] As a preferred embodiment, when evaluating image quality in real time based on a lightweight CNN and adjusting parameters accordingly, and simultaneously switching shooting modes automatically through semantic recognition of target categories, the specific features include: S51: Physical closed-loop control, which adjusts shooting parameters based on real-time ambient brightness data from the light sensor and IMU attitude feedback; S52: Intelligent closed-loop image optimization, which analyzes the image at 30fps through a lightweight CNN model deployed on the drone, outputs a sharpness score and the proportion of overexposed areas, and simultaneously performs semantic recognition to identify the content in the image and drive the shooting mode to change according to the content of the image. S53: Dual closed-loop collaborative strategy. Sudden targets are prioritized by intelligent closed-loop image optimization, while gradual environmental changes are dominated by physical closed-loop control. When the confidence level of the lightweight CNN model deployed on the drone is less than 0.6, it automatically degrades to physical closed-loop control.

[0034] In this embodiment, the present invention ensures basic image quality through a physical closed loop (lighting / IMU linkage parameter tuning), and achieves content-driven optimization through an intelligent closed loop (lightweight CNN evaluation of sharpness / overexposure + semantic recognition for mode switching). In the dual-loop collaboration, sudden targets are prioritized by the intelligent closed loop (switching to portrait mode within 0.2s), and when the CNN confidence score is <0.6, it degrades to physical control. This achieves a dual breakthrough in shooting accuracy and system fault tolerance.

[0035] As a preferred option, the replanning mechanism specifically includes: S61: Threshold monitoring, real-time detection of four types of environmental changes, including obstacle intrusion radius, key target movement speed, light intensity changes, and battery remaining. S62: Replanning trigger. When triggered by any threshold, obstacle and target movement classes call the DWA algorithm to generate obstacle avoidance paths, while lighting and battery classes are linked with the shooting module to adjust parameters. S63: DWA local planning, based on the current speed and obstacle position, calculates the safe window for steering angle and acceleration, selects the optimal trajectory with a deviation of less than 12% from the original path, and outputs heading and speed commands; S64: Global strategy adjustment, automatically adjusts ISO and shutter speed when sudden changes in light exceed the preset limit, shortens the total flight range by 40% and shuts down non-core sensors when the battery level is less than or equal to 15%; S65: Multi-module linkage, path replanning and synchronous compensation of gimbal angle, automatic switching to wide-angle lens when encountering obstacles and climbing; emergency hovering and sound and light alarm when DWA fails, forced return to home when battery is less than or equal to 10%.

[0036] In this embodiment, the present invention sets four thresholds (obstacle intrusion 5m, target movement >2m / s, illumination change ±30%, battery level ≤15%) to trigger replanning: for obstacles / targets, DWA generates an obstacle avoidance path within 0.1s (deviation <12%), and for illumination / battery levels, linkage parameters are adjusted (e.g., ISO adaptive ±400); simultaneously, gimbal angle compensation keeps the target centered (±5 pixels), and in case of DWA failure, emergency hovering + audible and visual alarm. This ensures mission continuity and failure safety.

[0037] An adaptive drone shooting system based on multi-sensor and AI includes: The intelligent sensing module is used to collect environmental data through a multi-modal sensor array, and outputs a 3D environmental model with confidence using a Transformer-GNN fusion architecture, and monitors sensor anomalies to trigger multi-level responses in real time. Decision planning module, used for RRT-based The system generates an initial path through global planning, and combines it with the DWA local obstacle avoidance algorithm to achieve dynamic replanning. It also incorporates the cost function of battery remaining capacity and wind resistance to optimize the trajectory and range. The adaptive shooting module is used to evaluate image quality in real time based on a lightweight CNN and adjust parameters accordingly. It automatically switches shooting modes through semantic recognition and uses dual closed-loop control to deal with unexpected scenarios. The collaborative communication module is responsible for real-time data transmission between the UAV and the ground terminal under the 5G link, synchronously receiving user commands and feeding back system status.

[0038] In this embodiment, the system of the present invention achieves a closed-loop perception-decision-execution process through a four-dimensional collaborative architecture consisting of an intelligent perception module (Transformer-GNN fusion + anomaly response), a decision planning module (RRT-DWA hierarchical optimization), an adaptive shooting module (CNN dual closed-loop control), and a collaborative communication module (5G end-to-end ≤100ms). This significantly improves the intelligence level and commercialization efficiency of UAV operations in complex scenarios.

[0039] Example 1: Panoramic Shooting Task of Ancient Building Complex Mission Background: The drone needs to conduct high-precision panoramic photography of a complex Song Dynasty architectural complex. The site is affected by tourist traffic, tree obstruction, and sudden kite interference.

[0040] Specific procedures: 1. Multi-sensor fusion perception LiDAR scans the three-dimensional structure of the eaves and brackets, millimeter-wave radar penetrates leaves to monitor the location of tourists, and light sensors capture the backlit areas of the building (brightness of only 800 lux). Transformer layer fuses time series data (5Hz refresh rate), GNN constructs dynamic environment graph: using roof ridge beasts as key nodes, predicts kite flight trajectory (confidence level 0.87).

[0041] 2. Hierarchical path planning RRT Generate a 50-meter-high aerial loop path (covering 98% of the building complex), calculate the headwind coefficient (wind speed 6m / s) using a cost function, and shorten the flight segment by 15%. DWA detects an approaching kite in real time (4.2m away) and plans a climb path within 80ms (9% deviation from the original flight path).

[0042] 3. Adaptive shooting control Physical closed loop: The backlit area automatically increases the ISO to 800+ and extends the exposure to 1 / 60s; Intelligent closed loop: Lightweight CNN identifies overexposed eaves (accounting for 18%), triggering the ISO step down to 400; the semantic module captures tourists' faces and switches to portrait mode (f / 1.8 blurring the background).

[0043] 4. Anomaly Response and Replanning LiDAR affected by tree shadows (confidence level 0.25): Switch millimeter-wave radar to the main sensor and compensate for 0.3m coordinate deviation; Battery level dropped to 18%: LiDAR was turned off, reducing the remaining range by 30%, but the core building was still photographed.

[0044] Results: Output panoramic images without overexposure / out-of-focus, kite avoidance time is 0.12 seconds, and battery life in headwind scenes is improved by 25%.

[0045] Example 2: High-voltage transmission line inspection task Task Background: Inspecting high-voltage power lines in mountainous areas during gale-force winds of level 8, requiring identification of insulator damage and avoidance of attacks by flocks of migratory birds.

[0046] Specific procedures: 1. Multimodal perception and fusion Millimeter-wave radar can penetrate rain and fog to monitor the location of transmission lines (accuracy ±0.2m), cameras can capture insulator details, and IMUs can report wind speed disturbances (9m / s). GNN constructs a power transmission line topology graph: nodes are cable joints, and edges contain tension parameters; Transformer predicts the movement trend of migratory bird flocks (confidence level 0.91).

[0047] 2. Energy consumption optimization path planning RRT Generate a flight path close to the cable (1.5m away), and inject wind resistance (weight 0.3) into the cost function to automatically reduce speed by 25%; DWA detects a diving falcon (distance 3.8m) and plans a lateral movement path within 70ms (acceleration 4m / s²). 2 ).

[0048] 3. Special shooting mode activated Physical closed loop: 200Hz gimbal stabilization compensation under strong winds, ISO locked at 400 to ensure details in dark areas; Intelligent closed loop: CNN identifies insulator cracks (sharpness score 0.92), semantic module switches to macro mode (focal length 80mm); bird flock detection automatically starts 10fps continuous shooting.

[0049] 4. Extreme operating condition response Heavy rain caused lidar failure (confidence level 0.18): switch to vision-millimeter wave radar fusion (weight ratio 0.4:0.6). Battery level drops sharply to 12%: Turn off panoramic shooting module and focus on key insulator locations.

[0050] Results: Insulator defect recognition rate of 98%, response delay for migratory bird avoidance of 0.08 seconds, and energy consumption reduced by 30% in strong wind scenarios.

[0051] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A drone adaptive shooting method based on multi-sensor and AI, characterized in that: Includes the following steps: S11: Multi-sensor data acquisition, which collects environmental data in real time through a multi-modal sensor array; S12: Deep learning fusion processing, using Transformer layers to process cross-sensor time series data, establishing temporal correlations, and constructing a dynamic environment map through graph neural networks, ultimately outputting a 3D environment model with confidence. S13: Multimodal anomaly handling, real-time monitoring of sensor data reliability, triggering multi-level response mechanisms; S14: Enhanced path planning for decision-making levels, through RRT A layered strategy of global planning and DWA local obstacle avoidance is used to optimize the trajectory and range by combining a cost function that includes battery capacity and wind resistance. S15: Improved adaptive shooting accuracy, based on lightweight CNN to evaluate image quality in real time and adjust parameters accordingly, and automatically switch shooting modes by semantically recognizing target categories; S16: Replanning mechanism, which sets four thresholds to trigger second-level dynamic path adjustment: obstacle intrusion, target speeding, sudden change in lighting, and low battery.

2. The drone adaptive shooting method based on multi-sensor and AI according to claim 1, characterized in that: When using Transformer layers to process cross-sensor time-series data, establishing temporal correlations, and constructing a dynamic environmental atlas through graph neural networks to ultimately output a confidence-based 3D environmental model, the specific steps include: S21: Spatiotemporal data preprocessing, timestamp alignment and coordinate system unification of raw data from multiple sensors, generating a synchronized time-series data stream; S22: Transformer temporal coding, which inputs synchronous data into the Transformer coding layer, extracts cross-sensor temporal correlation features through a multi-head attention mechanism, and outputs a fused temporal feature vector; S23: Graph structure modeling, using each obstacle and landmark in the environment as a node, injecting attribute data, calculating the relative distance and velocity direction angle between nodes, and generating weighted directed edges; S24: Graph Neural Network Inference. The dynamic graph is input into the message passing layer of the GNN. The features are updated by aggregating the information of neighboring nodes. After three iterations, the node state containing spatial topological relationships is obtained. S25: Trajectory prediction and confidence generation, decode the updated node features, predict the movement trajectory of key obstacles in the next 2 seconds, and calculate the model confidence based on feature reconstruction error; S26: Output of 3D environment model, which integrates Transformer temporal features and GNN spatial features to generate a 3D environment model with confidence labels.

3. The drone adaptive shooting method based on multi-sensor and AI according to claim 2, characterized in that: When monitoring the reliability of sensor data in real time and triggering a multi-level response mechanism, the specific measures include: S31: Real-time reliability monitoring, dynamically calculating the reliability score of each sensor through a quantitative evaluation model, with dedicated criteria designed for different sensors; S32: Multi-level response mechanism is triggered by dynamically calculating the confidence score of each sensor according to the quantitative evaluation model. The multi-level response mechanism includes main sensor switching, data weighted fusion, and abnormal alarm and tag encapsulation. S33: Real-time performance guarantee mechanism, using hardware acceleration scheme to control end-to-end latency: FPGA handles reliability calculation, preset switching path table realizes hot backup switching, GPU accelerates fusion computing, and combines 5G dedicated channel transmission.

4. The drone adaptive shooting method based on multi-sensor and AI according to claim 3, characterized in that: When designing specific criteria for different sensors, the following are included: A11: For lidar, the confidence level is calculated based on the point cloud density reduction rate; A12: For cameras, confidence is calculated using image blur and signal-to-noise ratio; A13: For millimeter-wave radar, confidence data is calculated by detecting multipath effect errors. A14: For other sensors, if three consecutive frames of data exceed the physical constraints, the sensor is directly deemed to have failed.

5. The drone adaptive shooting method based on multi-sensor and AI according to claim 4, characterized in that: The specific rules of the multi-level response mechanism are as follows: A21: Master Sensor Switching: When the confidence level of a sensor is less than or equal to 0.3, the sensor is judged to be faulty. The master dependency source is switched according to the preset priority, and coordinate transformation compensation is performed. A22: Data weighting fusion dynamically assigns weights to sensors with confidence levels greater than 0.3 and less than 0.6, according to the formula: The output sensor fusion data, among which, Let be the measurement value of the i-th sensor. The dynamic weight of the i-th sensor. This is the final output fusion value; A23: Abnormal alarm and tag encapsulation. For sensors with a confidence level greater than 0.6 and less than 0.95, encapsulate a JSON message containing a timestamp, anomaly code, confidence level, and backup sensor information, and send it to the ground terminal.

6. The drone adaptive shooting method based on multi-sensor and AI according to claim 5, characterized in that: When strengthening the decision-making path planning, the specific steps include: S41: Global path planning, based on the 3D model of the target area, uses the RRT algorithm to generate the initial path: a node tree is constructed by spatial random sampling, and when a new node falls within 5 meters of a key point, it is marked as a coverage. The path length and safety distance cost function are combined for iterative optimization, and finally the global route is output after smoothing by B-spline curve. S42: Local dynamic adjustment, detecting sudden obstacles every 15ms, triggering dynamic window method real-time planning: calculating the feasible speed space under the current speed and steering constraints, selecting the optimal direction through a cost function, and outputting response commands. The cost function is: ; S43: Energy consumption optimization mechanism, the path cost function integrates four types of parameters: path length, headwind resistance, altitude change and battery capacity, and dynamically adjusts the weights; S44: Dual planning layer collaborative control, based on the global path, with local obstacle avoidance dynamically covering waypoints: DWA adjusts the path deviation to no more than 15%, and energy consumption constraints are immediately injected during replanning.

7. The drone adaptive shooting method based on multi-sensor and AI according to claim 6, characterized in that: In the energy consumption optimization mechanism, the path cost function is specifically as follows: ; in, For path length, This is the headwind coefficient. The coefficient for height variation. This represents the remaining battery capacity.

8. The drone adaptive shooting method based on multi-sensor and AI according to claim 7, characterized in that: When evaluating image quality in real time based on a lightweight CNN and adjusting parameters accordingly, and simultaneously switching shooting modes automatically through semantic recognition of target categories, the specific steps include: S51: Physical closed-loop control, which adjusts shooting parameters based on real-time ambient brightness data from the light sensor and IMU attitude feedback; S52: Intelligent closed-loop image optimization, which analyzes the image at 30fps through a lightweight CNN model deployed on the drone, outputs a sharpness score and the proportion of overexposed areas, and simultaneously performs semantic recognition to identify the content in the image and drive the shooting mode to change according to the content of the image. S53: Dual closed-loop collaborative strategy. Sudden targets are prioritized by intelligent closed-loop image optimization, while gradual environmental changes are dominated by physical closed-loop control. When the confidence level of the lightweight CNN model deployed on the drone is less than 0.6, it automatically degrades to physical closed-loop control.

9. The drone adaptive shooting method based on multi-sensor and AI according to claim 8, characterized in that: The replanning mechanism specifically includes: S61: Threshold monitoring, real-time detection of four types of environmental changes, including obstacle intrusion radius, key target movement speed, light intensity changes, and battery remaining. S62: Replanning trigger. When triggered by any threshold, obstacle and target movement classes call the DWA algorithm to generate obstacle avoidance paths, while lighting and battery classes are linked with the shooting module to adjust parameters. S63: DWA local planning, based on the current speed and obstacle position, calculates the safe window for steering angle and acceleration, selects the optimal trajectory with a deviation of less than 12% from the original path, and outputs heading and speed commands; S64: Global strategy adjustment, automatically adjusts ISO and shutter speed when sudden changes in light exceed the preset limit, shortens the total flight range by 40% and shuts down non-core sensors when the battery level is less than or equal to 15%; S65: Multi-module linkage, path replanning and synchronous compensation of gimbal angle, automatic switching to wide-angle lens when encountering obstacles and climbing; emergency hovering and sound and light alarm when DWA fails, forced return to home when battery is less than or equal to 10%.

10. An adaptive drone shooting system based on multi-sensor and AI, characterized in that: include: The intelligent sensing module is used to collect environmental data through a multi-modal sensor array, and outputs a 3D environmental model with confidence using a Transformer-GNN fusion architecture, and monitors sensor anomalies to trigger multi-level responses in real time. Decision planning module, used for RRT-based The system generates an initial path through global planning, and combines it with the DWA local obstacle avoidance algorithm to achieve dynamic replanning. It also incorporates the cost function of battery remaining capacity and wind resistance to optimize the trajectory and range. The adaptive shooting module is used to evaluate image quality in real time based on a lightweight CNN and adjust parameters accordingly. It automatically switches shooting modes through semantic recognition and uses dual closed-loop control to deal with unexpected scenarios. The collaborative communication module is responsible for real-time data transmission between the UAV and the ground terminal under the 5G link, synchronously receiving user commands and providing feedback on system status.

Citation Information

Cited By

  • Autonomous mobile charging robot control method and system

    CN122172796A