Unmanned aerial vehicle control method and system based on multi-modal perception

Abstract

The application discloses a UAV control method and system based on multi-modal perception, comprising the following steps: collecting multi-element sensing data; using a multi-modal feature fusion network to perform space-time alignment and feature extraction on the multi-element sensing data, and generating a fused environment representation vector; step S3, based on the fused environment representation vector, constructing a three-dimensional environment map containing semantic information; inputting the three-dimensional environment map and a preset flight task target into an AI decision engine based on deep reinforcement learning to generate a flight control strategy; the AI decision engine comprises a policy network and a value network, and is trained using a proximal policy optimization algorithm; the flight control strategy is converted into motor control instructions through an adaptive hybrid controller to control the flight of the UAV. The application improves the autonomous obstacle avoidance capability and task execution efficiency of the UAV in a complex environment, while reducing the consumption of computing resources, and is suitable for various application scenarios such as city inspection, disaster rescue, logistics distribution and the like.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and unmanned aerial vehicle (UAV) control technology, specifically relating to a UAV control method and system based on multimodal perception. Background Technology

[0002] In recent years, drone technology has developed rapidly and has been widely used in various fields such as aerial surveying and mapping, agricultural plant protection, logistics and distribution, power line inspection, and emergency rescue. With the continuous expansion of application scenarios, the flight environment faced by drones is becoming increasingly complex, placing higher demands on their autonomous control capabilities.

[0003] Existing drone control technologies mainly suffer from the following technical problems: First, most existing drones rely solely on monocular or binocular vision sensors for environmental perception, lacking effective fusion and utilization of multiple sensors such as lidar and infrared sensors. In scenarios with poor lighting conditions, visual obstruction, or the need for temperature detection, the reliability and accuracy of a single vision sensor decrease significantly, seriously impacting flight safety. Second, traditional drones primarily rely on preset flight paths and simple manual rules for flight decisions, lacking real-time understanding and autonomous decision-making capabilities in complex dynamic environments. When encountering sudden obstacles or environmental changes, human intervention is often required, making truly autonomous flight difficult.

[0004] Furthermore, existing PID controller parameters typically use fixed values, making dynamic adjustment impossible based on flight status and environmental changes. This leads to significant differences in control performance under varying flight attitudes, load conditions, or wind interference, making it difficult to guarantee trajectory tracking accuracy. Finally, airborne embedded platforms have limited computing resources and power consumption budgets, while deep learning models typically involve substantial computational demands. Achieving efficient AI inference under resource-constrained conditions is a key bottleneck restricting the intelligence of drones. Summary of the Invention

[0005] This invention aims to solve the technical problems in existing UAV control technology, such as limited perception capabilities, low level of decision-making intelligence, lack of adaptability in control strategies, and limited computing resources.

[0006] To achieve the above objectives, a specific embodiment of the present invention provides a UAV control method based on multimodal perception, the method comprising: Step S1: Simultaneously collect multi-sensor data by using multiple sensors deployed on the UAV airborne platform. The multiple sensors include at least a visual sensor, a lidar, and an inertial measurement unit. Step S2: Use a multimodal feature fusion network to perform spatiotemporal alignment and feature extraction on the multi-sensor data to generate a fused environment representation vector; wherein, the multimodal feature fusion network uses an attention mechanism to calculate the correlation weights between different sensor modalities and dynamically adjusts the fusion weights of each modality through a gating mechanism; Step S3: Based on the fused environment representation vector, construct a three-dimensional environment map containing semantic information; Step S4: Input the three-dimensional environment map and the preset flight mission target into the AI ​​decision engine based on deep reinforcement learning to generate a flight control strategy; the AI ​​decision engine includes a policy network and a value network, and is trained using a near-end policy optimization algorithm; Step S5: The flight control strategy is converted into motor control commands through an adaptive hybrid controller to control the flight of the UAV; wherein the adaptive hybrid controller dynamically adjusts the control parameters according to the flight status and environmental feedback.

[0007] Compared to the limitations of traditional solutions where the perception, decision-making, and control stages are relatively independent and rely on manual rules, this invention, through deep fusion and reinforcement learning architecture, enables UAVs to understand environmental semantics in real time in complex dynamic environments, autonomously generate optimal strategies, and adjust control parameters online, achieving end-to-end autonomous flight from multimodal environmental perception and intelligent decision-making to adaptive control.

[0008] In one or more embodiments of the present invention, the multimodal feature fusion network in step S2 includes: A cross-modal attention subnetwork is used to calculate the association weight matrix between different sensor modalities, enabling adaptive interaction of information between modalities. For the visual modality V and the LiDAR modality L, the association weight matrix is: A_{V→L} = softmax(Q_V · K_L^T / √d_k), where Q_V is the query vector of the visual modality, K_L is the key vector of the LiDAR modality, and d_k is the feature dimension. A self-attention encoding subnetwork is used to extract deep temporal features of data from each modality using a multi-head self-attention mechanism; The feature fusion output layer is used to map multimodal features to a unified representation space using a weighted concatenation method, and dynamically adjusts the fusion weights through a gating mechanism: w_i = σ(gate(x_i)), where σ is the Sigmoid activation function.

[0009] In this invention, the cross-modal attention subnetwork allows the visual modality to "focus" on the geometric structure information of the LiDAR, and also allows the LiDAR modality to "focus" on the textural semantic information of the vision, thereby achieving adaptive complementarity between modalities at the feature level. The self-attention encoding subnetwork further captures the temporal evolution of the environmental state, so that the fused representation not only includes the spatial information of the current moment, but also implicitly contains the trend information of dynamic environmental changes. The gated fusion output layer dynamically adjusts the contribution weight of each modality in the final fused representation according to the quality and reliability of the data. For example, it automatically reduces the fusion weight of the visual modality and increases the weight of the LiDAR and infrared modalities when the lighting conditions are poor.

[0010] In one or more embodiments of the present invention, step S3, which involves constructing a three-dimensional environment map containing semantic information, includes: A three-dimensional semantic occupancy network k is used to divide the three-dimensional space into a voxel grid. For each voxel, the probability of its occupancy and its semantic category are predicted to generate a dense three-dimensional semantic occupancy map.

[0011] By constructing a 3D semantic occupancy graph using the aforementioned 3D semantic occupancy network k, this invention can not only acquire spatial location information of obstacles in the environment but also identify the semantic categories of obstacles (such as buildings, trees, vehicles, pedestrians, etc.), providing richer environmental context information for subsequent AI decision engines. Compared to traditional pure geometric occupancy graphs, semantic occupancy graphs enable decision engines to adopt differentiated obstacle avoidance strategies based on obstacle types, such as reserving a larger safety distance for pedestrians and allowing closer passage for static buildings.

[0012] In one or more embodiments of the present invention, the policy network in step S4 uses a Transformer architecture to encode the environment state sequence, and its input includes: The current pose of the drone, including its position, velocity, and attitude angle; The local observation window in the three-dimensional environment map is centered on the UAV and within a preset radius; And the location code of the mission target.

[0013] In one or more embodiments of the present invention, the objective function of the near-end policy optimization algorithm in step S4 is: L^{CLIP}(θ) = Ê_t [min(r_t(θ) Â_t, clip(r_t(θ), 1-ε, 1+ε) Â_t)], where r_t(θ) is the policy ratio, Â_t is the estimated value of the advantage function, and ε is the clipping hyperparameter.

[0014] Through the policy optimization based on the PPO algorithm described above, the AI ​​decision engine of this invention can gradually improve the quality of flight strategies while ensuring training stability. The pruning mechanism in the PPO algorithm effectively prevents performance crashes caused by excessive policy updates, making the training process more stable and reliable. The policy network adopts a Transformer architecture, which, compared to traditional CNN or LSTM structures, can more effectively capture long-distance spatial dependencies and temporal context, and has a stronger environmental understanding ability in complex 3D environments.

[0015] In one or more embodiments of the present invention, the adaptive hybrid controller in step S5 is an adaptive PID-MPC hybrid controller, wherein: The outer loop MPC controller is responsible for generating the desired trajectory and velocity commands according to the flight control strategy. Its optimization objective is to minimize trajectory tracking error and control input energy consumption. The inner-loop adaptive PID controller is responsible for converting the speed command into a motor speed command and adjusting the PID gain parameter online using the gradient descent method: K_p(t+1) = K_p(t) - η · J / K_p, where η is the learning rate and J is the control performance loss function.

[0016] In this invention, the outer-loop MPC controller predicts and optimizes the trajectory in the future time domain based on the system dynamics model, generating a smooth expected trajectory and speed command that meets the dynamic constraints; the inner-loop adaptive PID controller directly faces the motor execution layer and adjusts the PID gain in real time through the online gradient descent method, so that the actual control response can still quickly and accurately track the upper-level command under changes in flight attitude, load, or wind disturbance.

[0017] In one or more embodiments of the present invention, a step of training the AI ​​decision engine is included before step S4, the training step including: Construct a drone flight simulation environment in a high-fidelity simulation platform; The reward function is designed as r = w_1·r_{goal} + w_2·r_{collision} + w_3·r_{smooth} + w_4·r_{energy}, where r_{goal} is the reward for approaching the mission objective, r_{collision} is the collision penalty, r_{smooth} is the reward for smooth flight, r_{energy} is the reward for energy efficiency, and w_1, w_2, w_3, and w_4 are adjustable weight coefficients that are set according to mission requirements. The reward function is used to train the policy network and the value network.

[0018] The high-fidelity simulation platform in this invention provides a rich variety of scenarios and a safe trial-and-error environment. The composite reward function design guides the policy network to simultaneously optimize multiple objectives such as task completion efficiency, flight safety, trajectory smoothness, and energy economy, so that the trained policy has good comprehensive performance in different task scenarios.

[0019] In one or more embodiments of the present invention, the training step is followed by: The trained policy network is compressed into a lightweight student network using knowledge distillation technology to reduce the computational resource consumption during inference on the airborne platform. The knowledge distillation is achieved by minimizing the KL divergence between the output action probability distributions of the teacher network and the student network.

[0020] In this invention, the original policy network (teacher network) can be fully optimized by making full use of the large-scale computing resources on the server during the training phase; during the deployment phase, a lightweight student network that has been distilled and compressed is used to significantly reduce the number of parameters and computational load during inference while maintaining policy quality, thereby meeting the real-time requirements of the airborne platform.

[0021] In one or more embodiments of the present invention, the multiple sensors achieve microsecond-level time alignment through hardware synchronization trigger signals to ensure the time consistency of multi-source data; the multiple sensors also include an infrared thermal imaging sensor.

[0022] Another aspect of the present invention provides a multimodal perception-based unmanned aerial vehicle (UAV) control system, comprising: A multimodal perception module is used to simultaneously collect multi-sensor data through multiple sensors deployed on an UAV airborne platform. The multiple sensors include at least a visual sensor, a lidar, and an inertial measurement unit. The multimodal feature fusion module is used to calculate the correlation weights between different sensor modalities using an attention mechanism, and dynamically adjust the fusion weights of each modality through a gating mechanism to generate a fusion environment representation vector. A 3D semantic map construction module is used to construct a 3D environment map containing semantic information based on the fused environment representation vector; The AI ​​decision control module has a built-in policy network and value network based on deep reinforcement learning, which are used to generate flight control strategies based on the three-dimensional environment map and preset flight mission objectives. The adaptive execution module is used to convert the flight control strategy into motor control commands through the adaptive hybrid controller, and dynamically adjust the control parameters according to environmental feedback.

[0023] This invention achieves highly reliable autonomous flight in complex dynamic environments through an end-to-end architecture of multimodal perception fusion, deep reinforcement learning decision-making, and adaptive hybrid control. Multi-sensor fusion and attention mechanisms ensure stable perception capabilities even when a single sensor fails; the PPO-trained Transformer policy network significantly improves obstacle avoidance success rate; the adaptive PID-MPC hybrid controller maintains high-precision trajectory tracking under various flight conditions; knowledge distillation and model optimization ensure that AI inference latency meets real-time control requirements while reducing power consumption to extend flight time. This invention is applicable to various scenarios such as urban environmental inspection, disaster relief, and logistics delivery. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a UAV control method based on multimodal perception according to an embodiment of the present invention; Figure 2 This is a diagram illustrating the overall architecture of the UAV AI intelligent control system based on multimodal perception, as described in this invention. Figure 3 This is a schematic diagram of the structure of the multimodal feature fusion network of the present invention; Figure 4 This is a Transformer architecture diagram of the policy network in the AI ​​decision engine of this invention; Figure 5 This is a block diagram of the control structure of the adaptive PID-MPC hybrid controller of the present invention; Figure 6 This is a schematic diagram of the composition of the UAV onboard hardware platform of the present invention; Figure 7 This is a block diagram of the module structure of the UAV control system based on multimodal perception according to the present invention. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0027] like Figure 1 As shown, this application provides a UAV control method based on multimodal perception, the method comprising: Step S1: Simultaneously collect multi-sensor data using multiple sensors deployed on the UAV airborne platform. The multiple sensors include at least a visual sensor, a lidar, and an inertial measurement unit.

[0028] In one embodiment, as shown in the appendix Figure 6 As shown, the multimodal perception module includes the following sensor components: a binocular vision camera, employing a global shutter CMOS sensor with a resolution of 1920×1080, a frame rate of 60fps, and a horizontal field of view of 80° and a vertical field of view of 60°, used for visible light image acquisition and visual odometry calculation; a solid-state lidar, employing MEMS micromirror technology, with a detection range of 0.5m~200m, a horizontal field of view of 120°, a vertical field of view of 25°, and an angular resolution of 0.1°, used for precise three-dimensional ranging; an infrared thermal imaging sensor with a resolution of 640×512, a thermal sensitivity of less than 50mK, and a wavelength range of 8~14μm, used for night flight, smoke environment penetration, and target heat source detection; and an inertial measurement unit, including a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, used for attitude estimation and motion compensation.

[0029] Each sensor is synchronously triggered via an FPGA timing controller, with a synchronization accuracy of less than 1 microsecond, ensuring time consistency of multi-source data. Sensor data is transmitted to the edge computing unit via MIPI CSI-2 and SPI interfaces.

[0030] Step S2: Use a multimodal feature fusion network to perform spatiotemporal alignment and feature extraction on the multi-sensor data to generate a fusion environment representation vector; wherein, the multimodal feature fusion network uses an attention mechanism to calculate the correlation weights between different sensor modalities and dynamically adjusts the fusion weights of each modality through a gating mechanism.

[0031] As attached Figure 3 As shown, in one embodiment, the multimodal feature fusion network includes three core components: a cross-modal attention subnetwork, a self-attention encoding subnetwork, and a feature fusion output layer.

[0032] A cross-modal attention subnetwork is used to calculate the association weight matrix between different sensor modalities, enabling adaptive interaction of information between modalities. Specifically, for the visual modality V and the LiDAR modality L, the association weight matrix is: A_{V→L} = softmax(Q_V · K_L^T / √d_k), where Q_V is the query vector of the visual modality, K_L is the key vector of the LiDAR modality, and d_k is the feature dimension. Through this attention mechanism, the visual modality can "pay attention" to the geometric structure information related to its current position in the LiDAR modality, and the LiDAR modality can also "pay attention" to the texture and semantic information in the visual modality, achieving complementary enhancement between modalities.

[0033] The self-attention encoding subnetwork employs a multi-head self-attention mechanism to extract deep temporal features from data across various modalities. By computing multiple attention heads in parallel, each attention head focuses on a different subspace of the input sequence, thereby capturing the dynamic evolution of the environmental state. For example, through temporal attention, the network can perceive the movement trends and speeds of obstacles, providing temporal contextual information for subsequent obstacle avoidance decisions.

[0034] The feature fusion output layer uses a weighted concatenation method to map multimodal features to a unified representation space. The fusion weights are dynamically adjusted through a gating mechanism: w_i = σ(gate(x_i)), where σ is the sigmoid activation function. This gating mechanism allows the network to automatically adjust the contribution ratio of each modality in the final fused representation based on the current data quality and environmental conditions of each sensor. For example, during the day with good lighting conditions, the fusion weight of the visual modality is higher; at night or in smoky environments, the fusion weights of the infrared and lidar modalities are automatically increased; and when GPS signals are weak or lost, the fusion weights of the visual odometry and IMU are increased to maintain positioning accuracy.

[0035] It should be understood that single sensors inevitably suffer from blind spots or performance degradation in complex environments. Visual sensors experience significant performance degradation under conditions of drastic changes in lighting, rain, fog, or low illumination; lidar may produce erroneous ranging in scenarios with transparent objects, specular reflections, or strongly absorbing surfaces; and IMUs experience cumulative drift during long-term operation. This embodiment utilizes attention mechanisms and gated fusion to enable the fusion network to adaptively evaluate the quality and reliability of data from each modality and dynamically allocate fusion weights. Even when some sensors fail or their performance degrades, it maintains overall perception stability, significantly improving the flight safety of UAVs in complex environments.

[0036] Step S3: Based on the fused environment representation vector, construct a three-dimensional environment map containing semantic information.

[0037] In one embodiment, a three-dimensional semantic occupancy network k is used to divide the three-dimensional space into a voxel grid, and the probability of occupancy and semantic category of each voxel are predicted to generate a dense three-dimensional semantic occupancy map.

[0038] Specifically, the 3D semantic occupancy network k takes the fused environment representation vector as input and performs dense prediction of the 3D space within a preset range (e.g., a radius of 50 meters) around the drone through a 3D convolutional encoder-decoder structure. Each voxel outputs two values: occupancy probability (a continuous value between 0 and 1) and semantic category (the probability distribution of each category is output through a Softmax classifier, and the categories may include ground, buildings, vegetation, vehicles, pedestrians, overhead cables, etc.).

[0039] Compared to traditional geometric occupancy maps, which only provide binary information about whether an obstacle is "present" or "absent," the semantic occupancy map in this embodiment provides category information about "what" an obstacle is. This enables the AI ​​decision engine to adopt differentiated obstacle avoidance strategies based on the semantic type of different obstacles: allowing a larger safety distance and a smoother avoidance trajectory for dynamic obstacles such as pedestrians; allowing a shorter passage distance for static obstacles such as buildings; and making height adjustments in advance for thin obstacles such as overhead cables.

[0040] Step S4: Input the three-dimensional environment map and the preset flight mission target into the AI ​​decision engine based on deep reinforcement learning to generate a flight control strategy; the AI ​​decision engine includes a policy network and a value network, and is trained using a near-end policy optimization algorithm.

[0041] As attached Figure 4 As shown, in one embodiment, the policy network uses a Transformer architecture to encode the environmental state sequence. The inputs to the policy network include: the current pose state of the UAV (including 3D position, 3D velocity, and quaternion attitude angles); a local observation window in the 3D semantic occupancy map centered on the UAV within a preset radius (e.g., a radius of 30 meters); and an encoded vector of the target position (mapping the target's 3D coordinates to a high-dimensional feature space through position encoding).

[0042] The Transformer encoder incorporates a multi-layered, multi-head self-attention mechanism and a feedforward network, enabling it to effectively capture long-range spatial dependencies in environmental states. For example, the attention mechanism can establish direct attention connections between the UAV's current position, the position of obstacles ahead, and the position of distant targets, thereby generating flight decisions that both avoid obstacles and move towards the target.

[0043] The value network also uses the Transformer architecture to output a value estimate of the current state, which is used to calculate the advantage function in PPO training.

[0044] In one embodiment, the policy network and the value network are trained using a proximal policy optimization algorithm, with the objective function being: L^{CLIP}(θ) = Ê_t [min(r_t(θ) Â_t, clip(r_t(θ), 1-ε, 1+ε) Â_t)] Where r_t(θ) is the ratio of the action selection probabilities of the current policy to those of the old policy, Â_t is the estimated value of the dominance function, and ε is the pruning hyperparameter, typically set to 0.2. The PPO algorithm limits the policy update magnitude through a pruning mechanism, avoiding drastic oscillations in the policy during training and ensuring the stability and convergence of training.

[0045] In terms of setting up the training environment, this embodiment constructs a high-fidelity UAV flight simulation environment in the NVIDIA Isaac Sim simulation platform, including various scenes such as urban blocks, forests, and indoor corridors, supporting physical-level dynamics simulation and sensor data simulation. The reward function is designed as r = w_1·r_{goal} + w_2·r_{collision} + w_3·r_{smooth} + w_4·r_{energy}, comprehensively considering multiple optimization objectives such as mission objective achievement, collision avoidance, flight smoothness, and energy efficiency.

[0046] For model deployment, after training, knowledge distillation is used to compress the policy network into a lightweight student network. The original Transformer policy network is used as the teacher network, and the lightweight network with fewer parameters is used as the student network. Distillation training is performed by minimizing the KL divergence of the output action probability distributions of the teacher and student networks. The distilled student network has approximately 80% fewer parameters and an inference speed increase of approximately 4 times, meeting the real-time requirements of airborne embedded platforms.

[0047] Step S5: The flight control strategy is converted into motor control commands through an adaptive hybrid controller to control the flight of the UAV; wherein the adaptive hybrid controller dynamically adjusts the control parameters according to the flight status and environmental feedback.

[0048] As attached Figure 5 As shown, in one embodiment, the adaptive hybrid controller is an adaptive PID-MPC hybrid controller.

[0049] The outer-loop MPC controller is responsible for generating the desired trajectory and velocity commands based on the flight control strategy generated by the AI ​​decision engine. Based on the UAV's dynamic model, the MPC controller predicts the state evolution over a future predicted time domain (e.g., 1 second divided into 20 time steps) within each control cycle. It obtains the optimal control input sequence by solving a constrained optimization problem. The optimization objective is to minimize trajectory tracking error and control input energy consumption, with constraints including physical limitations such as the UAV's maximum speed, maximum acceleration, and maximum attitude angle. Only the first control variable in the optimized sequence is used as the output command.

[0050] The inner-loop adaptive PID controller is responsible for converting the speed command generated by the MPC into speed commands for the four motors. Unlike traditional fixed-parameter PID controllers, the adaptive PID controller in this embodiment uses the gradient descent method to adjust the PID gain parameter online: K_p(t+1) = K_p(t) - η · J / K_p, where η is the learning rate and J is the control performance loss function (usually the sum of squares of the trajectory tracking errors). In each control cycle, the PID controller calculates the gradient of the loss function J with respect to the gain parameters K_p, K_i, and K_d based on the current error and the rate of change of the error, and updates it in small steps along the negative gradient direction, so that the PID gain can adaptively match the current flight state and environmental conditions.

[0051] It should be understood that traditional fixed-parameter PID controllers exhibit significant performance differences under various UAV flight attitudes (hovering, high-speed forward flight, sharp turns), different load conditions (empty, fully loaded), or different wind interference. The online adaptive PID parameter adjustment mechanism in this embodiment enables the controller to continuously optimize its parameters during actual flight, maintaining high trajectory tracking accuracy under different operating conditions, with position tracking error reduced to within ±5cm.

[0052] In one embodiment, as shown in the appendix Figure 2As shown in the attached figure, this invention illustrates the overall architecture of the UAV AI intelligent control system based on multimodal perception. It adopts a modular, layered design, comprising, from top to bottom: a multimodal perception layer, including a visual sensor, LiDAR, inertial measurement unit, and infrared thermal imaging sensor, with each sensor achieving microsecond-level time alignment through hardware synchronous triggering; a feature fusion layer, which performs spatiotemporal alignment and feature extraction through a cross-modal attention sub-network and a self-attention encoding sub-network, generating a fused environment representation vector via a gated fusion output layer; a map construction layer, which constructs a 3D occupancy map containing semantic information based on the fused environment representation vector using a 3D semantic occupancy network k; an AI decision layer, which incorporates a deep reinforcement learning-based policy network and value network, inputting a 3D semantic map and flight mission objectives, and outputting a flight control strategy; an adaptive control layer, employing an adaptive PID-MPC hybrid controller to convert the flight control strategy into motor speed commands, and dynamically adjusting control parameters based on flight status and environmental feedback; and a safety takeover module, which monitors sensor health status, flight status deviation, and environmental risk level in parallel, automatically switching to a safety mode when the comprehensive risk score exceeds a threshold. This invention also includes a complete abnormal operating condition identification and safety takeover mechanism. The system monitors three types of signals in parallel: sensor health status (determined by data integrity, latency jitter, and noise variance), flight status deviation (determined by position error, attitude angle error, and control saturation), and environmental risk level (determined by dynamic obstacle density, wind field disturbance intensity, and communication link quality). When the comprehensive risk score exceeds the takeover threshold, the controller automatically switches to safety mode, limiting the maximum tilt angle and maximum angular velocity, reducing the target speed limit, and increasing the obstacle safety distance. If a critical sensor failure is detected, the system initiates a multi-source redundancy replacement mechanism; if the communication link is interrupted, the system performs hovering, return to base, or safe emergency landing according to a preset strategy. When the risk score remains below the recovery threshold, the system gradually recovers to normal flight mode according to a tiered release strategy.

[0053] like Figure 7 As shown, this application provides a multimodal perception-based unmanned aerial vehicle (UAV) control system, including: The multimodal perception module is used to simultaneously collect multi-sensor data through multiple sensors deployed on the UAV's onboard platform.

[0054] The multimodal feature fusion module is used to calculate the correlation weights between different sensor modalities using an attention mechanism, and dynamically adjust the fusion weights of each modality through a gating mechanism to generate a fusion environment representation vector.

[0055] The 3D semantic map construction module is used to construct a 3D environment map containing semantic information based on the fused environment representation vector and the 3D semantic occupancy network k.

[0056] The AI ​​decision-making and control module incorporates a policy network and a value network based on deep reinforcement learning. These networks are used to generate flight control strategies based on a 3D environment map and flight mission objectives. The policy network employs a Transformer architecture, is trained using the PPO algorithm, and is deployed through knowledge distillation compression.

[0057] The adaptive execution module is used to convert flight control strategies into motor control commands through an adaptive PID-MPC hybrid controller, and dynamically adjust control parameters based on environmental feedback.

[0058] For a specific embodiment of a UAV control system based on multimodal perception, please refer to the embodiment of a UAV control method based on multimodal perception described above, which will not be repeated here. Each module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0059] This embodiment describes the flight test verification results of the present invention on a real unmanned aerial vehicle platform.

[0060] Test platform: Quadcopter drone, 450mm wheelbase, 2.1kg takeoff weight, equipped with NVIDIA Jetson OrinNX edge computing module (100 TOPS INT8 computing power), flight time approximately 35 minutes. Test results are shown in Table 1.

[0061] Table 1. Statistical Table of Test Results

[0062] Based on the technical effects of the above embodiments, this invention achieves highly reliable autonomous flight in complex dynamic environments through an end-to-end architecture of multimodal perception fusion, deep reinforcement learning decision-making, and adaptive hybrid control. Multi-sensor fusion and attention mechanisms ensure stable perception capabilities even when a single sensor fails; the PPO-trained Transformer policy network achieves a 98.7% obstacle avoidance success rate in dynamic obstacle avoidance tests, a 12.3% improvement compared to traditional methods; the adaptive PID-MPC hybrid controller maintains high-precision trajectory tracking under various flight conditions, reducing position tracking error to within ±5cm; through knowledge distillation and model optimization, AI inference latency is controlled to within 50ms, power consumption is reduced by 35%, and the drone's endurance is extended. This invention is applicable to various application scenarios such as urban environmental inspection, disaster relief, and logistics delivery. Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0066] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0067] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for controlling a UAV based on multi-modal perception, the method comprising: Includes the following steps: Step S1: Simultaneously collect multi-sensor data through multiple sensors deployed on the UAV airborne platform. The multiple sensors include at least a visual sensor, a lidar, and an inertial measurement unit. Step S2: Use a multimodal feature fusion network to perform spatiotemporal alignment and feature extraction on the multi-sensor data to generate a fused environment representation vector; wherein, the multimodal feature fusion network uses an attention mechanism to calculate the correlation weights between different sensor modalities and dynamically adjusts the fusion weights of each modality through a gating mechanism; Step S3: Based on the fused environment representation vector, construct a three-dimensional environment map containing semantic information; Step S4: Input the three-dimensional environment map and the preset flight mission target into the AI ​​decision engine based on deep reinforcement learning to generate a flight control strategy; the AI ​​decision engine includes a policy network and a value network, and is trained using a near-end policy optimization algorithm; Step S5: The flight control strategy is converted into motor control commands through an adaptive hybrid controller to control the flight of the UAV; wherein the adaptive hybrid controller dynamically adjusts the control parameters according to the flight status and environmental feedback.

2. The method of claim 1, wherein, The multimodal feature fusion network in step S2 includes: A cross-modal attention subnetwork is used to calculate the association weight matrix between different sensor modalities, enabling adaptive interaction of information between modalities. For the visual modality V and the lidar modality L, the association weight matrix is: A_{V→L} = softmax(Q_V · K_L^T / √d_k), where Q_V is the query vector of the visual modality, K_L is the key vector of the lidar modality, and d_k is the feature dimension. A self-attention encoding subnetwork is used to extract deep temporal features of data from each modality using a multi-head self-attention mechanism; The feature fusion output layer is used to map multimodal features to a unified representation space using a weighted concatenation method, and dynamically adjusts the fusion weights through a gating mechanism: w_i = σ(gate(x_i)), where σ is the Sigmoid activation function.

3. The method of claim 1, wherein, Step S3, which involves constructing a 3D environment map containing semantic information, includes: A three-dimensional semantic occupancy network k is used to divide the three-dimensional space into a voxel grid. For each voxel, the probability of its occupancy and its semantic category are predicted to generate a dense three-dimensional semantic occupancy map.

4. The method according to claim 1, characterized in that, The policy network in step S4 uses a Transformer architecture to encode the environment state sequence, and its inputs include: The current pose of the drone, including its position, velocity, and attitude angle; The local observation window in the three-dimensional environment map is centered on the UAV and within a preset radius; And the location code of the mission target.

5. The method according to claim 1, characterized in that, The objective function of the near-end policy optimization algorithm in step S4 is: L^{CLIP}(θ) = Ê_t [min(r_t(θ) Â_t, clip(r_t(θ), 1-ε, 1+ε) Â_t)], where r_t(θ) is the policy ratio, Â_t is the estimated value of the advantage function, and ε is the clipping hyperparameter.

6. The method according to claim 1, characterized in that, The adaptive hybrid controller in step S5 is an adaptive PID-MPC hybrid controller, wherein: The outer loop MPC controller is responsible for generating the desired trajectory and velocity commands according to the flight control strategy. Its optimization objective is to minimize trajectory tracking error and control input energy consumption. The inner-loop adaptive PID controller is responsible for converting the speed command into a motor speed command and adjusting the PID gain parameter online using the gradient descent method: K_p(t+1) = K_p(t) - η · J / K_p, where η is the learning rate and J is the control performance loss function.

7. The method according to claim 1, characterized in that, Before step S4, there is also a step of training the AI ​​decision engine, which includes: Construct a drone flight simulation environment in a high-fidelity simulation platform; The reward function is designed as r = w_1·r_{goal} + w_2·r_{collision} + w_3·r_{smooth} + w_4·r_{energy}, where r_{goal} is the reward for approaching the mission objective, r_{collision} is the collision penalty, r_{smooth} is the reward for smooth flight, r_{energy} is the reward for energy efficiency, and w_1, w_2, w_3, and w_4 are adjustable weight coefficients that are set according to mission requirements. The reward function is used to train the policy network and the value network.

8. The method according to claim 7, characterized in that, The training steps are followed by: The trained policy network is compressed into a lightweight student network using knowledge distillation technology to reduce the computational resource consumption during inference on the airborne platform. The knowledge distillation is achieved by minimizing the K-L divergence between the output action probability distributions of the teacher network and the student network.

9. The method according to claim 1, characterized in that, The various sensors achieve microsecond-level time alignment through hardware synchronization trigger signals to ensure the time consistency of multi-source data; the various sensors also include an infrared thermal imaging sensor.

10. A UAV control system based on multimodal perception, characterized in that, include: A multimodal perception module is used to simultaneously collect multi-sensor data through multiple sensors deployed on an UAV airborne platform. The multiple sensors include at least a visual sensor, a lidar, and an inertial measurement unit. The multimodal feature fusion module is used to calculate the correlation weights between different sensor modalities using an attention mechanism, and dynamically adjust the fusion weights of each modality through a gating mechanism to generate a fusion environment representation vector. A 3D semantic map construction module is used to construct a 3D environment map containing semantic information based on the fused environment representation vector; The AI ​​decision control module has a built-in policy network and value network based on deep reinforcement learning, which are used to generate flight control strategies based on the three-dimensional environment map and preset flight mission objectives. The adaptive execution module is used to convert the flight control strategy into motor control commands through the adaptive hybrid controller, and dynamically adjust the control parameters according to environmental feedback.

Images