Multimodal unmanned aerial vehicle trajectory prediction method and system based on bipolar optimization

Abstract

The invention belongs to the technical field of unmanned aerial vehicle autonomous navigation, and discloses a multi-modal unmanned aerial vehicle trajectory prediction method and system based on bipolar optimization, and the method comprises the steps: collecting and preprocessing multi-modal data, synchronously obtaining an RGB image, a depth image and unmanned aerial vehicle state information, and carrying out the normalization processing; multi-modal feature extraction and fusion, wherein RGB, depth and IMU features are fused through a deep convolutional network and a cross-modal attention mechanism; generating an environment cost map, constructing a differentiable grid map based on the depth information, and adding a navigation guide item; performing bipolar optimization trajectory prediction, predicting key points through an upper neural network, and dynamically optimizing the trajectory by combining safety and exploratory performance through a lower differentiable optimizer; and performing safety assessment and output, and outputting a track or triggering an obstacle avoidance mechanism after comprehensively assessing the collision risk and the dynamic feasibility. According to the invention, through multi-modal fusion and bipolar optimization, the sensing precision, the real-time performance and the security are significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous navigation technology for unmanned aerial vehicles (UAVs), and in particular to a multimodal UAV trajectory prediction method and system based on bipolar optimization. Background Art

[0002] In recent years, drones have experienced rapid growth in military, industrial, and civilian applications. Autonomous drone navigation, a core capability of drone technology, has been widely applied in urban inspections, logistics distribution, emergency rescue, and agricultural plant protection. Autonomous drone navigation involves multiple technical aspects, including environmental perception, trajectory planning, path optimization, and motion control. Different application scenarios and mission requirements require different technical approaches.

[0003] Traditional drone trajectory planning methods primarily rely on pre-built environmental maps or simple reactive obstacle avoidance algorithms. These methods suffer from numerous limitations when faced with complex and dynamic environments. First, a single sensor modality struggles to provide comprehensive environmental information and is easily affected by specific environmental conditions. For example, relying solely on LiDAR can lose texture information, while relying solely on RGB cameras struggles to obtain accurate depth information. Second, traditional modular designs separate perception, planning, and control into independent components, lacking end-to-end joint optimization, resulting in suboptimal overall performance. Furthermore, existing methods generally lack robust safety mechanisms and are inadequate in responding to emergencies.

[0004] With the development of deep learning technology, learning-based trajectory prediction methods have shown great potential. However, existing deep learning methods still face challenges such as the difficulty in balancing real-time performance and accuracy, and insufficient generalization capabilities. In particular, performance often degrades significantly when migrating from training environments to actual application environments.

[0005] Therefore, it is necessary to develop a UAV trajectory prediction method that can integrate multimodal perception information, achieve end-to-end optimization, have strong security guarantees and good generalization capabilities to meet the needs of autonomous navigation in complex environments. Summary of the Invention

[0006] The purpose of the present invention is to provide a multimodal UAV trajectory prediction method and system based on bipolar optimization to solve the above-mentioned technical problems.

[0007] To solve the above technical problems, the specific technical solutions of the multimodal UAV trajectory prediction method and system based on bipolar optimization of the present invention are as follows:

[0008] A multimodal UAV trajectory prediction method based on bipolar optimization includes the following steps:

[0009] Step 1: Multimodal data acquisition and preprocessing: RGB image sequences, depth image sequences, and drone status information are collected. RGB and depth images are acquired synchronously via an onboard camera, and status information is acquired via an inertial measurement unit. The RGB images are normalized, and the depth images are cropped and normalized within their effective range to construct a 10-dimensional state vector containing position, velocity, attitude, and timestamp.

[0010] Step 2: Multimodal feature extraction and fusion: Use a deep convolutional neural network to extract RGB image features and depth image features respectively, learn the temporal dependency through the temporal attention mechanism, use the cross-modal attention mechanism to fuse RGB features, depth features and IMU features, and connect the fused image features with the encoded state features.

[0011] Step 3: Environment cost map generation: Generate a 2D occupancy grid map based on the depth image and position information; use Gaussian filtering to smooth the grid map to ensure differentiability; combine obstacle information and navigation guidance to generate a comprehensive cost map.

[0012] Step 4: Bipolar Optimization Trajectory Prediction: Build an upper-layer neural network to learn the mapping from multimodal input to keypoints. The upper-layer network predicts the coordinates and collision probabilities of N 3D keypoints. Build a lower-layer differentiable optimizer to define a trajectory cost function that incorporates traversability, goal orientation, and smoothness. Use learnable interpolation weights to interpolate keypoints into a complete T-step trajectory. Optimize keypoint positions via gradient descent, achieving a dynamic balance between positive optimization (safety) and negative optimization (exploration).

[0013] Step 5: Safety Assessment and Trajectory Output: Evaluate the collision risk and dynamic feasibility of the generated trajectory; calculate the trajectory quality score, including safety, efficiency, and smoothness indicators; if the safety assessment passes, output the predicted trajectory; if there is a safety risk, trigger the emergency obstacle avoidance mechanism.

[0014] Furthermore, the step 1 includes the following steps:

[0015] System status information is defined as:

[0016] x t =[p t ,v t ,q t ,ω t ,b a ,b g ] T ∈R 16

[0017] Among them, p t =[x,y,z] T ∈R 3Indicates the three-dimensional position coordinates of the drone in the world coordinate system, v t =[v x ,v y ,v z ] T ∈R 3 Represents the three-dimensional linear velocity vector of the UAV, q t =[q w ,q x ,q y ,q z ] T ∈R 4 Quaternion representation of the drone's attitude, w t =[w x ,w y ,w z ] T ∈R 3 represents the three-dimensional angular velocity vector of the drone, b a ,b g ∈R 3 are the bias vectors of the accelerometer and gyroscope, respectively.

[0018] Furthermore, the step 2 includes the following steps:

[0019] A deep convolutional neural network is used to extract RGB image features and depth image features respectively. The RGB feature extraction network adopts a 4-layer convolution structure with convolution kernel sizes of 7×7, 5×5, 3×3, and 3×3, respectively, and a step size of 2. Each convolution layer is followed by batch normalization and ReLU activation function. The deep feature extraction network adopts the same network structure, but the number of input channels is 1. The feature map is converted into a 256-dimensional feature vector through global average pooling and a fully connected layer.

[0020] Apply the temporal attention mechanism to the extracted image sequence features to learn the temporal dependencies between features at different moments.

[0021]

[0022] where e t represents the importance score of the feature at time t, α t ∈[0,1], represents the normalized attention weight of the feature at time t in temporal fusion, satisfying the constraint T represents the total length of the timing window;

[0023] The temporal features are modeled using an LSTM network. The LSTM hidden dimension is set to 256 and the number of layers is 2. The RGB features and depth features are fused through a cross-modal attention mechanism. The weights of each modality are adaptively adjusted according to the environmental conditions. The calculation is as follows:

[0024]

[0025] in, Represents the importance scores of RGB, depth and IMU features respectively;

[0026] α rgb ,α depth ,α imu ∈[0,1] is the normalized attention weight of each modality, satisfying α rgb +α depth +α imu =1;

[0027] Finally, the fused image features are connected with the encoded state features to form a comprehensive feature representation:

[0028] f fused =α rgb f rgb +α depth f depth +α imu f imu

[0029] in is the multimodal fusion feature vector, where the typical setting of the weight coefficient is: α is the forward optimization term weight, emphasizing security and smoothness.

[0030] Furthermore, the step 3 includes the following steps:

[0031] Based on the depth image and drone position information, a pinhole camera model is used to project the depth pixels into the world coordinate system to generate a 3D point cloud. The horizontal field of view is set to 60 degrees and the vertical field of view is set to 45 degrees. The 3D point cloud is projected onto a 2D plane to construct an occupancy grid map with a resolution of 128×128. Each grid represents an area of ​​0.15625 meters by 0.15625 meters.

[0032] The cost map is smoothed using a Gaussian filter with a standard deviation of 2.0 to ensure the differentiability of the cost function and provide a safety buffer for obstacles. Navigation guidance items are added to the cost map, including target attraction potential field and boundary repulsion potential field, to guide the trajectory planning process.

[0033] Furthermore, step 4 includes the following steps:

[0034] Construct an upper-layer neural network to learn the mapping relationship from multimodal input to trajectory key points. The upper-layer network is based on the LSTM structure and contains three output branches: a key point prediction head, a collision prediction head, and a quality prediction head. The key point prediction head outputs N 3D key point coordinates, the collision prediction head outputs the collision probability, and the quality prediction head outputs the trajectory quality score.

[0035] The predicted output of trajectory key points is:

[0036]

[0037] Where N is the number of key points, is the three-dimensional coordinate of the i-th key point, is the initial hidden state of LSTM, usually initialized to a zero vector;

[0038] Construct a lower-level differentiable optimizer to interpolate discrete keypoints into continuous trajectories and optimize them. The optimizer contains a learnable interpolation weight matrix and a trajectory cost function. The cost function consists of four parts: traversability cost, goal-oriented cost, dynamic constraint, and smoothness cost. The weights of each part can be learned and adjusted.

[0039] Define the comprehensive trajectory cost function:

[0040] J total =w1J obstacle +w2J smooth +w3J goal +w4J dynamics

[0041] Where: w1 is the obstacle cost weight, the highest priority is to ensure safety, J obstacle is the obstacle cost function, w2 is the smoothness cost weight, J smooth is the smoothness cost function, w3 is the goal-oriented cost weight, J goal is the goal-oriented cost function, w4 is the dynamic constraint cost weight, J dynamics is the dynamic constraint cost function;

[0042] The optimization process handles various constraints, including maximum velocity constraints, maximum acceleration constraints, and environmental boundary constraints. A multi-resolution optimization strategy is adopted, with rapid convergence at a coarse resolution followed by fine adjustments at a fine resolution. The maximum number of iterations is set to 50, and the optimization is terminated when the cost function change is less than a threshold or the maximum number of iterations is reached. A joint loss function is designed, including trajectory prediction loss, collision prediction loss, and key point consistency loss. Through a bipolar interaction mechanism, the output of the upper-layer network is used as the initial value for the lower-layer optimization, and the results of the lower-layer optimization are used to update the parameters of the upper-layer network.

[0043] The overall objective function of bipolar optimization is defined as:

[0044] J bipolar =αJ positive +βJ negative +γJ interaction

[0045] The typical settings of the weight coefficients are: α is the weight of the positive optimization term, emphasizing safety and smoothness, β is the weight of the negative optimization term, encouraging exploration and diversity, and γ is the weight of the bipolar interaction term, maintaining dynamic balance. The specific definitions of each term are as follows:

[0046]

[0047] J interaction =λ|J positive -J negative |

[0048] Where T is the number of trajectory discretization points, λ is the balance parameter of the interaction term, represents the obstacle cost of the i-th trajectory point, represents the trajectory smoothness cost, represents the goal-oriented cost, represents the exploration reward, represents the trajectory diversity reward.

[0049] Furthermore, step 4 includes a bipolar joint training strategy, specifically:

[0050] Prepare a training dataset containing a variety of environments and flight scenarios. The dataset size is no less than 10,000 trajectories, each containing 300-500 frames of data. Use a phased training strategy: in the first phase, train the upper network separately for 10 rounds, in the second phase, train the lower optimizer separately for 20 rounds, and in the third phase, perform joint training for 70 rounds.

[0051] During the joint training phase, the upper-layer network parameters and the lower-layer optimizer parameters are alternately optimized. In each iteration, the upper-layer network parameters are first fixed, and then the lower-layer weights are optimized. The AdamW optimizer is used, with an initial learning rate of 1e-3 and a cosine annealing strategy for adjusting the learning rate. Data augmentation techniques, including random image transformations, deep noise injection, and trajectory perturbations, are applied to improve the model's generalization capabilities.

[0052] Furthermore, the step 5 includes the following specific steps:

[0053] The safety of the generated trajectory is evaluated from multiple dimensions, including minimum obstacle distance, collision probability distribution, trajectory curvature, and velocity change rate. A trajectory is considered safe and executable if its collision probability is less than 0.3 and its quality score is greater than 0.7. Otherwise, a replanning mechanism or emergency stop procedure is triggered.

[0054] The optimized trajectory is converted into control instructions that can be executed by the drone, and position control or speed control instructions are generated according to the drone's control interface. The safety status is continuously monitored during the trajectory execution process, and when sudden danger is detected, it immediately switches to emergency obstacle avoidance mode.

[0055] The present invention also discloses a multimodal UAV trajectory prediction system based on bipolar optimization, comprising:

[0056] Data acquisition module, used to collect RGB images, depth images and drone status information;

[0057] Data preprocessing module, used to normalize and convert the collected multimodal data;

[0058] Multimodal encoder module, including RGB encoder, depth encoder, state encoder and cross-modal attention mechanism;

[0059] A bipolar optimization module, consisting of an upper-layer neural network and a lower-layer differentiable optimizer;

[0060] The cost map generation module is used to generate the environment cost map based on the depth information and position information; the trajectory prediction module is used to predict key points and generate complete trajectories;

[0061] The safety assessment module is used to evaluate the safety and quality of the trajectory; the real-time control module is used to generate navigation instructions based on the prediction results.

[0062] The multimodal UAV trajectory prediction method and system based on bipolar optimization of the present invention have the following advantages:

[0063] First, the perception accuracy is significantly improved: by fusing multimodal data such as RGB images, depth images and status information, the system has gained comprehensive environmental understanding capabilities, exhibited stronger robustness in complex environments, and significantly enhanced resistance to interference factors such as lighting changes and weather conditions.

[0064] Second, excellent real-time performance: The innovative bipolar optimization architecture realizes end-to-end joint optimization of perception and planning, avoiding the error accumulation problem of traditional cascade methods and significantly improving the overall performance of the system.

[0065] Third, safety is guaranteed to be reliable: The comprehensive safety assurance mechanism includes collision probability prediction, trajectory quality assessment, and real-time safety monitoring, demonstrating excellent performance in a large number of simulations and actual tests.

[0066] Fourth, strong generalization capability: Through data augmentation and bipolar optimization mechanisms, the system's performance degrades less when migrating between different environments, demonstrating excellent generalization capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is the overall architecture diagram of the system of the present invention;

[0068] Figure 2 This is a flowchart for multimodal feature extraction and fusion;

[0069] Figure 3 Detailed structural diagram for bipolar optimized architecture;

[0070] Figure 4 Deploy the architecture diagram for the system hardware;

[0071] Figure 5 This is a schematic diagram of Example 1;

[0072] Figure 6 This is a schematic diagram of Example 2. DETAILED DESCRIPTION

[0073] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a multimodal UAV trajectory prediction method and system based on bipolar optimization of the present invention in conjunction with the accompanying drawings.

[0074] like Figure 1 As shown, the multimodal UAV trajectory prediction method based on bipolar optimization of the present invention includes the following steps:

[0075] Step 1: Multimodal data acquisition and preprocessing: RGB image sequences, depth image sequences, and drone status information are collected. RGB and depth images are acquired synchronously via an onboard camera, and status information is acquired via an inertial measurement unit. The RGB images are normalized, and the depth images are cropped and normalized within their effective range to construct a 10-dimensional state vector containing position, velocity, attitude, and timestamp.

[0076] System status information is defined as:

[0077] x t =[p t ,v t ,q t ,ω t ,b a ,b g ] T ∈R 16

[0078] Among them, p t =[x,y,z] T∈R 3 Indicates the three-dimensional position coordinates of the drone in the world coordinate system, v t =[v x ,v y ,v z ] T ∈R 3 Represents the three-dimensional linear velocity vector of the UAV, q t =[q w ,q x ,q y ,q z ] T ∈R 4 Quaternion representation of the drone's attitude, w t =[w x ,w y ,w z ] T ∈R 3 represents the three-dimensional angular velocity vector of the drone, b a ,b g ∈R 3 are the bias vectors of the accelerometer and gyroscope, respectively.

[0079] Step 2: Multimodal feature extraction and fusion: Use a deep convolutional neural network to extract RGB image features and depth image features respectively, learn the temporal dependency through the temporal attention mechanism, use the cross-modal attention mechanism to fuse RGB features, depth features and IMU features, and connect the fused image features with the encoded state features.

[0080] like Figure 2 As shown in Figure 2, multimodal feature extraction and fusion include:

[0081] A deep convolutional neural network is used to extract RGB image features and depth image features. The RGB feature extraction network uses a four-layer convolutional architecture with kernel sizes of 7×7, 5×5, 3×3, and 3×3, respectively, with a stride of 2. Each convolution layer is followed by batch normalization and a ReLU activation function. The depth feature extraction network uses the same architecture, but with a single input channel. Feature maps are converted to 256-dimensional feature vectors through global average pooling and fully connected layers.

[0082] A temporal attention mechanism is applied to the extracted image sequence features to learn the temporal dependencies between features at different moments.

[0083]

[0084] where e t Represents the importance score of the feature at time t. t∈[0,1], represents the normalized attention weight of the feature at time t in temporal fusion, satisfying the constraint T represents the total length of the timing window

[0085] The temporal features are modeled using an LSTM network, with a hidden dimension of 256 and a layer number of 2. RGB features and depth features are fused through a cross-modal attention mechanism, and the weights of each modality are adaptively adjusted according to environmental conditions. The calculation is as follows:

[0086]

[0087] in, Represent the importance scores of RGB, depth and IMU features respectively.

[0088] α rgb ,α depth ,α imu ∈[0,1] is the normalized attention weight of each modality, satisfying α rgb +α depth +α imu =1.

[0089] Finally, the fused image features are connected with the encoded state features to form a comprehensive feature representation.

[0090] f fused =α rgb f rgb +α depth f depth +α imu f imu

[0091] in is the multimodal fusion feature vector. The typical setting of the weight coefficient is: α is the weight of the forward optimization term, emphasizing security and smoothness.

[0092] Step 3: Environment cost map generation: Generate a 2D occupancy grid map based on the depth image and position information; use Gaussian filtering to smooth the grid map to ensure differentiability; combine obstacle information and navigation guidance to generate a comprehensive cost map.

[0093] The environmental cost map generation method is:

[0094] Based on the depth image and drone position information, a pinhole camera model was used to project the depth pixels into the world coordinate system to generate a 3D point cloud. The horizontal field of view was set to 60 degrees and the vertical field of view was set to 45 degrees. The 3D point cloud was projected onto a 2D plane to construct an occupancy grid map with a resolution of 128×128, with each grid representing an area of ​​0.15625 meters by 0.15625 meters.

[0095] Cost values ​​are assigned to the occupancy grid, with grids occupied by obstacles assigned a value of 1.0, free space assigned a value of 0.0, and uncertain regions assigned intermediate values ​​based on occupancy probabilities. The cost map is smoothed using a Gaussian filter with a standard deviation of 2.0 to ensure the differentiability of the cost function while providing a safety buffer for obstacles. Navigation guidance items, including target attraction and boundary repulsion potential fields, are added to the cost map to guide the trajectory planning process.

[0096] Step 4: Bipolar Optimization Trajectory Prediction: Build an upper-layer neural network to learn the mapping from multimodal input to keypoints. The upper-layer network predicts the coordinates and collision probabilities of N 3D keypoints. Build a lower-layer differentiable optimizer to define a trajectory cost function that incorporates traversability, goal orientation, and smoothness. Use learnable interpolation weights to interpolate keypoints into a complete T-step trajectory. Optimize keypoint positions via gradient descent, achieving a dynamic balance between positive optimization (safety) and negative optimization (exploration).

[0097] like Figure 3 As shown, the bipolar optimized trajectory prediction includes:

[0098] A high-level neural network is constructed to learn the mapping relationship from multimodal input to trajectory keypoints. The high-level network is based on an LSTM architecture and consists of three output branches: a keypoint prediction head, a collision prediction head, and a quality prediction head. The keypoint prediction head outputs the coordinates of N 3D keypoints, the collision prediction head outputs the collision probability, and the quality prediction head outputs the trajectory quality score.

[0099] The predicted output of trajectory key points is:

[0100]

[0101] Where N is the number of key points, is the three-dimensional coordinate of the i-th key point, is the initial hidden state of LSTM, usually initialized to a zero vector.

[0102] A lower-level differentiable optimizer is constructed to interpolate and optimize discrete keypoints into continuous trajectories. The optimizer includes a learnable interpolation weight matrix and a trajectory cost function. The cost function consists of four components: a traversability cost, a goal-oriented cost, a dynamic constraint, and a smoothness cost, with each component weighted by learnable adjustments.

[0103] Define the comprehensive trajectory cost function:

[0104] J total =w1J obstacle +w2J smooth +w3J goal +w4J dynamics

[0105] Where: w1 is the obstacle cost weight, the highest priority is to ensure safety, J obstacle is the obstacle cost function, w2 is the smoothness cost weight, J smooth is the smoothness cost function, w3 is the goal-oriented cost weight, J goal is the goal-oriented cost function, w4 is the dynamic constraint cost weight, J dynamics is the dynamic constraint cost function.

[0106] The optimization process addresses various constraints, including maximum velocity, maximum acceleration, and environmental boundary constraints. A multi-resolution optimization strategy is employed, initially achieving rapid convergence at a coarse resolution, followed by fine-tuning at a finer resolution. The maximum number of iterations is set to 50, and the optimization is terminated when the cost function change falls below a threshold or when the maximum number of iterations is reached.

[0107] A joint loss function is designed, which includes trajectory prediction loss, collision prediction loss, and keypoint consistency loss. Through a bipolar interaction mechanism, the output of the upper network is used as the initial value for the lower optimization layer, and the results of the lower optimization are used to update the parameters of the upper network.

[0108] The overall objective function of bipolar optimization is defined as:

[0109] J bipolar =αJ positive +βJ negative +γJ interaction

[0110] The typical settings of the weight coefficients are: α is the weight of the positive optimization term, emphasizing safety and smoothness; β is the weight of the negative optimization term, encouraging exploration and diversity; γ is the weight of the bipolar interaction term, maintaining dynamic balance.

[0111] The specific definitions of each item are as follows:

[0112]

[0113] J interaction =λ|J positive -J negative |

[0114] Where T is the number of trajectory discretization points, λ is the balance parameter of the interaction term, represents the obstacle cost of the i-th trajectory point, represents the trajectory smoothness cost, represents the goal-oriented cost, represents the exploration reward, represents the trajectory diversity reward.

[0115] Furthermore, the bipolar combined training strategy is:

[0116] Prepare a training dataset containing a variety of environments and flight scenarios, with a minimum of 10,000 trajectories, each containing 300-500 frames of data. Use a phased training strategy: in the first phase, train the upper network separately for 10 rounds, in the second phase, train the lower optimizer separately for 20 rounds, and in the third phase, perform joint training for 70 rounds.

[0117] During the joint training phase, the upper-layer network parameters and the lower-layer optimizer parameters are alternately optimized. In each iteration, the upper-layer network parameters are first fixed, followed by the lower-layer weights. The AdamW optimizer is used, with an initial learning rate of 1e-3 and a cosine annealing strategy for adjusting the learning rate. Data augmentation techniques, including random image transformations, deep noise injection, and trajectory perturbations, are applied to improve the model's generalization capabilities.

[0118] Step 5: Safety Assessment and Trajectory Output: Evaluate the collision risk and dynamic feasibility of the generated trajectory; calculate the trajectory quality score, including safety, efficiency, and smoothness indicators; if the safety assessment passes, output the predicted trajectory; if there is a safety risk, trigger the emergency obstacle avoidance mechanism.

[0119] Safety assessment and trajectory output include:

[0120] The safety of the generated trajectory is evaluated across multiple dimensions, including minimum obstacle distance, collision probability distribution, trajectory curvature, and velocity change rate. A trajectory is considered safe and executable if its collision probability is less than 0.3 and its quality score is greater than 0.7. Otherwise, a replanning mechanism or emergency stop procedure is triggered.

[0121] The optimized trajectory is converted into control instructions that can be executed by the drone, and position control or speed control instructions are generated according to the drone's control interface. The safety status is continuously monitored during trajectory execution, and emergency obstacle avoidance mode is immediately switched when sudden danger is detected.

[0122] like Figure 4 As shown, the present invention provides a multimodal UAV trajectory prediction system based on bipolar optimization, comprising:

[0123] The data acquisition module is used to collect RGB images, depth images and drone status information; the data preprocessing module is used to normalize and convert the collected multimodal data; the multimodal encoder module contains an RGB encoder, a depth encoder, a state encoder and a cross-modal attention mechanism; the bipolar optimization module contains an upper-layer neural network and a lower-layer differentiable optimizer; the cost map generation module is used to generate an environmental cost map based on depth information and position information; the trajectory prediction module is used to predict key points and generate a complete trajectory; the safety assessment module is used to evaluate the safety and quality of the trajectory; the real-time control module is used to generate navigation instructions based on the prediction results.

[0124] Example 1

[0125] like Figure 5 As shown, this embodiment demonstrates the application of the system in an industrial warehouse scenario. Taking into account the special needs of the industrial environment, the system adopts enhanced hardware configuration and optimization strategy.

[0126] In terms of hardware, the system uses the NVIDIA Jetson Orin NX as the main computing platform, equipped with a depth camera for environmental perception. The communication system uses a 5G module as the primary link, with 4G and WiFi as backup to ensure stable data transmission.

[0127] In terms of software optimization, TensorRT is used to optimize neural network models, enabling FP16 mixed-precision inference and increasing inference speed by 2-3 times. A multi-threaded parallel processing strategy is implemented, allowing tasks such as sensor data acquisition, feature extraction, and trajectory optimization to be executed in parallel. Ring buffers and zero-copy technology are used to reduce memory usage and data transmission overhead.

[0128] System parameters have been adjusted to address the unique characteristics of warehouse environments. The costmap resolution has been increased to 256×256, providing a more detailed representation of the environment. The number of keypoints has been increased to 8, improving maneuverability in narrow corridors. The maximum flight speed has been reduced to 2 meters per second, and the safety margin has been increased to 0.5 meters.

[0129] The system boasts robust autonomous inspection capabilities, enabling precise navigation between shelves up to 12 meters high. Inspection tasks encompass a wide range of tasks, including checking cargo status, inventory counting, and identifying potential safety hazards. Leveraging advanced computer vision algorithms, the system can identify barcodes, monitor product placement, assess storage conditions, and generate detailed inspection reports.

[0130] A bipolar optimization algorithm ensures optimal inspection paths, maximizing inspection efficiency while ensuring flight safety. The system dynamically adjusts flight paths based on shelf layout and inspection tasks, achieving multi-objective collaborative optimization. In a typical inspection, the drone can complete a comprehensive inspection of four key targets within five minutes, achieving millimeter-level accuracy.

[0131] Example 2

[0132] like Figure 6 As shown, this embodiment demonstrates the system's adaptability in a variety of complex scenarios. Through parameter adjustment and function expansion, the system has been successfully applied to challenging scenarios such as outdoor inspections, nighttime flight, and dynamic obstacle avoidance.

[0133] In outdoor power line inspection applications, the system has expanded its GPS fusion capabilities, incorporating GPS information as an additional status input. A wind compensation module has been added to estimate wind speed based on IMU data and adjust control strategies. The prediction window has been extended to 5 seconds to accommodate long-distance flights.

[0134] This example specifically addresses the unique needs of power line inspections, demonstrating the system's technical adaptability and practicality in complex outdoor environments. Power line inspections, characterized by harsh operating environments, stringent safety requirements, and high detection accuracy, pose significant technical challenges to UAV systems.

[0135] The main challenges facing power line inspections include strong electromagnetic interference, complex terrain, and volatile weather conditions. The system has implemented comprehensive technical adaptability improvements to address these challenges. The GPS-fused positioning module uses satellite positioning information as an additional state input, combining it with visual inertial odometry to provide centimeter-level positioning accuracy. In mountainous or canyon environments with weak GPS signals, the system automatically switches to visual SLAM mode to ensure continuous and reliable navigation.

[0136] The wind compensation module is a key innovative component of the system. Using high-precision IMU data to estimate wind speed and direction in real time, the system predicts wind impacts on the flight trajectory and proactively compensates during trajectory planning. The dynamic constraints in the bipolar optimization algorithm specifically account for wind, ensuring the generated trajectory remains stable under realistic wind conditions.

[0137] Power line inspections often require covering tens of kilometers, placing stringent demands on the system's endurance and path planning efficiency. The system extends the prediction window to 5 seconds to accommodate the specialized demands of long-distance flight. The trajectory optimization algorithm is specifically optimized for power line alignment, automatically generating the optimal inspection path along the line.

[0138] Energy optimization is a key technology for long-distance operations. Through intelligent speed regulation and altitude optimization, the system minimizes energy consumption while meeting inspection quality requirements. By appropriately increasing flight speed in downwind sections and reducing speed and optimizing altitude in headwind sections, refined energy management extends operating time.

[0139] Professional testing of power equipment is the core functional module of the system. The system integrates multiple specialized sensors, including infrared thermal imaging, ultraviolet discharge detection, and high-resolution visible light imaging. The infrared thermal imaging module can detect abnormal heating in power equipment and promptly identify potential equipment failures. The ultraviolet discharge detection module is specifically designed to detect corona discharge in high-voltage lines, a significant safety hazard that is difficult to detect using traditional inspection methods.

[0140] Interference resistance is a special requirement for power inspections. The system utilizes differential GPS technology and inertial navigation backup to maintain accurate navigation even in environments with strong electromagnetic interference. The communication system employs frequency hopping and error correction coding to ensure reliable data transmission in complex electromagnetic environments. The sensor data fusion algorithm is specifically optimized for electromagnetic interference, effectively filtering out interference signals and extracting true environmental information.

[0141] Through these technical improvements and functional enhancements, the two implementation examples fully demonstrate the technical advantages and practical value of the multimodal UAV trajectory prediction system based on bipolar optimization in different application scenarios, providing important technical references and practical cases for the industrial development of UAV intelligent applications.

[0142] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A multimodal UAV trajectory prediction method based on bipolar optimization, characterized in that: The steps include: Step 1: Multimodal data acquisition and preprocessing: RGB image sequences, depth image sequences, and drone status information are collected. RGB and depth images are acquired synchronously via an onboard camera, and status information is acquired via an inertial measurement unit. The RGB images are normalized, and the depth images are cropped and normalized within their effective range to construct a 10-dimensional state vector containing position, velocity, attitude, and timestamp. Step 2: Multimodal feature extraction and fusion: Use a deep convolutional neural network to extract RGB image features and depth image features respectively, learn the temporal dependency through the temporal attention mechanism, use the cross-modal attention mechanism to fuse RGB features, depth features and IMU features, and connect the fused image features with the encoded state features. Step 3: Environment cost map generation: Generate a 2D occupancy grid map based on the depth image and position information; use Gaussian filtering to smooth the grid map to ensure differentiability; combine obstacle information and navigation guidance to generate a comprehensive cost map. Step 4: Bipolar Optimization Trajectory Prediction: Build an upper-layer neural network to learn the mapping from multimodal input to key points. The upper-layer network predicts the coordinates and collision probabilities of N 3D key points; a lower-layer differentiable optimizer is constructed to define a trajectory cost function that includes passability, goal orientation, and smoothness; key points are interpolated into T-step complete trajectories using learnable interpolation weights; and key point positions are optimized through gradient descent to achieve a dynamic balance between positive optimization (safety) and negative optimization (exploration). Step 5: Safety assessment and trajectory output: Evaluate the collision risk and dynamic feasibility of the generated trajectory; Calculate trajectory quality scores, including safety, efficiency, and smoothness indicators; If the safety assessment passes, the predicted trajectory is output; If there is a safety risk, the emergency obstacle avoidance mechanism will be triggered.

2. The multimodal UAV trajectory prediction method based on bipolar optimization according to claim 1 is characterized in that: The step 1 comprises the following steps: System status information is defined as: x t =[p t ,v t ,q t ,ω t ,b a ,b g ] T ∈R 16 Among them, p t =[x,y,z] T ∈R 3 Indicates the three-dimensional position coordinates of the drone in the world coordinate system, v t =[v x ,v y ,v z ] T ∈R 3 Represents the three-dimensional linear velocity vector of the UAV, q t =[q w ,q x ,q y ,q z ] T ∈R 4 Quaternion representation of the drone's attitude, w t =[w x ,w y ,w z ] T ∈R 3 represents the three-dimensional angular velocity vector of the drone, b a ,b g ∈R 3 are the bias vectors of the accelerometer and gyroscope, respectively.

3. The multimodal UAV trajectory prediction method based on bipolar optimization according to claim 1 is characterized in that: The step 2 comprises the following steps: A deep convolutional neural network is used to extract RGB image features and depth image features respectively. The RGB feature extraction network adopts a 4-layer convolution structure with convolution kernel sizes of 7×7, 5×5, 3×3, and 3×3, respectively, and a step size of 2. Each convolution layer is followed by batch normalization and ReLU activation function. The deep feature extraction network adopts the same network structure, but the number of input channels is 1. The feature map is converted into a 256-dimensional feature vector through global average pooling and a fully connected layer. Apply the temporal attention mechanism to the extracted image sequence features to learn the temporal dependencies between features at different moments. where e t represents the importance score of the feature at time t, α t ∈[0,1], represents the normalized attention weight of the feature at time t in temporal fusion, satisfying the constraint T represents the total length of the timing window; The temporal features are modeled using an LSTM network. The LSTM hidden dimension is set to 256 and the number of layers is 2. The RGB features and depth features are fused through a cross-modal attention mechanism. The weights of each modality are adaptively adjusted according to the environmental conditions. The calculation is as follows: in, Represents the importance scores of RGB, depth and IMU features respectively; α rgb ,α depth ,α imu ∈[0,1] is the normalized attention weight of each modality, satisfying α rgb +α depth +α imu =1; Finally, the fused image features are connected with the encoded state features to form a comprehensive feature representation: f fused =a rgb f rgb +a depth f depth +a imu f imu in is the multimodal fusion feature vector, where the typical setting of the weight coefficient is: α is the forward optimization term weight, emphasizing security and smoothness.

4. The multimodal UAV trajectory prediction method based on bipolar optimization according to claim 1 is characterized in that: The step 3 comprises the following steps: Based on the depth image and drone position information, a pinhole camera model is used to project the depth pixels into the world coordinate system to generate a 3D point cloud. The horizontal field of view is set to 60 degrees and the vertical field of view is set to 45 degrees. The 3D point cloud is projected onto a 2D plane to construct an occupancy grid map with a resolution of 128×128. Each grid represents an area of ​​0.15625 meters by 0.15625 meters. The cost map is smoothed using a Gaussian filter with a standard deviation of 2.0 to ensure the differentiability of the cost function and provide a safety buffer for obstacles. Navigation guidance items are added to the cost map, including target attraction potential field and boundary repulsion potential field, to guide the trajectory planning process.

5. The multimodal UAV trajectory prediction method based on bipolar optimization according to claim 1 is characterized in that: The step 4 comprises the following steps: Construct an upper-layer neural network to learn the mapping relationship from multimodal input to trajectory key points. The upper-layer network is based on the LSTM structure and contains three output branches: a key point prediction head, a collision prediction head, and a quality prediction head. The key point prediction head outputs N 3D key point coordinates, the collision prediction head outputs the collision probability, and the quality prediction head outputs the trajectory quality score. The predicted output of trajectory key points is: Where N is the number of key points, is the three-dimensional coordinate of the i-th key point, is the initial hidden state of LSTM, usually initialized to a zero vector; Construct a lower-level differentiable optimizer to interpolate discrete keypoints into continuous trajectories and optimize them. The optimizer contains a learnable interpolation weight matrix and a trajectory cost function. The cost function consists of four parts: traversability cost, goal-oriented cost, dynamic constraint, and smoothness cost. The weights of each part can be learned and adjusted. Define the comprehensive trajectory cost function: J total =w1J obstacle +w2J smooth +w3J goal +w4J dynamics Where: w1 is the obstacle cost weight, the highest priority is to ensure safety, J obstacle is the obstacle cost function, w2 is the smoothness cost weight, J smooth is the smoothness cost function, w3 is the goal-oriented cost weight, J goal is the goal-oriented cost function, w4 is the dynamic constraint cost weight, J dynamics is the dynamic constraint cost function; The optimization process handles various constraints, including maximum velocity constraints, maximum acceleration constraints, and environmental boundary constraints. A multi-resolution optimization strategy is adopted, with rapid convergence at a coarse resolution followed by fine adjustments at a fine resolution. The maximum number of iterations is set to 50, and the optimization is terminated when the cost function change is less than a threshold or the maximum number of iterations is reached. A joint loss function is designed, including trajectory prediction loss, collision prediction loss, and key point consistency loss. Through a bipolar interaction mechanism, the output of the upper-layer network is used as the initial value for the lower-layer optimization, and the results of the lower-layer optimization are used to update the parameters of the upper-layer network. The overall objective function of bipolar optimization is defined as: J bipolar =αJ positive +βJ negative +γJ interaction The typical settings of the weight coefficients are: α is the weight of the positive optimization item, emphasizing safety and smoothness, β is the weight of the negative optimization item, encouraging exploration and diversity, and γ is the weight of the bipolar interaction item, maintaining dynamic balance. The specific definitions of each item are as follows: I interaction =λ|J positive -J negative | Where T is the number of trajectory discretization points, λ is the balance parameter of the interaction term, represents the obstacle cost of the i-th trajectory point, represents the trajectory smoothness cost, represents the goal-oriented cost, represents the exploration reward, represents the trajectory diversity reward.

6. The multimodal UAV trajectory prediction method based on bipolar optimization according to claim 1 is characterized in that: Step 4 includes a bipolar combined training strategy, specifically: Prepare a training dataset containing a variety of environments and flight scenarios. The dataset size is no less than 10,000 trajectories, each containing 300-500 frames of data. Use a phased training strategy: in the first phase, train the upper network separately for 10 rounds, in the second phase, train the lower optimizer separately for 20 rounds, and in the third phase, perform joint training for 70 rounds. During the joint training phase, the upper-layer network parameters and the lower-layer optimizer parameters are alternately optimized. In each iteration, the upper-layer network parameters are first fixed, and then the lower-layer weights are optimized. The AdamW optimizer is used, with an initial learning rate of 1e-3 and a cosine annealing strategy for adjusting the learning rate. Data augmentation techniques, including random image transformations, deep noise injection, and trajectory perturbations, are applied to improve the model's generalization capabilities.

7. The multimodal UAV trajectory prediction method based on bipolar optimization according to claim 1 is characterized in that: The step 5 includes the following specific steps: The safety of the generated trajectory is evaluated from multiple dimensions, including minimum obstacle distance, collision probability distribution, trajectory curvature, and velocity change rate. A trajectory is considered safe and executable if its collision probability is less than 0.3 and its quality score is greater than 0.

7. Otherwise, a replanning mechanism or emergency stop procedure is triggered. The optimized trajectory is converted into control instructions that can be executed by the drone, and position control or speed control instructions are generated according to the drone's control interface. The safety status is continuously monitored during the trajectory execution process, and when sudden danger is detected, it immediately switches to emergency obstacle avoidance mode.

8. A multimodal UAV trajectory prediction system based on bipolar optimization, characterized in that: include: Data acquisition module, used to collect RGB images, depth images and drone status information; Data preprocessing module, used to normalize and convert the collected multimodal data; Multimodal encoder module, including RGB encoder, depth encoder, state encoder and cross-modal attention mechanism; A bipolar optimization module, consisting of an upper-layer neural network and a lower-layer differentiable optimizer; The cost map generation module is used to generate the environment cost map based on the depth information and position information; the trajectory prediction module, Used to predict key points and generate complete trajectories; Safety assessment module, used to evaluate trajectory safety and quality; The real-time control module is used to generate navigation instructions based on the prediction results.

Images