An unmanned aerial vehicle trajectory decision-making method based on multi-modal visual perception

Abstract

The application discloses a kind of unmanned aerial vehicle trajectory decision-making methods based on multi-modal visual perception, it is related to multi-modal environment perception technical field, the method includes: obtaining the multi-modal perception data of target area and based on unmanned aerial vehicle trajectory decision-making model, the trajectory decision result of unmanned aerial vehicle is generated in real time;Unmanned aerial vehicle trajectory decision-making model includes the alignment module of alignment to multi-modal perception data, the space-time feature encoder of independent coding to the multi-modal perception data after alignment, the information coordination module of weighted fusion to multi-modal feature, the collaborative understanding module of generating environment semantic map, the real-time supervisor of updating environment semantic map and the verification module of verifying alignment module, information coordination module and collaborative understanding module;The application can improve the safety and task execution efficiency of unmanned aerial vehicle task execution.

Description

Technical Field

[0001] This application relates to the field of multimodal environment perception technology, and in particular to a method for unmanned aerial vehicle (UAV) trajectory decision-making based on multimodal visual perception. Background Technology

[0002] The mission scenarios of drones (such as power line inspection, infrastructure monitoring, disaster assessment, terrain mapping and environmental monitoring) are usually characterized by rapid changes in targets or obstacles, frequent occlusion, drastic fluctuations in light, sensor degradation and asynchronous multi-source data. This makes it easy for drones to have problems such as missing information, miscalculation of representation or lag in decision-making in the closed-loop chain of perception-understanding-decision, thus affecting the continuity of mission and the safety of drone operations.

[0003] In related technologies, the optical and laser sensors relied upon by drones in bridges and power transmission lines are prone to image overexposure and sparse point clouds in strong light reflection or nighttime environments, leading to unstable target detection and requiring manual intervention. In complex buildings or low-light environments, drones' obstacle avoidance systems based solely on vision are susceptible to occlusion and reflection, resulting in misjudgments and trajectory deviations, making it difficult to guarantee continuous and reliable autonomous flight. Furthermore, the environment in which drones operate is highly time-varying; moving pedestrians, vehicles, wind-swept cables and vegetation, and sudden occlusion can all cause rapid changes in environmental structure. Continuing to use outdated environmental representations after critical changes can lead to delayed decision-making and risk accumulation.

[0004] Therefore, based on the above problems, there is an urgent need to provide a UAV trajectory decision-making method based on multimodal visual perception to improve the safety and efficiency of UAV mission execution. Summary of the Invention

[0005] The purpose of this application is to provide a UAV trajectory decision-making method based on multimodal visual perception, which can improve the safety and efficiency of UAV mission execution.

[0006] To achieve the above objectives, this application provides the following solution: Acquire multimodal perception data of the target area; the multimodal perception data includes RGB images and depth maps of the target area, motion information of dynamic targets, and motion status of the UAV; Based on multimodal perception data and a UAV trajectory decision model, trajectory decision results for the UAV are generated in real time. The UAV trajectory decision model includes an alignment module, a spatiotemporal feature encoder, an information collaboration module, a collaborative understanding module, a real-time supervisor, and a verification module. The alignment module aligns the multimodal perception data using a hardware triggering mechanism, calibration algorithm, and dynamic calibration algorithm to obtain aligned multimodal perception data. The spatiotemporal feature encoder independently encodes the aligned multimodal perception data to obtain multimodal features, including visual features, radar features, and UAV state features. The information collaboration module performs weighted fusion of multimodal features based on a multi-head cross-attention mechanism to generate a unified semantic map. The collaborative understanding module generates an environmental semantic map including the UAV's trajectory based on the unified semantic map, using deep learning methods and a large language model. The real-time supervisor performs low-latency dynamic change detection and event-driven re-understanding scheduling on the environmental semantic map to obtain an updated environmental semantic map. The verification module performs security verification on the alignment module, information collaboration module, and collaborative understanding module based on a verifiable security decision mechanism. Based on the real-time trajectory decision results of the drone, control the drone to execute the current task.

[0007] Optionally, acquiring multimodal sensing data of the target area specifically includes: Use an RGB-D camera to acquire RGB images of the target area; Use a time-of-flight depth sensor to obtain a depth map of the target area; Use millimeter-wave radar or lidar to acquire motion information of dynamic targets in the target area; The motion state of the UAV is obtained using an inertial measurement unit.

[0008] Optionally, the spatiotemporal feature encoder includes parallel RGB-D mode coding, radar mode coding, and inertial measurement unit coding; The RGB-D modal encoding is used to extract visual features from the aligned RGB image and the aligned depth map using a visual transformer; The radar mode coding is used to extract the geometry and trajectory of dynamic targets in the aligned motion information using a sparse convolutional neural network, thereby obtaining radar features; The inertial measurement unit encoding is used to capture the position and orientation changes of the UAV in the aligned motion state using a gated loop unit, thereby obtaining the UAV state characteristics.

[0009] Optionally, the processing procedure of the information collaboration module includes: Semantic and geometric alignment are performed on the multimodal features to obtain aligned multimodal features; Based on information uncertainty and physical consistency, the credibility of the aligned multimodal features is determined respectively. Based on the credibility of the aligned multimodal features, a weighted fusion of the aligned multimodal features is performed using a multi-head cross-attention mechanism to generate a unified semantic graph.

[0010] Optionally, the step of performing semantic and geometric alignment on the multimodal features to obtain aligned multimodal features specifically includes: Using formula Perform semantic and geometric alignment on multimodal features; in, For joint losses, For semantic task loss, For semantic loss weights, For semantic loss, For geometric consistency residual weights, For geometrically consistent residuals.

[0011] Optionally, determining the credibility of the aligned multimodal features based on information uncertainty and physical consistency specifically includes: Using formula Determine the credibility of the aligned multimodal features ; in, for function, For uncertainty coefficient, Due to information uncertainty, The physical consistency residual coefficient. This represents the physical consistency residual.

[0012] Optionally, the step of weightedly fusing the aligned multimodal features based on the credibility of the aligned multimodal features using a multi-head cross-attention mechanism to generate a unified semantic graph specifically includes: Using formula Generate a unified semantic graph; in, To unify semantic graphs, For normalization operations, This is a multi-head cross-attention mechanism. As a visual feature, For querying the matrix, The key matrix, It is a value matrix.

[0013] Optionally, the processing procedure of the collaborative understanding module includes: Based on the unified semantic graph, the flight paths of static objects, dynamic objects, and drones are identified using deep learning methods, and the identification results are obtained. Based on the recognition results, a dynamic scene graph is constructed using a graph neural network. The dynamic scene graph is encoded into structured fields based on the structured prompt builder; Based on structured fields, semantic reasoning is performed on the dynamic changes of the environment using a large language model to obtain the interaction relationships between static and dynamic objects in the target area; the interaction relationships include the interaction relationships between dynamic and static objects as well as the interaction relationships between dynamic objects. Based on the interaction relationship, a motion trajectory prediction model is constructed using a long short-term memory network; and the motion trajectory prediction model is used to predict the future motion trajectory of dynamic objects. Based on the future motion trajectory, dynamic scene map and interaction relationship, a risk potential energy function is constructed, and high-risk area constraints and 4D occupation-risk characterization are determined; Based on high-risk area constraints and 4D occupancy-risk characterization, an environmental semantic map including the drone's flight path is generated.

[0014] Optionally, the step of constructing a risk potential function based on future motion trajectories, dynamic scene maps, and interaction relationships, and determining high-risk area constraints and 4D occupancy-risk representations, specifically includes: Using formula Construct the risk potential function; in, Let be the risk potential function. This is the current location of the drone. For a moment, For gravitational potential energy, It is a collection of dynamic and static objects. voxels At any moment The probability of occupancy, For uncertainty term, As the first weighting coefficient, It is the first monotonically decreasing kernel function. for The location of the drone is constantly monitored. for and The distance between them It is a three-dimensional voxel set. This is the second weighting coefficient. It is the second monotonically decreasing kernel function. This is the third weighting coefficient.

[0015] Optionally, the real-time monitor includes: a fast summarizer, a change scorer, and a change point determiner; The fast summarizer is used to determine the corresponding supervision elements based on the environmental semantic map and compress the supervision elements; the supervision elements include: 4D occupancy-risk representation, dynamic scene graph, credibility and geometric consistency residual; The change scorer is used to determine the intensity of change of the compressed supervisory elements at adjacent time points; The change point determiner is used to determine whether to update the environmental semantic map based on the intensity of the change. If the intensity of the change is lower than a preset intensity threshold, the environmental semantic map is not updated and is used as the updated environmental semantic map. If the intensity of the change is higher than the preset intensity threshold, the environmental semantic map is re-scheduled for event-driven resolution to obtain the updated environmental semantic map.

[0016] This application provides a method for UAV trajectory decision-making based on multimodal visual perception, including: According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a UAV trajectory decision-making method based on multimodal visual perception. Using multimodal perception data of the target area, and based on a UAV trajectory decision-making model, the method generates UAV trajectory decision results in real time. The alignment module in the UAV trajectory decision-making model aligns the multimodal perception data using a hardware triggering mechanism, calibration algorithm, and dynamic calibration algorithm, ensuring the consistency of the multimodal perception data. A spatiotemporal feature encoder independently encodes the aligned multimodal perception data, improving the robustness of the UAV trajectory decision-making model. An information collaboration module uses a dynamic gating attention mechanism and a multi-head attention mechanism to perform weighted fusion of multimodal features, enabling the UAV trajectory decision-making model to focus on specific key areas, thus improving the accuracy of UAV trajectory decisions. The application enhances the navigation capabilities of the UAV. Through a collaborative understanding module, an environmental semantic map, including the UAV's trajectory, is generated based on deep learning methods and a large language model, providing rich semantic information for path planning and task execution. A supervisor is implemented to perform low-latency dynamic change detection and event-driven re-understanding scheduling on the environmental semantic map, maintaining the timeliness and consistency of the UAV's perception of the dynamic environment, reducing unnecessary computation, and improving the efficiency of UAV trajectory decision-making. A verification module verifies the alignment module, information collaboration module, and collaborative understanding module, enabling the UAV to possess verifiable safety decision-making capabilities in complex environments. In summary, this application can improve the safety and efficiency of UAV mission execution. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a UAV trajectory decision-making method based on multimodal visual perception in one embodiment of this application; Figure 2 This is a schematic diagram of a drone trajectory decision model in one embodiment of this application; Figure 3 This is a schematic diagram of the alignment module processing flow in one embodiment of this application; Figure 4 This is a schematic diagram of the spatiotemporal encoder processing flow in one embodiment of this application; Figure 5 This is a schematic diagram of the information collaboration module processing flow in one embodiment of this application; Figure 6 This is a schematic diagram of the collaborative understanding module processing flow in one embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] In one exemplary embodiment, such as Figure 1 As shown, a method for UAV trajectory decision-making based on multimodal visual perception is provided, including the following S1 to S3. Wherein: S1: Acquire multimodal sensing data of the target area.

[0022] Multimodal perception data includes RGB images and depth maps of the target area, motion information of dynamic targets, and the motion state of the UAV. Specifically, an RGB-D camera is used to acquire RGB images of the target area; a time-of-flight depth sensor is used to acquire depth maps of the target area; millimeter-wave radar or lidar is used to acquire motion information of dynamic targets in the target area; and an inertial measurement unit is used to acquire the motion state of the UAV. In other words, this application integrates an RGB-D camera, a time-of-flight depth sensor, millimeter-wave radar (or lidar), and an inertial measurement unit to achieve high-precision acquisition of optical, depth, motion, and 3D point cloud data of the target area environment.

[0023] The RGB-D camera in this application supports 4K resolution and millimeter-level depth measurement, and can adapt to both strong light and low light environments, providing high-quality visual data. Millimeter-wave radar or lidar can effectively detect the velocity of dynamic targets based on the Doppler effect, penetrate visual obstacles such as fog and haze, and provide real-time motion information of dynamic targets. An inertial measurement unit (IMU) can acquire the UAV's own motion state in real time, improving the accuracy of overall environmental perception.

[0024] S2: Based on multimodal perception data and the UAV trajectory decision model, generate UAV trajectory decision results in real time. The UAV trajectory decision model includes: an alignment module, a spatiotemporal feature encoder, an information collaboration module, a collaborative understanding module, a real-time representation supervisor (RTRS), and a validation module. The processing procedure of the UAV trajectory decision model is as follows: Figure 2 As shown.

[0025] The alignment module aligns the multimodal sensing data based on a hardware triggering mechanism, calibration algorithm, and dynamic calibration algorithm, resulting in aligned multimodal sensing data, such as... Figure 3 As shown.

[0026] Specifically, for asynchronous sampling of sensor arrays (RGB-D cameras, time-of-flight depth sensors, millimeter-wave radar or lidar, and inertial measurement units), a hardware triggering mechanism is used to align the initial timestamps, controlling the temporal differences between sensor arrays to the millisecond level to ensure precise temporal alignment of the data stream. Simultaneously, a calibration algorithm is employed to establish a unified coordinate system among the sensor arrays to ensure spatial consistency of multimodal sensing data, guaranteeing the accuracy of subsequent multimodal feature fusion and the reliability of environmental perception. Furthermore, a dynamic calibration algorithm is used to perform online calibration of the sensor array's extrinsic parameters to correct pose offsets between sensor arrays, ensuring optimal collaborative performance between different sensors. The dynamic calibration algorithm can be implemented using extended Kalman filtering. It uses IMU pre-integration to provide short-time motion priors and the geometric consistency residuals of millimeter-wave radar or lidar as observations. It jointly estimates the extrinsic parameter perturbations and time offsets of the sensor array within a sliding time window. The extrinsic parameter perturbations are used to dynamically correct extrinsic parameter drift. When the observation exceeds the preset observation threshold, the extrinsic parameter re-estimation is triggered, and a backtracking is performed to recalibrate the extrinsic parameters, thus avoiding the propagation of systematic errors caused by static calibration failure.

[0027] The sensor arrays in this application work together to achieve high-precision acquisition and alignment of multimodal perception data, comprehensively improving the ability to perceive the environment and ensuring that the UAV accurately perceives dynamic and static object information in the surrounding environment.

[0028] The spatiotemporal feature encoder is used to independently encode aligned multimodal sensing data to obtain multimodal features, such as... Figure 4 As shown. Multimodal features include visual features, radar features, and UAV state features. The spatiotemporal feature encoder includes parallel RGB-D modal encoding, radar modal encoding, and inertial measurement unit encoding; RGB-D modal encoding is used to extract features from the aligned RGB image and aligned depth map using a Visual Transformer (ViT) to obtain visual features, also known as global context features; radar modal encoding is used to extract the geometry and trajectory of dynamic targets in the aligned motion information using a Sparse Convolutional Neural Network (Sparse CNN) to obtain radar features; inertial measurement unit encoding is used to capture the position and orientation changes of the UAV in the aligned motion state using a Gated Recurrent Unit (GRU) to obtain UAV state features, i.e., the UAV's motion state vector.

[0029] In RGB-D modal coding, the aligned RGB image and the aligned depth map are divided into 16×16 non-overlapping blocks, and ViT is used to extract features from each block to obtain global contextual information, thus obtaining visual features. ViT focuses on the boundaries, textures, and context of objects in the scene. ViT includes 12 Transformer layers and a multi-head self-attention mechanism (8 attention heads). The formula for determining visual features is as follows: ; in, As a visual feature, Encoded for a multi-head self-attention mechanism, used to fuse the global correlation of features. For splicing operations, The RGB features are obtained by encoding the aligned RGB image. The depth features are obtained by encoding the aligned depth map.

[0030] In radar modal coding, a sparse convolutional neural network is used to extract the geometry and trajectory of dynamic targets from aligned motion information based on 3D sparse convolutional kernels, thereby obtaining radar features. The calculation formula is as follows: ; in, For sparse convolutional neural network operations, This refers to the sparse voxelized representation of radar / lidar point clouds, i.e., aligned motion information. These are the parameters of the sparse convolution kernel.

[0031] In the inertial measurement unit encoding, a gated cyclic unit is used to capture the acceleration and angular velocity sequences of the UAV in the aligned motion state to determine the position and orientation changes of the UAV, and further determine the UAV state characteristics, i.e., the UAV motion state vector.

[0032] The multimodal features obtained in this application form the basis for further environmental understanding and decision generation, providing UAVs with global and dynamic environmental cognition capabilities.

[0033] The information collaboration module is used to perform weighted fusion of multimodal features based on dynamic gating attention and multi-head attention mechanisms to generate a unified semantic graph, such as... Figure 5As shown, firstly, semantic and geometric alignment are performed on the multimodal features to obtain aligned multimodal features; secondly, the credibility of the aligned multimodal features is determined based on information uncertainty and physical consistency; subsequently, based on the credibility of the aligned multimodal features, a weighted fusion is performed on the aligned multimodal features using a multi-head cross-attention mechanism to generate a unified semantic graph.

[0034] Specifically, to eliminate the heterogeneity of sensor arrays in the feature space, a joint alignment strategy of semantic alignment and geometric alignment is adopted to align multimodal features. Semantic alignment is used as a soft constraint for "candidate alignment". Semantic alignment uses InfoNCE loss as semantic loss to map multimodal features of the same spatiotemporal segment to a common representation space. To eliminate false matches, geometric consistency constraints, also known as geometric consistency residuals (such as reprojection error or point-to-surface distance), are used in geometric alignment to suppress false alignment. False alignment includes pseudo alignment caused by occlusion, reflection, and point cloud sparsity. The two work together to ensure the interpretability and verifiability of subsequent multimodal feature fusion.

[0035] The joint loss can be obtained by combining semantic loss with geometric consistency residuals. By minimizing the joint loss, multimodal features can be made consistent both semantically and geometrically, reducing ambiguity during multimodal feature fusion. The formula for calculating the joint loss is as follows: ; in, For joint losses, For semantic task loss, For semantic loss weights, For semantic loss, , For geometric consistency residual weights, For geometrically consistent residuals, For cosine similarity, For temperature coefficient, and All are time periods. For the first Visual characteristics of different time periods For the first Radar characteristics over time period For the first Radar characteristics over a given time period.

[0036] After aligning the multimodal features, the credibility of the aligned multimodal features is determined based on information uncertainty and physical consistency.

[0037] Specifically, determine each feature in the aligned multimodal features. Information uncertainty (entropy) Physical consistency residuals (For example, reprojection error, registration error, or velocity consistency error). Each feature here refers to the aligned visual feature, aligned radar feature, and aligned UAV state feature. The formula for calculating the information uncertainty of each feature is as follows: ; Features In the Normalized probability over time period For discrete time period indexes, This represents the total number of time periods, corresponding to the total number of categories or the total number of voxels.

[0038] Further determine each feature Credibility Also known as normalized weights, the calculation formula is as follows: ; in, for function, For uncertainty coefficient, Due to information uncertainty, The physical consistency residual coefficient. This represents the physical consistency residual.

[0039] When vision is affected by low light or glare, the information uncertainty or physical consistency residual corresponding to the aligned visual features increases, and the credibility decreases, that is, the weight of the visual features decreases; when radar multipath reflection is enhanced or point clouds are sparse, the credibility of the aligned radar features will also decrease.

[0040] Based on the credibility of each feature, a multi-head cross-attention mechanism is used for weighted fusion to generate a unified semantic graph. The aligned visual features are then used as the query matrix. Joint features of aligned radar features and aligned UAV state features as a bond matrix Sum matrix Calculate the cross-modal correlation matrix.

[0041] ; ; ; in, To query the weight matrix, The key weight matrix, The value weight matrix.

[0042] The dynamic gating attention mechanism generates corresponding gating coefficients based on the credibility of each feature to modulate the attention weights of multimodal features. Based on this, a multi-head cross-attention mechanism is used to weightedly fuse the aligned multimodal features to obtain a unified semantic graph. This application can prioritize the most relevant features based on the complexity of the scene. For example, in densely populated environments, radar features and UAV status features are given priority, while in simple environments, visual features are given priority.

[0043] The formula for calculating the multi-head cross-attention mechanism is as follows: ; in, This is a multi-head cross-attention mechanism. The dimension of the key vector. for Function, superscript This is a transpose.

[0044] To prevent gradient vanishing and preserve the uniqueness of multimodal features, residual connections and layer normalization are introduced into the multi-head cross-attention mechanism. This allows for compensation using the high reliability of other modalities when a feature degenerates, while avoiding overfitting to a single modality. The formulas for calculating residual connections and layer normalization are as follows: ; ; in, The output characteristics after residual connection The input features for the multi-head cross-attention mechanism are the aligned multimodal features. For multi-head cross-attention function, , For the first The output of each attention head, , , , , The output features are after layer normalization. For normalization operations, For the first A query vector with attention heads. For the first The weight matrix of the query vector for each attention head. For the first The key vector of each attention head. For the first The weight matrix of the key vectors of each attention head. For the first A vector of values ​​for each attention head. For the first The weight matrix of the value vector of each attention head. For attention head index, The output transformation matrix is ​​used to linearly map the output vector after concatenating multiple attention heads, so as to achieve feature dimension compression and secondary fusion of multi-head information.

[0045] Residual connections and layer normalization are internal computational units of the multi-head cross-attention mechanism, and their output features are the inputs for generating the unified semantic graph.

[0046] The formula for calculating the unified semantic graph is as follows: ; Among them, among them, To unify semantic graphs, For normalization operations, This is a multi-head cross-attention mechanism. As a visual feature, For querying the matrix, The key matrix, It is a value matrix.

[0047] This application integrates complementary information from multimodal features and fuses them into a unified semantic graph, achieving comprehensive environmental understanding. This enables UAVs to achieve adaptive enhancement by utilizing high-confidence radar features even when visual features fail in low-light conditions. By dynamically suppressing low-confidence features, the robustness and accuracy of UAV navigation are improved.

[0048] The collaborative understanding module is used to generate an environmental semantic map, including the drone's flight trajectory, based on a unified semantic graph, deep learning methods, and a large language model. Figure 6 As shown. Specifically, based on the unified semantic graph, deep learning methods are used to identify static objects, dynamic objects, and the flight paths of drones, yielding identification results. Based on these results, a dynamic scene graph is constructed using a graph neural network. A structured prompt builder encodes the dynamic scene graph into structured fields. Based on these structured fields, a large language model is used to perform semantic reasoning on the dynamic changes in the environment, obtaining the interaction relationships between static and dynamic objects in the target area. Based on these interaction relationships, a motion trajectory prediction model is constructed using a long short-term memory network. This model is then used to predict the future motion trajectories of dynamic objects. Based on the future motion trajectories, the dynamic scene graph, and the interaction relationships, a risk potential function is constructed, and high-risk area constraints and 4D occupancy-risk representations are determined. Based on the high-risk area constraints and the 4D occupancy-risk representations, an environmental semantic map including the drone's flight trajectory is generated. The 4D occupancy-risk representation introduces a time dimension into a three-dimensional voxel grid, jointly encoding the occupancy probability and corresponding risk value of each voxel at each prediction time.

[0049] In one exemplary embodiment, a dynamic scene graph The construction of this application enables it to possess composable relational reasoning capabilities, wherein, For a set of nodes, Let be a set of edges, nodes Representing observable entities, including static objects (such as cables and navigation marks) and dynamic objects (such as vehicles and pedestrians), edges This represents the interaction relationships between spatially adjacent objects, including relative distance, relative orientation, visibility occlusion, and relative velocity, at each moment. The dynamic scene graph is updated using a graph neural network, and the update formula is as follows: ; in, For the first dynamic scene graph Layer Time Node The representation of vector nodes, It is a non-linear activation function. For neighboring nodes, For the set of neighboring nodes, The normalization coefficient is... For the first dynamic scene graph Layer learnable weight matrix, For the first dynamic scene graph Layer Time Node The representation vector.

[0050] By updating the dynamic scene graph, we can not only obtain the distribution of objects, but also the interaction relationships between objects and how the intensity of the interaction changes over time.

[0051] The structured cue builder encodes dynamic scene graphs into structured fields. Example fields for these structured fields include: Nodes = {id, category, position, velocity, size, visibility, uncertainty}; Edges = {relative distance, relative orientation, interaction type, risk level}; Drone state S = {position, velocity, attitude, remaining battery}; Task constraints C = {target point, no-fly zone, maximum tilt angle / maximum acceleration / safe distance}; Risk summary R = {Top-K high-risk voxels / sectors and their potential energy values}. These structured fields can be organized into JSON or key-value tables as input for subsequent large language models. The structured fields not only cover the physical motion states of objects but also quantify spatial adjacency and interaction features in the environment, providing standardized data input for subsequent inference.

[0052] Large language models can perform semantic reasoning on dynamic changes in the environment based on structured fields, and identify and extract the interaction relationships between static and dynamic objects in the target area. The interaction relationships include the interaction relationships between dynamic and static objects as well as the interaction relationships between dynamic objects. This interaction relationship is a deep understanding of the dynamic changes in the environment and can reveal the potential influence mechanisms between objects (such as avoidance, following or occlusion), enabling UAVs to understand complex environmental contexts, such as the interaction between objects, the priority of targets and mission requirements.

[0053] Based on the interaction relationships, a motion trajectory prediction model is constructed using a Long Short-Term Memory (LSTM) network; and this model is then used to predict the future motion trajectories of dynamic objects. First, an initial motion trajectory prediction model is constructed. Essentially, the initial motion trajectory prediction model is a Sequence Prediction Model (SPM) based on spatiotemporal context awareness. The core architecture of the initial motion trajectory prediction model is a Long Short-Term Memory (LSTM) network, designed to capture the non-linear motion characteristics of dynamic objects in complex interactive environments. The initial motion trajectory prediction model includes an input layer, a hidden layer (feature extraction layer), and an output layer. The hidden layer consists of three stacked LSTM layers, each containing 256 hidden units, used for deep temporal encoding of semantic interaction relationships. The initial trajectory prediction model takes the historical interaction relationship feature sequence and corresponding historical motion state vector (historical interaction relationship) of multiple consecutive frames as input, and the predicted spatial coordinate sequence of dynamic objects in multiple future frames (historical trajectory of dynamic objects) as output. A supervised learning method is employed, using the Euclidean distance or mean square error between the predicted coordinate values ​​and the actual sensor observations to construct a loss function. The weight parameters of the initial trajectory prediction model are continuously updated through backpropagation until the initial trajectory prediction model converges, resulting in the final trajectory prediction model. This model can predict the future trajectory of dynamic objects based on the current interaction relationships, further improving the navigation accuracy of UAVs.

[0054] To quantify the collision risk of drones in complex environments, this application constructs a risk potential energy function for drones based on the future trajectory of dynamic objects, dynamic scene graphs, and interaction relationships. High-risk area constraints and 4D occupancy-risk characterization were identified. ,in, This is a voxel or bird's-eye view grid (BEV). For the given time, the specific formula for calculating the risk potential energy function is as follows: ; in, Let be the risk potential function. This is the current location of the drone. For a moment, For gravitational potential energy, It is a collection of dynamic and static objects. voxels At any moment The probability of occupancy, For uncertainty term, As the first weighting coefficient, It is the first monotonically decreasing kernel function. , For The location of the drone is constantly monitored. It is a three-dimensional voxel set. This is the second weighting coefficient. It is the second monotonically decreasing kernel function. The third weighting coefficient, For safe distance threshold, For distance measurement, This is an indicator function.

[0055] The risk cost of the drone's trajectory can be obtained from the risk potential energy function. The calculation formula is as follows: ; in, for The drone's location at any given time. For drones Risk potential function at time t, To predict the length of the time domain.

[0056] The goal is to minimize the risk cost, and the risk potential function is forced. This allows us to obtain the high-risk area mask or immersion sector, and further generate the drone's real-time obstacle avoidance trajectory.

[0057] Based on the constraints of high-risk areas, an environmental semantic map including the drone's operating trajectory is determined; this process enables the drone to make efficient analysis and judgment of the surrounding environment in real time, which assists in the execution of actions and path planning.

[0058] Environmental semantic maps may become invalid in the next instant due to issues such as object movement, occlusion, sensor degradation, or extrinsic parameter drift, leading to map expiration. Using an expired environmental semantic map can cause decision-making delays and induce risks. Frequent global re-understanding of the environmental semantic map can result in uncontrollable computational power and latency. Therefore, this application introduces a real-time supervisor. The advantage of the real-time supervisor is that it upgrades the environmental semantic map re-understanding from experience-triggered to verifiable change point determination, and introduces hierarchical updates of local or global data. While ensuring safety and timeliness, it significantly reduces unnecessary repetitive calculations and is more suitable for the real-time constraints of UAVs. The real-time supervisor is located in the middle link of the unified semantic graph, environmental semantic map, and trajectory decision-making, and is used to continuously monitor the effectiveness of the environmental semantic map in a lightweight and low-latency manner.

[0059] A real-time supervisor is used to perform low-latency dynamic change detection and event-driven re-disclosure scheduling on the environmental semantic map to obtain an updated environmental semantic map. The real-time supervisor includes a fast summarizer, a change scorer, and a change point determiner. The fast summarizer determines the corresponding supervisory elements based on the environmental semantic map and compresses these elements. Supervisory elements include: 4D occupancy-risk representation, dynamic scene graph, credibility, and geometric consistency residuals. The change scorer determines the change intensity of the compressed supervisory elements at adjacent time points and unifies the supervisory elements to the same trigger scale. The change point determiner decides whether to update the environmental semantic map based on the change intensity. If the change intensity is below a preset intensity threshold, the environmental semantic map is not updated, and the updated environmental semantic map is used. If the change intensity is above the preset intensity threshold, the environmental semantic map is re-disclosed using an event-driven re-disclosure scheduling mechanism to obtain the updated environmental semantic map. Specifically, the change point determiner performs sequential detection of change intensity to suppress jitter. When the change intensity exceeds a preset intensity threshold, it selects either local or global re-understanding based on the degree of change concentration and security relevance. Local re-understanding involves locally updating the environmental semantic map, while global re-understanding involves globally updating the environmental semantic map. Furthermore, when the change point determiner detects a significant anomaly in the supervised elements (geometric consistency residuals exceeding a set threshold or an overall decrease in credibility), it triggers a safety-closed-loop rollback strategy to regenerate the environmental semantic map, achieving stable operation with real-time environmental semantic map updates, controllable resources, and anomaly rollback.

[0060] The real-time supervisor in this application employs four supervisory elements to jointly determine the usability of the environmental semantic map from four aspects: risk, interaction, reliability, and geometry, rather than using a single threshold to determine it; it responds quickly to sudden changes in the environment through event-driven re-understanding scheduling, while reducing the amount of computation and maintaining low-overhead operation during stable phases.

[0061] The verification module is used to verify the alignment module, information collaboration module, and collaborative understanding module based on the Verified Safe Decision Mechanism (VSDM). Specifically, the verification module includes an extrinsic parameter verification unit, a feature fusion verification unit, and a risk verification unit. The extrinsic parameter verification unit performs safety verification on the geometric consistency residuals of the sensor array in the alignment module. When the geometric consistency residuals are less than a preset residual threshold, the safety verification passes, and the extrinsic parameter verification unit outputs the extrinsic parameters of the sensor array calibration to judge the accuracy of the sensor array extrinsic parameter calibration. The feature fusion verification unit performs safety verification on the information uncertainty, physical consistency, and credibility in the information collaboration module. When the physical consistency residuals are less than a physical consistency threshold, the safety verification passes, and the feature fusion verification unit outputs the information consistency residuals of each feature in the multimodal features. The credibility status (normal or abnormal) is used to determine the accuracy of multimodal feature weighted fusion. The risk verification unit is used to perform safety verification on the 4D occupancy-risk representation, risk potential function, and risk cost in the collaborative understanding module. When the risk potential function is less than the risk threshold, the safety verification passes. The risk verification unit outputs the current risk constraint geometry and the safe trajectory and actions of the UAV (hovering, deceleration, ascent, return, etc.) to determine the safety of the UAV trajectory. When the safety verification fails, a rollback is performed, that is, the alignment module, information collaboration module, and collaborative understanding module are respectively controlled to recalculate the corresponding parameters to ensure the accuracy of the UAV trajectory determination.

[0062] Any action and trajectory of the drone must be verified by the verification module before it can be executed. If geometric consistency residuals are found to be abnormal, risks are abnormal, or the boundary is unreachable (whether the boundary is reachable is set according to the actual situation), a conservative strategy or backtracking and recalculation will be triggered to ensure that the safety boundary is not breached. This completes the safety decision chain of online extrinsic parameter calibration, multimodal feature fusion, risk cost visualization, safety verification, and anomaly backtracking.

[0063] The verification module of this application elevates the UAV system from a potentially correct black-box decision-making process to a verifiable, degradable, and well-defined closed-loop link. It addresses the multimodal feature fusion distortion caused by sensor array degradation, extrinsic parameter drift, occlusion, and multipath interference in dynamic environments. The parallel network structure enables "safety verification" to be executed independently. The backtracking closed-loop link ensures that a conservative feasible solution is still output when the UAV trajectory decision model is uncertain or observations are abnormal. Furthermore, "verifiable safety" is decomposed into three checkable conditions: observation consistency (geometric consistency residuals), characterization interpretability (4D occupancy-risk characterization, risk potential function, and risk cost), and decision verifiability (boundary reachability), thereby obtaining robust safety guarantees in complex dynamic scenarios.

[0064] S3: Based on the real-time trajectory decision results of the drone, control the drone to execute the current task.

[0065] This application achieves weighted fusion and collaborative understanding of multimodal features through a UAV trajectory decision model, significantly improving the UAV's environmental understanding capability and mission execution efficiency in dynamic and complex scenarios. It also introduces a real-time supervisor to perform low-latency dynamic change detection and event-driven re-understanding scheduling of the environmental semantic map, ensuring its timeliness and consistency. Furthermore, a verifiable security decision mechanism is used to perform security verification on the parameters in the alignment module, information collaboration module, and collaborative understanding module, explicitly transforming the "understanding results" into verifiable risk constraints and reachability judgment conditions, thereby improving the security and efficiency of UAV mission execution.

[0066] This application can be widely used in UAV mission scenarios such as power line inspection, infrastructure monitoring, disaster assessment, and topographic mapping. It is especially suitable for high dynamic conditions such as rapid changes in targets / obstacles, sensor degradation, and strong occlusion interference. It provides efficient, accurate, and adaptive technical support for intelligent UAV perception and decision-making, and provides a highly robust solution for multi-scenario aerial perception systems with a systematic robustness of "real-time updatable, verifiable security, and rollback closed loop".

[0067] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0068] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for UAV trajectory decision-making based on multimodal visual perception, characterized in that, The UAV trajectory decision-making method based on multimodal visual perception includes: Acquire multimodal perception data of the target area; the multimodal perception data includes RGB images and depth maps of the target area, motion information of dynamic targets, and motion status of the UAV; Based on multimodal perception data and a UAV trajectory decision model, trajectory decision results for the UAV are generated in real time. The UAV trajectory decision model includes an alignment module, a spatiotemporal feature encoder, an information collaboration module, a collaborative understanding module, a real-time supervisor, and a verification module. The alignment module aligns the multimodal perception data using a hardware triggering mechanism, calibration algorithm, and dynamic calibration algorithm to obtain aligned multimodal perception data. The spatiotemporal feature encoder independently encodes the aligned multimodal perception data to obtain multimodal features, including visual features, radar features, and UAV state features. The information collaboration module performs weighted fusion of multimodal features based on a multi-head cross-attention mechanism to generate a unified semantic map. The collaborative understanding module generates an environmental semantic map including the UAV's trajectory based on the unified semantic map, using deep learning methods and a large language model. The real-time supervisor performs low-latency dynamic change detection and event-driven re-understanding scheduling on the environmental semantic map to obtain an updated environmental semantic map. The verification module performs security verification on the alignment module, information collaboration module, and collaborative understanding module based on a verifiable security decision mechanism. Based on the real-time trajectory decision results of the drone, control the drone to execute the current task; The processing steps of the information collaboration module include: Semantic and geometric alignment are performed on the multimodal features to obtain aligned multimodal features; Based on information uncertainty and physical consistency, the credibility of the aligned multimodal features is determined respectively. Based on the credibility of the aligned multimodal features, a weighted fusion of the aligned multimodal features is performed using a multi-head cross-attention mechanism to generate a unified semantic graph.

2. The UAV trajectory decision-making method based on multimodal visual perception according to claim 1, characterized in that, The acquisition of multimodal sensing data of the target area specifically includes: Use an RGB-D camera to acquire RGB images of the target area; Use a time-of-flight depth sensor to obtain a depth map of the target area; Use millimeter-wave radar or lidar to acquire motion information of dynamic targets in the target area; The motion state of the UAV is obtained using an inertial measurement unit.

3. The UAV trajectory decision-making method based on multimodal visual perception according to claim 1, characterized in that, The spatiotemporal feature encoder includes parallel RGB-D mode coding, radar mode coding, and inertial measurement unit coding; The RGB-D modal encoding is used to extract visual features from the aligned RGB image and the aligned depth map using a visual transformer; The radar mode coding is used to extract the geometry and trajectory of dynamic targets in the aligned motion information using a sparse convolutional neural network, thereby obtaining radar features; The inertial measurement unit encoding is used to capture the position and orientation changes of the UAV in the aligned motion state using a gated loop unit, thereby obtaining the UAV state characteristics.

4. The UAV trajectory decision-making method based on multimodal visual perception according to claim 1, characterized in that, The process of semantically and geometrically aligning the multimodal features to obtain aligned multimodal features specifically includes: Using formula Perform semantic and geometric alignment on multimodal features; in, For joint losses, For semantic task loss, For semantic loss weights, For semantic loss, For geometric consistency residual weights, For geometrically consistent residuals.

5. The UAV trajectory decision-making method based on multimodal visual perception according to claim 1, characterized in that, The determination of the credibility of aligned multimodal features based on information uncertainty and physical consistency specifically includes: Using formula Determine the credibility of the aligned multimodal features ; in, for function, For uncertainty coefficient, Due to information uncertainty, The physical consistency residual coefficient. This represents the physical consistency residual.

6. The UAV trajectory decision-making method based on multimodal visual perception according to claim 1, characterized in that, The step involves weighted fusion of the aligned multimodal features based on their credibility using a multi-head cross-attention mechanism to generate a unified semantic graph, specifically including: Using formula Generate a unified semantic graph; in, To unify semantic graphs, For normalization operations, This is a multi-head cross-attention mechanism. As a visual feature, For querying the matrix, The key matrix, It is a value matrix.

7. The UAV trajectory decision-making method based on multimodal visual perception according to claim 1, characterized in that, The processing steps of the collaborative understanding module include: Based on the unified semantic graph, the flight paths of static objects, dynamic objects, and drones are identified using deep learning methods, and the identification results are obtained. Based on the recognition results, a dynamic scene graph is constructed using a graph neural network. The dynamic scene graph is encoded into structured fields based on the structured prompt builder; Based on structured fields, semantic reasoning is performed on the dynamic changes of the environment using a large language model to obtain the interaction relationships between static and dynamic objects in the target area; the interaction relationships include the interaction relationships between dynamic and static objects as well as the interaction relationships between dynamic objects. Based on the interaction relationship, a motion trajectory prediction model is constructed using a long short-term memory network; and the motion trajectory prediction model is used to predict the future motion trajectory of dynamic objects. Based on the future motion trajectory, dynamic scene map and interaction relationship, a risk potential energy function is constructed, and high-risk area constraints and 4D occupation-risk characterization are determined; Based on high-risk area constraints and 4D occupancy-risk characterization, an environmental semantic map including the drone's flight path is generated.

8. The UAV trajectory decision-making method based on multimodal visual perception according to claim 7, characterized in that, The process involves constructing a risk potential function based on future motion trajectories, dynamic scene maps, and interaction relationships, and determining high-risk area constraints and 4D occupancy-risk characterization, specifically including: Using formula Construct the risk potential function; in, Let be the risk potential function. This is the current location of the drone. For a moment, For gravitational potential energy, It is a collection of dynamic and static objects. voxels At any moment The probability of occupancy, For uncertainty term, As the first weighting coefficient, It is the first monotonically decreasing kernel function. for The location of the drone is constantly monitored. for and The distance between them It is a three-dimensional voxel set. This is the second weighting coefficient. It is the second monotonically decreasing kernel function. This is the third weighting coefficient.

9. The UAV trajectory decision-making method based on multimodal visual perception according to claim 7, characterized in that, The real-time monitor includes: a fast summarizer, a change scorer, and a change point determiner; The fast summarizer is used to determine the corresponding supervision elements based on the environmental semantic map and compress the supervision elements; the supervision elements include: 4D occupancy-risk representation, dynamic scene graph, credibility and geometric consistency residual; The change scorer is used to determine the intensity of change of the compressed supervisory elements at adjacent time points; The change point determiner is used to determine whether to update the environmental semantic map based on the intensity of the change. If the intensity of the change is lower than a preset intensity threshold, the environmental semantic map is not updated and is used as the updated environmental semantic map. If the intensity of the change is higher than the preset intensity threshold, the environmental semantic map is re-scheduled for event-driven resolution to obtain the updated environmental semantic map.

Images