A multi-modal perception based unmanned aerial vehicle cooperative task planning method

Abstract

The application discloses a kind of unmanned plane cooperative task planning methods based on multi-modal perception, comprising the following steps: data in the process of unmanned plane execution task is collected, and multi-modal perception dataset is generated;Multi-modal perception dataset is input into task influence domain perception Perceiver IO model, and generates task query set;Task influence domain is constructed, and task influence domain set is generated;Determine the overlapping relationship, and generate influence domain overlapping relationship set;Determine the cooperative execution relationship, and generate cooperative task planning result;Unmanned plane cooperative task planning result is output.The application utilizes task influence domain perception Perceiver IO model, realizes unmanned plane cooperative task planning, with the advantages of accurate planning, fast response.

Description

Technical Field

[0001] This invention relates to the field of UAV collaborative mission planning, and in particular to a UAV collaborative mission planning method based on multimodal perception. Background Technology

[0002] With the widespread application of drones in scenarios such as inspection, surveying, logistics, emergency rescue and security monitoring, the collaborative execution of multiple drones has become an important technical means to improve operational efficiency and expand the scope of mission coverage. Existing drone collaborative mission planning methods usually make comprehensive use of visible light images, infrared images, lidar point clouds, flight status and location information to perform path planning, mission allocation and collision avoidance control for drone missions, so as to achieve collaborative operation between multiple drones.

[0003] However, most existing technologies focus on the state perception or local collision avoidance analysis of a single UAV, lacking the ability to model the task influence domain for the task execution process. This makes it difficult to accurately characterize the spatial and temporal mutual influence relationships of different tasks, resulting in the difficulty in timely identification of potential overlapping relationships between tasks. At the same time, for the adjustment of collaborative relationships caused by dynamic changes in tasks, the overall replanning method is often adopted, which has a large computational cost and cannot meet the real-time and dynamic update requirements of collaborative task planning in multi-UAV scenarios, thus affecting the accuracy and execution efficiency of collaborative task planning results. Summary of the Invention

[0004] One objective of this invention is to propose a UAV collaborative task planning method based on multimodal perception. This invention utilizes the Perceiver IO model for task influence domain perception to realize UAV collaborative task planning, which has the advantages of accurate planning and fast response.

[0005] A UAV cooperative mission planning method based on multimodal perception according to an embodiment of the present invention includes the following steps: Collect visible light images, infrared images, lidar point clouds, flight status data, location information, and mission information during the UAV's mission execution process, and preprocess them to generate a multimodal perception dataset; The multimodal perception dataset is input into the task influence domain perception Perceiver IO model, which includes a multimodal perception encoding layer, a task query construction layer, an influence boundary expansion layer, an influence domain overlap attention layer, and a collaborative planning output layer. The task query set is generated through the multimodal perception encoding layer and the task query construction layer. Based on the task query set, the task influence domain corresponding to each task is constructed using the influence boundary extension layer, and a task influence domain set is generated. The overlapping attention layer of influence domains is invoked to perform pairwise correlation analysis on each task influence domain in the task influence domain set, determine the overlapping relationship between each task influence domain, and generate a set of influence domain overlapping relationships. Based on the set of overlapping influence domains, the collaborative execution relationships between tasks are determined, and collaborative task planning results are generated through the collaborative planning output layer. Based on real-time updated multimodal perception data, the task influence domain set, influence domain overlap relationship set, collaborative execution relationship, and collaborative task planning results are updated synchronously, and the UAV collaborative task planning results are output.

[0006] Optionally, the preprocessing specifically includes time synchronization, coordinate unification, anomaly removal, and standardization.

[0007] Optionally, the generation of the task query set specifically includes: The multimodal sensing dataset is input into the task influence domain sensing Perceiver IO model, and the multimodal sensing dataset is sent to the multimodal sensing encoding layer for encoding processing. The multimodal sensing coding layer encodes visible light images, infrared images, lidar point clouds, flight status data, location information, and mission information respectively, generates coding features, and performs mapping processing according to a unified feature dimension to form a multimodal coding feature set. The multimodal coding feature set is input into the task query construction layer, the task information of each task is traversed, the multimodal coding features corresponding to each task are extracted, and the corresponding associations are made according to the task identifier to form a task candidate feature set; The task query construction layer uses the task information corresponding to each task as the query vector, performs interactive query processing on the multimodal encoded features in the task candidate feature set, and generates task query features. The query features of each task are grouped according to the task identifier, and a mapping relationship between each query feature and the task is established to generate a task query set.

[0008] Optionally, the generation of the task influence domain set specifically includes: Input the task query set into the influence boundary extension layer, read each task query in the task query set in sequence, and extract the location information, flight status data and task information corresponding to each task query. The influence boundary expansion layer takes each task query as the center, performs boundary expansion calculation based on the corresponding location information, flight status data and task information, generates the first expansion result corresponding to each task query, and uses each first expansion result as the initial input for the next round of boundary expansion calculation; The influence boundary expansion layer continues to perform boundary expansion calculations based on the first expansion result, and updates the expansion result obtained in this round of boundary expansion calculations with the expansion result obtained in the previous round of boundary expansion calculations accordingly. The boundary expansion calculations are performed continuously until the expansion results obtained in two consecutive rounds of boundary expansion calculations no longer change, and the task influence domain corresponding to each task is generated. Organize the task influence domains corresponding to each task in a unified manner, and establish the correspondence between each task influence domain and the task according to the task identifier; The influence domains of each task are grouped and processed according to the task identifier to generate a set of task influence domains.

[0009] Optionally, the generation of the set of overlapping influence domains specifically includes: Input the set of task influence domains into the influence domain overlap attention layer, read each task influence domain in sequence according to the task identifier, and combine each task influence domain with the other task influence domains in pairs to generate task influence domain combinations. The overlapping influence domain attention layer reads the combination of influence domains of each task, performs synchronous processing on the spatial location and corresponding time contained in the two task influence domains, and generates the corresponding arrangement result. The overlapping attention layer of the influence domain performs overlapping association calculations based on the corresponding arrangement results, determines the spatial location that the two task influence domains jointly cover at each corresponding time, and continuously associates the jointly covered spatial locations according to the corresponding time to generate the overlapping part corresponding to each task influence domain combination; Based on the overlapping parts corresponding to each task influence domain combination, determine the overlapping relationship between the corresponding task influence domains, and establish the correspondence between each overlapping relationship and the corresponding task influence domain combination.

[0010] Optionally, the generation of the collaborative task planning results specifically includes: Input the set of overlapping relationships of the influence domain into the collaborative planning output layer, extract the task, spatial location and corresponding time corresponding to each overlapping relationship according to the task identifier, and arrange the overlapping relationships corresponding to the same task according to the corresponding time. Perform collaborative association calculations on the overlapping relationships after arrangement, and determine the common execution relationship, staggered execution relationship or spatial avoidance relationship between the tasks based on the common coverage spatial location and its corresponding time, and generate collaborative execution relationship; For joint execution relationships, the entry order, execution content, and exit order of the corresponding tasks are determined according to the task information; To address the staggered execution relationship, adjust the entry or exit time of the corresponding tasks so that the corresponding times of each task are staggered. In response to spatial avoidance relationships, the flight direction, flight altitude, or flight path of the corresponding mission is adjusted based on location information and flight status data; The adjustment results will be associated with the corresponding task identifiers and collaborative execution relationships; The collaborative execution relationships and adjustment results of each task are uniformly arranged according to the corresponding time, and tasks that do not have overlapping relationships are retained according to their original task information to generate collaborative task planning results.

[0011] Optionally, the generation of the UAV collaborative task planning results specifically includes: Acquire current multimodal perception data, extract location information, flight status data and task information corresponding to each task, and compare them with the task information corresponding to the collaborative task planning results to determine the tasks that have changed; Based on the changed tasks, the task influence domain of the corresponding tasks is regenerated, and the affected tasks that share a common spatial location with the updated task influence domain are identified. For the affected tasks, redetermine the set of overlapping relationships of the affected domains, and redetermine the corresponding joint execution relationships, staggered execution relationships, or spatial avoidance relationships based on the updated set of overlapping relationships of the affected domains; Based on the newly determined joint execution relationship, staggered execution relationship, or spatial avoidance relationship, the entry time, exit time, flight direction, flight altitude, or flight path of the affected mission are adjusted to generate updated adjustment results; The updated adjustment results are merged with the original collaborative task planning results corresponding to the unaffected tasks, and rearranged according to the corresponding time to generate the updated UAV collaborative task planning results.

[0012] The beneficial effects of this invention are: This invention proposes a multimodal perception-based UAV collaborative mission planning method. By fusing visible light images, infrared images, lidar point clouds, flight status data, location information, and mission information, a Perceiver I / O model for mission influence domain perception is constructed. This model performs unified encoding and fusion analysis of multi-source perception information, further generating a set of mission influence domains and a set of overlapping influence domain relationships. Based on this, joint execution relationships, staggered execution relationships, or spatial avoidance relationships are determined, enabling UAV collaborative mission planning. Compared to existing technologies that only analyze the state of a single UAV or local collision avoidance, this invention can uniformly model the spatial and temporal influences between different UAV missions at the mission execution level, improving the overall accuracy of multi-UAV collaborative mission planning.

[0013] Meanwhile, by constructing task influence domains and their overlapping relationships, this invention can accurately identify potential influence relationships between different tasks during task execution and generate corresponding collaborative execution relationships based on these relationships. This makes the collaborative task planning results more consistent with actual flight scenarios. When the UAV's flight status, task information, or environmental perception information changes, only the task influence domain, its overlapping relationship, and the collaborative task planning results corresponding to the changed task are updated. There is no need to replan all tasks, which effectively reduces the computational overhead during dynamic adjustment and improves the real-time response capability and dynamic update efficiency of collaborative task planning. As a result, it can better adapt to complex and ever-changing UAV collaborative operation scenarios, improve the efficiency of UAV collaborative task execution, and enhance overall operational stability. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a UAV cooperative mission planning method based on multimodal perception proposed in this invention; Figure 2 This is a schematic diagram of the Perceiver IO model structure, which is a task influence domain perception method for UAV cooperative task planning based on multimodal perception proposed in this invention. Detailed Implementation

[0015] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0016] refer to Figures 1-2 A method for collaborative mission planning of unmanned aerial vehicles (UAVs) based on multimodal perception includes the following steps: Collect visible light images, infrared images, lidar point clouds, flight status data, location information, and mission information during the UAV's mission execution process, and preprocess them to generate a multimodal perception dataset; The multimodal perception dataset is input into the task influence domain perception Perceiver IO model, which includes a multimodal perception encoding layer, a task query construction layer, an influence boundary expansion layer, an influence domain overlap attention layer, and a collaborative planning output layer. The task query set is generated through the multimodal perception encoding layer and the task query construction layer. Based on the task query set, the task influence domain corresponding to each task is constructed using the influence boundary extension layer, and a task influence domain set is generated. The overlapping attention layer of influence domains is invoked to perform pairwise correlation analysis on each task influence domain in the task influence domain set, determine the overlapping relationship between each task influence domain, and generate a set of influence domain overlapping relationships. Based on the set of overlapping influence domains, the collaborative execution relationships between tasks are determined, and collaborative task planning results are generated through the collaborative planning output layer. Based on real-time updated multimodal perception data, the task influence domain set, influence domain overlap relationship set, collaborative execution relationship, and collaborative task planning results are updated synchronously, and the UAV collaborative task planning results are output.

[0017] In this embodiment, the preprocessing specifically includes time synchronization, coordinate unification, anomaly removal, and standardization. The generation process of the multimodal perception dataset is as follows: During the UAV's mission, visible light images, infrared images, LiDAR point clouds, flight status data, location information, and mission information are collected. The resolution of the visible light images is 1920×1080, and the acquisition frequency is 30 frames per second. The resolution of the infrared images is 640×512, and the acquisition frequency is 25 frames per second. The acquisition frequency of the LiDAR point clouds is 10 frames per second. The acquisition frequency of the flight status data and location information is 20 times per second. The flight status data includes flight speed, flight direction, and flight altitude. The mission information includes mission identifier, mission execution content, mission execution area, and mission execution time. Based on the time information corresponding to each data, visible light images, infrared images, LiDAR point clouds, and flight status data with a time difference of no more than 50 milliseconds are... Location and task information are mapped to the same time to achieve time synchronization. Using the coordinate system corresponding to the location information as a reference, coordinate transformation is performed on the spatial position in the LiDAR point cloud, and the visible light image, infrared image, and LiDAR point cloud are transformed to the corresponding spatial position to achieve coordinate unification. Anomaly removal is performed on the visible light image, infrared image, LiDAR point cloud, flight status data, location information, and task information after time synchronization and coordinate unification, removing data with missing time information, duplicate data, missing images, missing LiDAR point cloud data, and abnormal flight status data and location information. Standardization processing is performed on the visible light image, infrared image, LiDAR point cloud, flight status data, location information, and task information after anomaly removal to unify the data format and numerical range of each data, generating a multimodal perception dataset.

[0018] In this embodiment, the generation of the task query set specifically includes: The multimodal sensing dataset is input into the task influence domain sensing Perceiver IO model, and the multimodal sensing dataset is sent to the multimodal sensing encoding layer for encoding processing. The improvements to the task-influence-aware Perceiver IO model specifically include: In terms of task query construction, traditional Perceiver IO uses a shared latent query vector to interact with all input features. Different queries mainly rely on feature similarity to complete information aggregation, but lack task constraints. In UAV collaborative task planning scenarios, there are often multiple tasks with adjacent execution areas and similar flight states at the same time. Relying solely on feature similarity for querying can easily aggregate visible light images, infrared images, LiDAR point clouds, and flight state data corresponding to different tasks into the same query process, making the information boundaries between tasks gradually blurred. When constructing the task influence domain later, cross-referencing of perception information corresponding to different tasks can easily occur, affecting the accuracy of the task influence domain. To address the above issues, the task query construction layer incorporates task identifier, task execution content, task execution area, and task execution time into the task query construction process. This ensures that each task query always interacts around the multimodal coding features corresponding to the current task, avoiding information mixing between different tasks, improving the accuracy of the correspondence between task queries and the current task, and providing a stable data foundation for the construction of the task influence domain. II. In terms of multimodal perception coding, although traditional Perceiver IO can uniformly process different modal inputs such as images, point clouds, and structured data, its coding goal is mainly to obtain a unified feature representation. The information fusion between different modalities is usually completed in the same way. For UAV collaborative mission planning, the roles played by different modalities are significantly different. For example, visible light images mainly reflect environmental texture, infrared images reflect target thermal features, lidar point clouds reflect spatial structure, flight status data and position information reflect the UAV's motion status, and mission information determines the subsequent planning goals. If only unified coding is performed, the different modalities are easily diluted with each other, making it difficult to fully reflect the important role of each modality in mission planning. The multimodal perception coding layer performs independent coding on each modal data and uniformly maps it to the same feature space. Then, it combines the task query to extract the multimodal coding features corresponding to the current task, so that different modalities can maintain their own information characteristics and can be fused around the current task, improving the expressive ability of multimodal coding features for mission planning. Thirdly, regarding the construction of the task influence domain, traditional Perceiver IO typically uses classification, detection, or prediction as its output target. Its output primarily corresponds to the target itself, lacking a continuous expression of the target's influence range. In the process of UAV collaborative task planning, what truly affects whether multiple tasks can be executed collaboratively is not the UAV's current position, but rather the spatial range that the UAV may continuously occupy during its current flight state and task execution. If analysis is based solely on the current position, changes in UAV speed, flight direction, or task execution method can easily lead to a lag in subsequent collaborative relationship analysis, failing to reflect potential spatial overlaps between tasks in advance. Therefore, the influence boundary extension layer, based on task queries and combined with location information, flight state data, and task information, continuously expands the task boundary and forms a stable task influence domain through multiple rounds of boundary updates. This expands the model output from a single task state to the task influence range, improving the task spatial representation capability and providing a unified foundation for the analysis of influence domain overlap relationships. IV. Regarding the analysis of overlapping influence domains, traditional Perceiver IO typically treats each query as an independent prediction object for inference, lacking a mechanism for continuous correlation analysis between queries. Therefore, while it can obtain the results corresponding to each task, it is difficult to further analyze the mutual influence relationships between multiple tasks. In the process of UAV collaborative task planning, what truly determines whether a task needs to be coordinated is whether there is continuous overlap between the influence domains of multiple tasks, rather than the state of each task itself. Therefore, simply obtaining the results of each task is still insufficient to directly support collaborative planning. To solve the above problems, the influence domain overlap attention layer treats the task influence domain as the correlation object, performs pairwise correlation analysis on each task influence domain, and generates a set of influence domain overlap relationships based on the common coverage spatial location and corresponding time. This extends the model from single-task inference to multi-task correlation inference, improving the ability to analyze spatial relationships between multiple tasks. V. Regarding collaborative planning output, traditional Perceiver IO outputs typically correspond to a single query, lacking the ability to generate business decisions based on the relationships between multiple queries. In the process of UAV collaborative task planning, even if it is determined that multiple tasks overlap spatially, it is still necessary to combine task type, task priority, and flight status to further determine different collaborative strategies such as joint execution, staggered execution, or spatial avoidance between different tasks. Therefore, traditional Perceiver IO cannot directly meet the needs of collaborative task planning. To address the above issues, the collaborative planning output layer uses the set of overlapping relationships of influence domains as a basis, combines task information and flight status data to generate joint execution relationships, staggered execution relationships, or spatial avoidance relationships, and further generates corresponding collaborative task planning results. This allows the model output to directly correspond to UAV collaborative task planning, improving the applicability of the model output to actual task planning. VI. Regarding dynamic updates, traditional Perceiver IO typically employs a one-input, one-output inference approach. When the input data changes, the entire inference process often needs to be re-executed. In UAV collaborative task planning, multiple UAVs fly continuously, and their position information, flight status data, and task information are constantly changing. If all tasks are recalculated for each change, not only is the computational load large, but it can also easily affect the real-time update capability of the collaborative planning results. Therefore, this model combines real-time updated multimodal perception data and only updates the task influence domain corresponding to the changed task. It also simultaneously adjusts the overlapping relationship of the influence domains of the affected tasks, the collaborative execution relationship, and the collaborative task planning results. This eliminates the need to recalculate all tasks, ensuring real-time updates of the collaborative planning results while improving the model's computational efficiency and responsiveness. The task-influence-domain-aware Perceiver IO model comprises a multimodal perception encoding layer, a task query construction layer, an influence boundary expansion layer, an influence domain overlap attention layer, and a collaborative planning output layer. The multimodal perception encoding layer receives a multimodal perception dataset, encodes visible light images, infrared images, LiDAR point clouds, flight status data, location information, and task information, and maps them uniformly to the same feature space to generate a multimodal encoded feature set. The task query construction layer, connected to the multimodal perception encoding layer, receives the multimodal encoded feature set and generates a task query set based on task information. The influence boundary expansion layer, connected to the task query construction layer, receives the task query set and performs boundary expansion calculations based on location information, flight status data, and task information to generate a task influence domain set. The influence domain overlap attention layer, connected to the influence boundary expansion layer, receives the task query set and generates a task query set based on task information. The system sets an impact domain set and performs correlation analysis on the impact domains of each task to generate an impact domain overlap relationship set. The collaborative planning output layer is connected to the impact domain overlap attention layer and the task query construction layer, respectively. It receives the impact domain overlap relationship set and determines the collaborative execution relationship by combining task information and flight status data. The collaborative execution relationship includes joint execution relationship, staggered execution relationship and spatial avoidance relationship, and generates collaborative task planning results. When the multimodal perception dataset is updated, the updated multimodal perception dataset is updated according to the data stream of multimodal coding feature set, task query set, task impact domain set, impact domain overlap relationship set and collaborative task planning results. Only the data corresponding to the changed tasks are calculated, and the impact domain overlap relationship set and collaborative task planning results corresponding to the affected tasks are updated synchronously. The training process of the task impact domain awareness Perceiver IO model includes: collecting historical UAV collaborative task data, which includes visible light images, infrared images, LiDAR point clouds, flight status data, location information, task information, and corresponding collaborative task planning results; aligning the execution time of the historical UAV collaborative task data, removing outlier data, completing missing data, and performing unified encoding processing to generate a model training dataset; and inputting the model training dataset into the task impact domain awareness Perceiver. The IO model employs a multimodal sensing encoding layer to generate a multimodal encoding feature set, a task query construction layer to generate a task query set, an influence boundary expansion layer to generate a task influence domain set, an influence domain overlap attention layer to generate an influence domain overlap relationship set, and a collaborative planning output layer to generate collaborative task planning results. The generated collaborative task planning results are compared with the collaborative task planning results corresponding to the model training dataset to construct the model's total loss. This total loss includes the planning result prediction loss, task influence domain constraint loss, influence domain overlap relationship constraint loss, collaborative execution relationship constraint loss, and model parameter regularization loss. Specifically, the planning result prediction loss constrains the prediction error between the model's output collaborative task planning results and the training labels; the task influence domain constraint loss ensures that the spatial coverage of the task influence domain matches the actual task influence range; the influence domain overlap relationship constraint loss ensures that the influence domain overlap relationships generated by the model match the actual task overlap relationships; the collaborative execution relationship constraint loss ensures that the collaborative execution relationships generated by the model match the actual collaborative execution relationships; and the model parameter regularization loss suppresses overfitting during model training and improves the model's generalization ability. Backpropagation is then performed based on the total model loss to update the task influence domain sensing perceiver. The network parameters of each layer of the IO model are updated, and the model training dataset is input for iterative training. After each update of the model parameters, the total loss of the model is calculated using the validation dataset. When the change rate of the total loss of the model corresponding to the validation dataset is less than 1% for ten consecutive iterations, the model training ends, and the trained task-influence-aware Perceiver IO model is obtained. The multimodal sensing coding layer encodes visible light images, infrared images, lidar point clouds, flight status data, location information, and mission information respectively, generates coding features, and performs mapping processing according to a unified feature dimension to form a multimodal coding feature set. The multimodal coding feature set is input into the task query construction layer, the task information of each task is traversed, the multimodal coding features corresponding to each task are extracted, and the corresponding associations are made according to the task identifier to form a task candidate feature set; The formation process of the task candidate feature set is as follows: The task query construction layer reads the multimodal encoded feature set and traverses each task sequentially according to the task identifier in the task information; for the current task, the corresponding task execution area and task execution time are read. The task execution area ranges from 10,000 m² to 250,000 m², and the task execution time lasts from 2 to 10 seconds; based on the task execution area, the corresponding visible light image, infrared image, and lidar point cloud are extracted. The ground resolution of the visible light image is 2 cm / pixel to 10 cm / pixel, the temperature range of the infrared image is -20℃ to 120℃ with a temperature measurement error of no more than ±2℃, and the lidar point cloud density is 100 points / m². ~800 points / m²; extract corresponding flight status data and position information based on mission execution time. Flight status data includes flight speed 5m / s~20m / s, flight altitude 30m~150m, and flight direction 0°~360°. Position information error is no greater than 0.5m. Read the corresponding visible light image coding features, infrared image coding features, lidar point cloud coding features, flight status data coding features, position information coding features, and mission information coding features, and associate them according to the mission identifier to form mission candidate features. Complete the construction of mission candidate features for each mission in the same way, and collect them according to the mission identifier to generate a set of mission candidate features. The task query construction layer uses the task information corresponding to each task as the query vector, performs interactive query processing on the multimodal encoded features in the task candidate feature set, and generates task query features. The generation of task query features specifically includes: the task query construction layer sequentially reads each task candidate feature in the task candidate feature set and reads the corresponding task information, and constructs a task query vector together with the task identifier, task execution content, task execution area, and task execution time; the task execution area is 10,000m² to 250,000m², and the task execution time lasts 2s to 10s; the task query construction layer uses the task query vector to perform interactive query processing with the multimodal coding features in the task candidate features: the task query construction layer uses the task query vector to sequentially access the visible light image coding features, infrared image coding features, lidar point cloud coding features, flight status data coding features, and position information coding features in the task candidate features; the spatial position deviation, time deviation, flight speed deviation, flight altitude deviation, and flight direction deviation between the task query vector and each coding feature are calculated respectively, wherein the spatial position deviation does not exceed 5m, the time deviation does not exceed 100ms, the flight speed deviation does not exceed 2m / s, the flight altitude deviation does not exceed 10m, and the flight direction deviation does not exceed 15°. For coding features that meet the deviation range, the corresponding coding information is read and combined according to the task identifier to form the task query feature; The query features of each task are grouped according to the task identifier, and a mapping relationship between each query feature and the task is established to generate a task query set.

[0019] In this embodiment, the generation of the task influence domain set specifically includes: Input the task query set into the influence boundary extension layer, read each task query in the task query set in sequence, and extract the location information, flight status data and task information corresponding to each task query. The influence boundary expansion layer takes each task query as the center, performs boundary expansion calculation based on the corresponding location information, flight status data and task information, generates the first expansion result corresponding to each task query, and uses each first expansion result as the initial input for the next round of boundary expansion calculation; The specific process of boundary expansion calculation is as follows: The boundary expansion layer reads the location information queried by the current task and establishes the initial boundary based on the current location; it reads flight status data and determines the boundary expansion direction and distance based on flight speed, flight direction, and flight altitude, with the flight direction serving as the boundary expansion direction; when the flight speed is 5m / s to 10m / s, the boundary expansion distance is 5m to 20m; when the flight speed is 10m / s to 15m / s, the boundary expansion distance is 20m to 35m; when the flight speed is 15m / s to 20m / s, the boundary expansion distance is 35m to 50m; the boundary expansion distance is further corrected based on the flight altitude: when the flight altitude is 30m to 60m, the boundary expansion distance remains unchanged; when the flight altitude is 60m to 100m, the boundary expansion distance is increased by 2m to 5m; when the flight altitude is 100m to 150m, the boundary expansion distance is increased by 5m to 10m; the task information is read, and the boundary expansion is adjusted according to the task execution area and task execution content. The scope is defined as follows: when the task execution area is 10,000 m² to 50,000 m², the boundary expansion range remains near the task execution area; when the task execution area is 50,000 m² to 150,000 m², the boundary expansion range extends outwards from the task execution area; when the task execution area is 150,000 m² to 250,000 m², the boundary expansion range covers the entire task execution area and its surrounding areas; when the task execution content is an inspection task, the boundary expansion range extends continuously along the task execution path; when the task execution content is a search task, the boundary expansion range extends evenly around the task execution area; when the task execution content is a transportation task, the boundary expansion range extends preferentially along the task flight direction; based on the boundary expansion direction, boundary expansion distance, and boundary expansion range, the initial boundary is expanded to generate the first expansion result; after generating the first expansion result for each task query, each first expansion result is saved and used as the initial input for the next round of boundary expansion for the corresponding task query; The influence boundary expansion layer continues to perform boundary expansion calculations based on the first expansion result, and updates the expansion result obtained in this round of boundary expansion calculations with the expansion result obtained in the previous round of boundary expansion calculations accordingly. The boundary expansion calculations are performed continuously until the expansion results obtained in two consecutive rounds of boundary expansion calculations no longer change, and the task influence domain corresponding to each task is generated. The generation of the task influence domain specifically includes: the influence boundary expansion layer reads the expansion results of each task and uses these results as input for boundary expansion, continuing to perform boundary expansion according to the boundary expansion method, and updating the expansion results; comparing the updated expansion results with the original expansion results, and sequentially calculating position changes, coverage area changes, and boundary point changes, wherein, when the position change is greater than 0.5m, boundary expansion continues; when the position change is not greater than 0.5m, coverage area changes are further compared, and when the coverage area change is greater than 10m², boundary expansion continues; when the coverage area change is not greater than 10m², boundary point changes are further compared, and when the number of boundary points increases by more than 5, boundary expansion continues; when the number of boundary points increases by no more than 5, boundary expansion stops, and the expansion results are determined as the task influence domain of the corresponding task. Organize the task influence domains corresponding to each task in a unified manner, and establish the correspondence between each task influence domain and the task according to the task identifier; The influence domains of each task are grouped and processed according to the task identifier to generate a set of task influence domains.

[0020] In this embodiment, the generation of the set of overlapping influence domain relationships specifically includes: Input the set of task influence domains into the influence domain overlap attention layer, read each task influence domain in sequence according to the task identifier, and combine each task influence domain with the other task influence domains in pairs to generate task influence domain combinations. The overlapping influence domain attention layer reads the combination of influence domains of each task, performs synchronous processing on the spatial location and corresponding time contained in the two task influence domains, and generates the corresponding arrangement result. The process for generating the corresponding arrangement result is as follows: The overlapping attention layer reads the two task influence domains in the task influence domain combination, and reads the spatial positions and corresponding times contained in the two task influence domains; the spatial positions of the two task influence domains are read sequentially from earliest to latest according to the corresponding time. When the two task influence domains have the same corresponding time, the spatial positions under the corresponding time are arranged accordingly; when the corresponding times of the two task influence domains are inconsistent, the spatial positions corresponding to the later corresponding times of the task influence domain with the earlier corresponding time are read again until the two task influence domains have the same corresponding time; the corresponding arrangement process is repeated until the spatial positions of all corresponding times are arranged, and the corresponding arrangement result is generated; the corresponding time is recorded at time intervals of 0.5 s, 1 s, or 2 s; when the UAV performs an inspection task, the corresponding time recording interval is 0.5 s; when performing a search task, the corresponding time recording interval is 1 s; when performing a transportation task, the corresponding time recording interval is 2 s; the spatial position is represented by three-dimensional coordinates, including X, Y, and Z coordinates; when RTK positioning is used, the spatial position positioning accuracy is 0.1. m; When using GNSS positioning, the spatial positioning accuracy is 0.5 m to 1 m; Each corresponding time corresponds to a set of spatial locations, and each set of spatial locations contains 20 to 200 spatial locations; The overlapping attention layer of the influence domain arranges the spatial locations in the order of X, Y, and Z coordinates, and maintains that the same corresponding time corresponds to a set of spatial locations; After completing all corresponding time correspondences, the spatial location sets are organized in order from earliest to latest corresponding time to form the corresponding arrangement result; The overlapping attention layer of the influence domain performs overlapping association calculations based on the corresponding arrangement results, determines the spatial location that the two task influence domains jointly cover at each corresponding time, and continuously associates the jointly covered spatial locations according to the corresponding time to generate the overlapping part corresponding to each task influence domain combination; The formation of the overlapping portion corresponding to the combination of influence domains of each task specifically includes: the influence domain overlap attention layer reads the corresponding arrangement results, reads the spatial positions corresponding to the two task influence domains in order from earliest to latest according to the corresponding time, and compares the spatial positions at the corresponding time one by one; when the coordinate difference between two spatial positions does not exceed 1 m, the corresponding spatial positions are determined as common coverage spatial positions; otherwise, the next spatial position is read and compared until all spatial positions at the current corresponding time are compared; after all common coverage spatial positions at all corresponding times are determined, the common coverage spatial positions are read in order from earliest to latest according to the corresponding time; when there are common coverage spatial positions at adjacent corresponding times, and the coordinate difference between the corresponding common coverage spatial positions does not exceed 2 m, the corresponding common coverage spatial positions are connected in sequence; the connection process is repeated until all common coverage spatial positions are connected to generate the overlapping portion; Based on the overlapping parts corresponding to each task influence domain combination, determine the overlapping relationship between the corresponding task influence domains, and establish the correspondence between each overlapping relationship and the corresponding task influence domain combination; The specific process for establishing the correspondence between each overlapping relationship and the corresponding task influence domain combination is as follows: The influence domain overlap attention layer sequentially reads the overlapping part corresponding to each task influence domain combination, and reads the common coverage spatial location and corresponding time contained in the overlapping part; according to the corresponding time from early to late, the common coverage spatial location is read sequentially. When two UAVs performing line inspection tasks fly in the same direction, the difference in flight altitude does not exceed 10 m, and the flight speed is maintained at 5-8 m / s, the common coverage spatial location under the continuous corresponding time is determined as the same overlapping relationship; when two UAVs performing area search tasks fly in an intersecting state, and the flight altitude is maintained at 80-120 m, the common coverage spatial location under the continuous corresponding time is determined as the same overlapping relationship; when the difference in flight altitude between two UAVs performing material transportation tasks exceeds 30 m, the flight direction changes, or there is no common coverage spatial location under the corresponding time, the current overlapping relationship is ended, and a new overlapping relationship is established based on the subsequent common coverage spatial location; the process is repeated until all overlapping relationships are established; then, each overlapping relationship is established as a correspondence with the corresponding task influence domain combination one by one, generating the correspondence relationship; The overlapping relationships of each task's influence domain combination are collected and processed according to the task identifier, and the spatial location and corresponding time of each overlapping relationship are retained to generate a set of influence domain overlapping relationships.

[0021] In this embodiment, the generation of collaborative task planning results specifically includes: Input the set of overlapping relationships of the influence domain into the collaborative planning output layer, extract the task, spatial location and corresponding time corresponding to each overlapping relationship according to the task identifier, and arrange the overlapping relationships corresponding to the same task according to the corresponding time. Perform collaborative association calculations on the overlapping relationships after arrangement, and determine the common execution relationship, staggered execution relationship or spatial avoidance relationship between the tasks based on the common coverage spatial location and its corresponding time, and generate collaborative execution relationship; The process of generating collaborative execution relationships is as follows: The collaborative planning output layer reads the arranged overlapping relationships, obtains the task type and priority based on the task information, and obtains the flight direction, flight altitude, flight speed, and flight path based on the flight status data; when the overlapping relationships corresponding to two tasks overlap in time, both tasks are line inspection, both have a flight speed of 5-8 m / s, both have a flight altitude of 40-60 m, and both have the same flight direction, a common execution relationship is generated; when the overlapping relationships corresponding to two tasks overlap in time, the task types are area search and material transportation, and the task priorities are level one and level two, respectively, a staggered execution relationship is generated; when the overlapping relationships corresponding to two tasks overlap in time, but the flight directions are different, or the flight altitudes are 40-60 m and 80-120 m, respectively, or the flight paths are different, a spatial avoidance relationship is generated; the collaborative execution relationships corresponding to all overlapping relationships are generated sequentially, thus generating the collaborative execution relationship. For joint execution relationships, the entry order, execution content, and exit order of the corresponding tasks are determined according to the task information; The generation of entry order, task execution content, and exit order specifically includes: the collaborative planning output layer reads the task information corresponding to the common execution relationship, and arranges the corresponding tasks in order of priority (Level 1, Level 2, and Level 3); when multiple tasks have the same priority, they are arranged in order of entry from earliest to latest according to their corresponding time; when the corresponding times are the same, they are arranged in order of entry from lowest to highest flight altitude, with route inspection tasks at flight altitudes of 40–60 m entering first, and area search tasks at flight altitudes of 80–120 m entering later; the task execution content is arranged according to the entry order, with route inspection tasks and area search tasks being executed together, and route inspection being executed first, followed by area search; when route inspection tasks and material transportation tasks are executed together, route inspection is executed first, followed by material transportation; after all task execution content is arranged, the exit order is arranged in reverse order of entry, generating the entry order, task execution content, and exit order. To address the staggered execution relationship, adjust the entry or exit time of the corresponding tasks so that the corresponding times of each task are staggered. The specific process for adjusting the entry or exit time of corresponding tasks is as follows: The collaborative planning output layer reads the task information and corresponding time corresponding to the time-sharing relationship, and arranges the tasks from high to low priority; when the task priorities are level one and level two respectively, the corresponding time of the level one task remains unchanged, and the entry time of the level two task is adjusted backward by 5 seconds; when the priorities of two tasks are the same, the tasks are arranged from early to late according to the corresponding time, and the entry time of the later-entering task is adjusted backward by 3 seconds; when the corresponding times still overlap after the entry time adjustment, the entry time remains unchanged, and the exit time of the first-exiting task is advanced by 2 seconds; after all the corresponding time adjustments are completed, the corresponding time is updated according to the adjusted entry and exit times to generate the adjusted entry or exit time; In response to spatial avoidance relationships, the flight direction, flight altitude, or flight path of the corresponding mission is adjusted based on location information and flight status data; The adjustment of flight direction, flight altitude, or flight path specifically includes: the collaborative planning output layer reads the position information and flight status data corresponding to the spatial avoidance relationship, determines the position of the corresponding task based on the position information, and obtains the flight direction, flight altitude, and flight path based on the flight status data; when the corresponding positions of two tasks are located in the same flight direction, the flight direction of the later-entering task is adjusted by 15°; when the corresponding positions still overlap after the flight direction adjustment, the flight altitude of the later-entering task is adjusted from 50 m to 70 m; when the corresponding positions still overlap after the flight altitude adjustment, the flight path is offset by 20 m; after completing the adjustment of all tasks corresponding to the spatial avoidance relationship, the flight direction, flight altitude, or flight path of the corresponding task is updated; The adjustment results will be associated with the corresponding task identifiers and collaborative execution relationships; The collaborative execution relationships and adjustment results of each task are uniformly arranged according to the corresponding time, and tasks that do not have overlapping relationships are retained according to their original task information to generate collaborative task planning results.

[0022] In this embodiment, the generation of UAV collaborative task planning results specifically includes: Acquire current multimodal perception data, extract location information, flight status data and task information corresponding to each task, and compare them with the task information corresponding to the collaborative task planning results to determine the tasks that have changed; Based on the changed tasks, the task influence domain of the corresponding tasks is regenerated, and the affected tasks that share a common spatial location with the updated task influence domain are identified. The process for generating affected tasks is as follows: read the location information, flight status data, and task information corresponding to the changed task; regenerate the task influence domain of the corresponding task based on the updated location information, flight status data, and task information; compare the regenerated task influence domain with the task influence domains of the other tasks one by one to determine the tasks that share a common spatial location; mark the tasks that share a common spatial location as affected tasks, and summarize all affected tasks to generate affected tasks. For the affected tasks, redetermine the set of overlapping relationships of the affected domains, and redetermine the corresponding joint execution relationships, staggered execution relationships, or spatial avoidance relationships based on the updated set of overlapping relationships of the affected domains; The process of redetermining the corresponding joint execution relationships, staggered execution relationships, or spatial avoidance relationships specifically includes: reading the updated set of overlapping influence domain relationships, and sequentially obtaining the common coverage spatial location, common coverage time, and corresponding task information for each affected task; when the common coverage time reaches 15 seconds and the task types in the corresponding task information are consistent, redetermining the corresponding tasks as having a joint execution relationship; when the common coverage spatial location is consistent, the common coverage time is less than 15 seconds, and the task types in the corresponding task information are different but the flight altitude is the same, redetermining the corresponding tasks as having a staggered execution relationship; when the common coverage spatial location is consistent, the common coverage time reaches 15 seconds, and the flight altitude difference is less than 8 meters, redetermining the corresponding tasks as having a spatial avoidance relationship; after redetermining all the corresponding relationships of the affected tasks, generating the updated joint execution relationships, staggered execution relationships, or spatial avoidance relationships; Based on the newly determined joint execution relationship, staggered execution relationship, or spatial avoidance relationship, the entry time, exit time, flight direction, flight altitude, or flight path of the affected mission are adjusted to generate updated adjustment results; The updated adjustment results are generated as follows: The redefined joint execution relationships, staggered execution relationships, or spatial avoidance relationships are read sequentially. When the corresponding tasks have a joint execution relationship, the entry and exit times of the corresponding tasks are adjusted synchronously according to the redefined joint execution relationship. When the corresponding tasks have a staggered execution relationship, the entry time of the later-entering task is delayed by 5 seconds, and the corresponding exit time is adjusted synchronously. When the corresponding tasks have a spatial avoidance relationship, the flight direction of the later-entering task is adjusted by 15°. If the corresponding tasks still share a common spatial location after the flight direction adjustment, the flight altitude is adjusted from 50 m to 70 m. If the corresponding tasks still share a common spatial location after the flight altitude adjustment, the flight path is offset by 20 m. After completing the adjustments to the entry time, exit time, flight direction, flight altitude, and flight path for all affected tasks, the updated adjustment results are generated. The updated adjustment results are merged with the original collaborative task planning results corresponding to the unaffected tasks, and rearranged according to the corresponding time to generate the updated UAV collaborative task planning results.

[0023] Example 1: To verify the feasibility of this invention in practice, it was applied to an intelligent inspection scenario of a power transmission line in a mountainous area. The inspection area covered approximately 65 km², with a total line length of about 48 km and 126 transmission towers along the line. The terrain included mountains, hills, and valleys, with ground elevation ranging from 180 m to 1260 m. The vegetation coverage was approximately 72%, with some areas experiencing tree cover, mountain obstruction, and significant slope undulations. During the inspection, 12 drones were deployed to collaboratively perform the task. Each drone was equipped with a visible light camera and an infrared camera. Thermal imaging cameras and lidar are used, with flight altitudes set at 60m to 120m and flight speeds controlled at 8m / s to 15m / s. Each drone is responsible for 8 to 12 inspection tasks. Each inspection task includes the task area, inspection route, estimated execution time, and inspection target. There is some spatial and temporal overlap between different tasks. If the fixed plan is followed directly, multiple drones may enter adjacent areas, cross routes, or conduct duplicate inspections. Therefore, the collaborative task planning needs to be dynamically adjusted based on the real-time operating status of the drones and the task execution status.

[0024] After the inspection begins, each UAV continuously collects visible light images, infrared images, LiDAR point clouds, flight status data, location information, and task information. Following a unified timeline, time alignment, outlier removal, missing data completion, and unified encoding are performed to form a multimodal perception dataset. This dataset is then input into the task-influence-domain perception Perceiver IO model. The multimodal perception encoding layer encodes the tower appearance features in the visible light images, temperature distribution features in the infrared images, spatial structure features in the LiDAR point clouds, as well as the flight status data, location information, and task information, mapping them uniformly to the same feature space to generate a multimodal coded feature set. The task query construction layer generates a task query set based on the task number, task area, inspection target, estimated execution time, and the corresponding UAV, ensuring each task corresponds to an independent query vector and establishing a correspondence between tasks and multimodal perception features.

[0025] The influence boundary extension layer receives the task query set and, in conjunction with the UAV's current position, flight direction, flight speed, flight altitude, and task execution range, performs extension calculations on the execution boundaries of each inspection task, generating a corresponding set of task influence domains. Each task influence domain contains information on the task execution area, the boundary extension area, and the task execution direction. For example, when a UAV performs a pole inspection task and its flight direction deviates, the model synchronously adjusts the spatial coverage of the task influence domain based on the current flight direction and flight speed, so that the task influence domain can reflect the area that the UAV may affect during actual execution. When a certain area is obstructed by trees, causing the UAV to need to adjust its flight path, the corresponding task influence domain changes synchronously, providing basic data for subsequent task overlap analysis.

[0026] Subsequently, the overlapping influence domain attention layer receives the set of task influence domains and performs spatial and temporal correlation analysis on all task influence domains to generate a set of influence domain overlap relationships. During the implementation process, a total of 118 sets of task influence domains were generated, of which 31 sets of spatial overlap relationships, 18 sets of temporal overlap relationships, and 9 sets of influence relationships that have both spatial and temporal overlap. For task combinations with an overlap area exceeding 35m² and an expected entry time interval of less than 25s, the model further analyzes the degree of influence between tasks and calculates the corresponding influence relationships based on the current flight status and task execution status of each UAV, providing a basis for subsequent collaborative task planning.

[0027] The collaborative planning output layer receives the set of overlapping influence domain relationships and the set of task queries, respectively. It performs a comprehensive analysis of the overlapping influence domain relationships. When two drones have the same target and inspection direction, a joint execution relationship is determined. When the task areas overlap but the expected arrival times are different and the sequential execution conditions can be met, a staggered execution relationship is determined. When there is a risk of overlapping flight paths, a spatial avoidance relationship is determined. Based on the collaborative execution relationship, the corresponding collaborative task planning results are generated. For example, when two drones are inspecting adjacent power transmission towers, due to the obstruction of mountains, the flight speed of one drone decreases from 12m / s to 8m / s, causing the expected entry time of the two tasks into the overlapping area to change. The model re-analyzes the corresponding task influence domain and the overlapping influence domain relationship, adjusts the original joint execution relationship to a staggered execution relationship, and updates the corresponding task entry time and flight path simultaneously. When the other drone temporarily avoids the high-voltage line protection area, the task influence domain expands to the west by about 18m. The model regenerates the overlapping influence domain relationship and adjusts the original planned path to a spatial avoidance path, so that the two drones can complete the inspection task while maintaining a safe distance.

[0028] During the inspection process, the multimodal perception data was continuously updated. The system dynamically updated the data stream according to the multimodal coding feature set, task query set, task influence domain set, influence domain overlap relationship set, and collaborative task planning results. Only for tasks that changed, the corresponding task influence domain and influence domain overlap relationship were recalculated, while tasks that did not change continued to maintain the original collaborative task planning results. During the entire inspection period, approximately 36,800 visible light images, approximately 35,200 infrared images, and approximately 1.62 million LiDAR point cloud frames were processed, forming approximately 256,000 multimodal coding feature sets, 118 task query sets, 118 task influence domains, and 58 influence domain overlap relationships. A total of 214 collaborative task planning updates were completed, of which 126 updates were triggered by changes in flight status, 53 updates were triggered by changes in task status, and 35 updates were triggered by changes in environment. Each update directly performed local adjustments for the affected tasks, enabling the UAV to maintain a collaborative execution state throughout the inspection process in complex mountainous areas and complete continuous collaborative operations in all inspection areas.

[0029] Table 1. Performance Comparison Results of Multi-UAV Collaborative Task Planning

[0030] As shown in Table 1, in the scenario of multi-UAV collaborative inspection of power transmission lines in the same mountainous area, the method of the present invention outperforms the rule-based collaborative planning method and the traditional deep learning-based collaborative planning method in terms of collaborative task planning accuracy, task conflict rate, average time for dynamic adjustment, single task planning calculation time, and UAV collaborative task completion rate. The data correspond to each other, indicating that the present invention can ensure planning accuracy while taking into account dynamic adjustment efficiency and collaborative execution capability.

[0031] In terms of the accuracy of collaborative task planning, the rule-based collaborative planning method achieved 78.6%, the traditional deep learning-based collaborative planning method achieved 84.3%, and the present invention achieved 88.7%, which is 10.1 percentage points and 4.4 percentage points higher than the other two methods, respectively. The reason for this is that the present invention first integrates visible light images, infrared images, lidar point clouds, flight status data, location information, and task information to construct a unified multimodal feature. Then, it uses the task influence domain perception Perceiver IO model to establish a set of task influence domains and a set of influence domain overlap relationships. This allows the model to simultaneously analyze the spatial and temporal correlations between different tasks, thus making the collaborative task planning results more accurate.

[0032] In terms of task conflict incidence, the rule-based collaborative planning method has an incidence of 14.8%, the traditional deep learning-based collaborative planning method has an incidence of 9.6%, and this invention reduces it to 5.2%, which is a reduction of 9.6 percentage points and 4.4 percentage points respectively. The main reason is that this invention does not only perform collision avoidance analysis based on the current position of the UAV, but further establishes the task influence domain and determines the joint execution relationship, staggered execution relationship and spatial avoidance relationship based on the overlapping relationship of the influence domain. The collaborative relationship can be adjusted before the UAV enters the potential conflict area, thus effectively reducing task overlap, flight path conflict and repeated inspection.

[0033] In terms of the average time consumption for dynamic adjustment, the rule-based collaborative planning method has an average time consumption of 286ms, the traditional deep learning-based collaborative planning method has an average time consumption of 214ms, and the present invention reduces it to 147ms, which is a reduction of 139ms and 67ms respectively. Since the present invention only recalculates the corresponding task influence domain, influence domain overlap relationship and collaborative execution relationship for the affected task after the flight status, task status or environmental perception information changes, while the unchanged tasks continue to maintain the original planning results, the computational overhead caused by overall replanning is reduced and the efficiency of dynamic adjustment is improved.

[0034] In terms of single task planning computation time, the rule-based collaborative planning method takes 431ms, the traditional deep learning-based collaborative planning method takes 328ms, and this invention reduces it to 246ms, a reduction of 185ms and 82ms respectively. The reason for this is that this invention uses the Perceiver IO model to complete multimodal information fusion and adopts a corresponding calculation method between the task query set and the task influence domain set. It only analyzes the influence relationship between related tasks, avoiding repeated correlation calculations between all tasks. Therefore, it reduces the overall computational complexity in the collaborative task planning process and improves planning efficiency.

[0035] In terms of the completion rate of UAV collaborative tasks, the rule-based collaborative planning method achieved 81.4%, the traditional deep learning-based collaborative planning method achieved 86.8%, and the present invention achieved 89.3%, which is an improvement of 7.9 percentage points and 2.5 percentage points, respectively. Because the present invention can continuously update the task influence domain and the overlap relationship of the influence domain, and adjust the entry time, flight path and flight direction in real time according to the latest collaborative execution relationship, the UAV can complete the task collaboration more reasonably. Therefore, it reduces the task interruption caused by conflict adjustment, repeated execution and temporary detour, and improves the overall collaborative task completion rate.

[0036] As shown in Table 1, this invention achieves good performance in terms of collaborative task planning accuracy, task conflict rate, average dynamic adjustment time, single task planning calculation time, and UAV collaborative task completion rate. Specifically, the planning accuracy reaches 88.7%, the task conflict rate is reduced to 5.2%, the average dynamic adjustment time is shortened to 147ms, the single task planning calculation time is reduced to 246ms, and the collaborative task completion rate reaches 89.3%. These indicators collectively demonstrate that this invention, by constructing task influence domains, influence domain overlap relationships, and collaborative execution relationships, achieves unified perception, dynamic analysis, and local updates in the multi-UAV collaborative task planning process, making collaborative task planning more aligned with the actual needs of multi-UAV collaborative operations in complex environments.

[0037] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for cooperative mission planning of unmanned aerial vehicles (UAVs) based on multimodal perception, characterized in that, Includes the following steps: Collect visible light images, infrared images, lidar point clouds, flight status data, location information, and mission information during the UAV's mission execution process, and preprocess them to generate a multimodal perception dataset; The multimodal perception dataset is input into the task influence domain perception Perceiver IO model, which includes a multimodal perception encoding layer, a task query construction layer, an influence boundary expansion layer, an influence domain overlap attention layer, and a collaborative planning output layer. The task query set is generated through the multimodal perception encoding layer and the task query construction layer. Based on the task query set, the task influence domain corresponding to each task is constructed using the influence boundary extension layer, and a task influence domain set is generated. The overlapping attention layer of influence domains is invoked to perform pairwise correlation analysis on each task influence domain in the task influence domain set, determine the overlapping relationship between each task influence domain, and generate a set of influence domain overlapping relationships. Based on the set of overlapping influence domains, the collaborative execution relationships between tasks are determined, and collaborative task planning results are generated through the collaborative planning output layer. Based on real-time updated multimodal perception data, the task influence domain set, influence domain overlap relationship set, collaborative execution relationship, and collaborative task planning results are updated synchronously, and the UAV collaborative task planning results are output.

2. The UAV cooperative mission planning method based on multimodal perception according to claim 1, characterized in that, The preprocessing specifically includes time synchronization, coordinate unification, anomaly removal, and standardization.

3. The UAV cooperative mission planning method based on multimodal perception according to claim 1, characterized in that, The generation of the task query set specifically includes: The multimodal sensing dataset is input into the task influence domain sensing Perceiver IO model, and the multimodal sensing dataset is sent to the multimodal sensing encoding layer for encoding processing. The multimodal sensing coding layer encodes visible light images, infrared images, lidar point clouds, flight status data, location information, and mission information respectively, generates coding features, and performs mapping processing according to a unified feature dimension to form a multimodal coding feature set. The multimodal coding feature set is input into the task query construction layer, the task information of each task is traversed, the multimodal coding features corresponding to each task are extracted, and the corresponding associations are made according to the task identifier to form a task candidate feature set; The task query construction layer uses the task information corresponding to each task as the query vector, performs interactive query processing on the multimodal encoded features in the task candidate feature set, and generates task query features. The query features of each task are grouped according to the task identifier, and a mapping relationship between each query feature and the task is established to generate a task query set.

4. The UAV cooperative mission planning method based on multimodal perception according to claim 1, characterized in that, The generation of the task influence domain set specifically includes: Input the task query set into the influence boundary extension layer, read each task query in the task query set in sequence, and extract the location information, flight status data and task information corresponding to each task query. The influence boundary expansion layer takes each task query as the center, performs boundary expansion calculation based on the corresponding location information, flight status data and task information, generates the first expansion result corresponding to each task query, and uses each first expansion result as the initial input for the next round of boundary expansion calculation; The influence boundary expansion layer continues to perform boundary expansion calculations based on the first expansion result, and updates the expansion result obtained in this round of boundary expansion calculations with the expansion result obtained in the previous round of boundary expansion calculations accordingly. The boundary expansion calculations are performed continuously until the expansion results obtained in two consecutive rounds of boundary expansion calculations no longer change, and the task influence domain corresponding to each task is generated. Organize the task influence domains corresponding to each task in a unified manner, and establish the correspondence between each task influence domain and the task according to the task identifier; The influence domains of each task are grouped and processed according to the task identifier to generate a set of task influence domains.

5. The UAV cooperative mission planning method based on multimodal perception according to claim 1, characterized in that, The generation of the set of overlapping influence domains specifically includes: Input the set of task influence domains into the influence domain overlap attention layer, read each task influence domain in sequence according to the task identifier, and combine each task influence domain with the other task influence domains in pairs to generate task influence domain combinations. The overlapping influence domain attention layer reads the combination of influence domains of each task, performs synchronous processing on the spatial location and corresponding time contained in the two task influence domains, and generates the corresponding arrangement result. The overlapping attention layer of the influence domain performs overlapping association calculations based on the corresponding arrangement results, determines the spatial location that the two task influence domains jointly cover at each corresponding time, and continuously associates the jointly covered spatial locations according to the corresponding time to generate the overlapping part corresponding to each task influence domain combination; Based on the overlapping parts corresponding to each task influence domain combination, determine the overlapping relationship between the corresponding task influence domains, and establish the correspondence between each overlapping relationship and the corresponding task influence domain combination.

6. The UAV cooperative mission planning method based on multimodal perception according to claim 1, characterized in that, The generation of the collaborative task planning results specifically includes: Input the set of overlapping relationships of the influence domain into the collaborative planning output layer, extract the task, spatial location and corresponding time corresponding to each overlapping relationship according to the task identifier, and arrange the overlapping relationships corresponding to the same task according to the corresponding time. Perform collaborative association calculations on the overlapping relationships after arrangement, and determine the common execution relationship, staggered execution relationship or spatial avoidance relationship between the tasks based on the common coverage spatial location and its corresponding time, and generate collaborative execution relationship; For joint execution relationships, the entry order, execution content, and exit order of the corresponding tasks are determined according to the task information; To address the staggered execution relationship, adjust the entry or exit time of the corresponding tasks so that the corresponding times of each task are staggered. In response to spatial avoidance relationships, the flight direction, flight altitude, or flight path of the corresponding mission is adjusted based on location information and flight status data; The adjustment results will be associated with the corresponding task identifiers and collaborative execution relationships; The collaborative execution relationships and adjustment results of each task are uniformly arranged according to the corresponding time, and tasks that do not have overlapping relationships are retained according to their original task information to generate collaborative task planning results.

7. The UAV cooperative mission planning method based on multimodal perception according to claim 1, characterized in that, The generation of the UAV collaborative task planning results specifically includes: Acquire current multimodal perception data, extract location information, flight status data and task information corresponding to each task, and compare them with the task information corresponding to the collaborative task planning results to determine the tasks that have changed; Based on the changed tasks, the task influence domain of the corresponding tasks is regenerated, and the affected tasks that share a common spatial location with the updated task influence domain are identified. For the affected tasks, redetermine the set of overlapping relationships of the affected domains, and redetermine the corresponding joint execution relationships, staggered execution relationships, or spatial avoidance relationships based on the updated set of overlapping relationships of the affected domains; Based on the newly determined joint execution relationship, staggered execution relationship, or spatial avoidance relationship, the entry time, exit time, flight direction, flight altitude, or flight path of the affected mission are adjusted to generate updated adjustment results; The updated adjustment results are merged with the original collaborative task planning results corresponding to the unaffected tasks, and rearranged according to the corresponding time to generate the updated UAV collaborative task planning results.

Images