Unmanned aerial vehicle cooperative tracking system based on multi-mode photoelectric fusion

Abstract

The invention discloses an unmanned aerial vehicle cooperative tracking system based on multi-mode photoelectric fusion, and particularly relates to the technical field of unmanned aerial vehicle cooperative control, a three-dimensional occupation graph and shielding body voxel distribution are constructed and dynamically updated, a three-dimensional view cone model is established based on the real-time pose and sensor parameters of each unmanned aerial vehicle, and the visible volume of each unmanned aerial vehicle is calculated. Obtaining a combined visible range volume and a combined blind area volume of the unmanned aerial vehicle cluster through set operation; according to the probability that the target falls into the combined blind area volume, the blind area uncertainty measurement and the target concealment comprehensive score, carrying out weighted fusion to generate a three-dimensional blind area risk comprehensive index; the method comprises the following steps: generating candidate cooperative trajectory schemes for multiple unmanned aerial vehicles, predicting joint visual field volume and joint blind area volume evolution under each scheme, constructing a blind area suppression cooperative efficiency comprehensive index for each candidate cooperative trajectory scheme, and selecting an optimal cooperative trajectory scheme for execution by taking the index as a target. Meanwhile, risks are monitored and online reconfiguration is triggered.

Description

Technical Field

[0001] This invention relates to the field of collaborative control technology for unmanned aerial vehicles (UAVs), and more specifically, to a collaborative tracking system for UAVs based on multimodal optoelectronic fusion. Background Technology

[0002] With the rapid development of drone technology, multi-drone collaboration has shown broad application prospects in target tracking, area surveillance, and disaster relief. However, in environments with overpasses and forest canopies, the presence of buildings, terrain, and other obstructions can easily create dynamic blind spots in the joint observation field of a drone swarm, leading to frequent loss of target tracking.

[0003] In existing technologies, multi-UAV cooperative tracking methods mainly focus on path planning and local field of view optimization, lacking systematic modeling and quantitative evaluation of the blind zone formation mechanism under 3D occlusion structures and multi-UAV frustum fusion. Traditional methods often rely on a single sensor or a simple field of view overlap strategy, failing to fully consider the target's hiding behavior preferences in the blind zone, the uncertainty of multi-source observations, and the prior information of the hidden space accumulated in historical missions. This leads to problems such as target loss or delayed re-acquisition in complex scenarios. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a UAV cooperative tracking system based on multimodal optoelectronic fusion to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] The UAV cooperative tracking system based on multimodal optoelectronic fusion includes the following modules:

[0007] The 3D situational awareness module is used to construct and dynamically update the 3D occupancy map and the voxel distribution of the occluded body. Based on the real-time pose and sensor parameters of each UAV, it establishes a 3D view frustum model, calculates the visible volume of each UAV, and then obtains the joint visible volume and joint blind zone volume of the UAV cluster through set operations.

[0008] The target risk quantification module is used to calculate the probability of a target falling into the joint blind zone volume based on the joint visible area volume and the joint blind zone volume, construct a blind zone uncertainty measure, and calculate the target's concealment comprehensive score by combining the concealment space prior field. Then, it generates a three-dimensional blind zone risk comprehensive index by weighted fusion of the probability of the target falling into the joint blind zone volume, the blind zone uncertainty measure, and the target's concealment comprehensive score.

[0009] The collaborative decision-making and trajectory optimization module is used to generate candidate collaborative trajectory schemes for multiple UAVs, predict the evolution of joint visible area volume and joint blind zone volume under each scheme, and construct a blind zone suppression collaborative effectiveness comprehensive index for each candidate collaborative trajectory scheme. The blind zone suppression collaborative effectiveness comprehensive index is generated by weighted fusion of the target-level blind zone risk average, key concealed space coverage insufficiency measure and collaborative execution cost comprehensive quantity, and the optimal collaborative trajectory scheme is selected for execution with the goal of minimizing the index, while monitoring risks and triggering online reconfiguration.

[0010] In a preferred embodiment, the 3D situational intelligent perception module includes a pose estimation subunit, a 3D occupancy modeling subunit, and a view frustum modeling and viewable field calculation subunit.

[0011] The pose estimation subunit is used to continuously estimate the pose of each UAV in the mission coordinate system using visual and inertial odometry, lidar odometry or fusion algorithms, and combine the estimated pose with the pre-calibrated sensor extrinsic matrix to uniformly project the multimodal observation data onto the world coordinate system or local map coordinate system.

[0012] The 3D occupancy modeling subunit is used to spatially discretize the task area using a 3D grid or voxel structure with a preset resolution, divide it into several voxel units, and maintain the occupancy probability, reflection intensity and material category attributes of each voxel unit based on observation evidence from multimodal sensors, and mark occluding voxels.

[0013] The frustum modeling and visible field of view computation subunit is used to build a 3D frustum model based on the real-time pose and sensor parameters of each UAV, calculate the visible and invisible volumes of each UAV, and obtain the joint visible field volume and joint blind zone volume of the UAV cluster through set operations.

[0014] In a preferred embodiment, the frustum modeling and visible field of view calculation subunit calculates the joint visible field of view volume and the joint blind zone volume in the following manner: based on the three-dimensional frustum model of each UAV, it traverses voxel by voxel along the direction of the virtual ray in the three-dimensional occupancy map. When the ray encounters an occluding voxel, the voxel and subsequent voxels are marked as invisible regions, and the voxels before the occluding voxel and within the effective detection range of the sensor are marked as visible volumes; the visible volumes of all UAVs are combined to obtain the joint visible field of view volume; the complement of the joint visible field of view volume in the task area is defined as the joint blind zone volume.

[0015] In a preferred embodiment, the target risk quantification module includes a target state estimation and probability representation subunit, a visibility mapping and blind spot determination subunit, and a hidden space prior modeling subunit.

[0016] The target state estimation and probability representation sub-unit obtains the target state estimate in a unified coordinate system and uses a particle set to discretize the uncertainty distribution of the target position.

[0017] The visibility mapping and blind zone determination subunit is used to map each particle sample to the joint visible volume and the joint blind zone volume. By statistically analyzing the ratio of the sum of the weights of particles falling into the joint blind zone volume to the sum of the total weights, the probability metric of the target falling into the joint blind zone volume is calculated.

[0018] The concealed space prior modeling subunit is used to construct a concealed space prior field based on historical mission data, and to calculate the target's comprehensive concealment score by weighting the weights of all particles according to the concealment weight of the voxel where the current particle sample is located.

[0019] In a preferred embodiment, the blind zone uncertainty measure is obtained by calculating the Shannon entropy of the binary probability distribution based on the probabilities of two mutually exclusive events: the target being within the joint visible volume and the target being within the joint blind zone volume.

[0020] In a preferred embodiment, the collaborative decision-making and trajectory optimization module includes a candidate trajectory generation and line-of-sight prediction subunit and a blind zone entropy assessment and cost construction subunit.

[0021] The candidate trajectory generation and vision prediction subunit is used to generate a set of smooth candidate trajectories that are feasible under dynamic and obstacle avoidance constraints for each UAV, and to predict the evolution trajectory of the joint visible volume and joint blind zone volume in the future planning time domain for each cooperative scheme composed of these candidate trajectories.

[0022] The blind zone entropy assessment and cost construction sub-unit is used to calculate the target-level average blind zone risk, the critical hidden space coverage insufficiency metric, and the comprehensive cost of collaborative execution for each candidate collaborative trajectory scheme.

[0023] In a preferred embodiment, the target-level blind zone risk average is obtained by averaging the three-dimensional blind zone risk comprehensive index of each target over time within the planning time domain and weighting it according to the target importance.

[0024] In a preferred embodiment, the critical concealment space coverage insufficiency metric is obtained by statistically analyzing the spatial intersection between voxel regions with high concealment weights and the predicted joint blind zone volume within the planning time domain, and then normalizing the results.

[0025] In a preferred embodiment, the comprehensive cost of collaborative execution is obtained by weighted fusion of trajectory deviation, attitude adjustment range, and formation reconstruction range.

[0026] In a preferred embodiment, the system further includes a system strategy self-evolution module, which is used to perform correlation analysis on the three-dimensional blind zone risk comprehensive index and the blind zone suppression synergy effectiveness comprehensive index, and to evaluate and adaptively update the weight coefficients in the target risk quantification module and the collaborative decision-making and trajectory optimization module.

[0027] The technical effects and advantages of this invention are as follows:

[0028] This invention, by constructing a 3D occupancy map and a view frustum fusion model, achieves for the first time the dynamic quantification of the joint field of view and blind zone volume of a UAV swarm, fundamentally changing the limitations of traditional methods that rely solely on two-dimensional field-of-view overlap or a single sensor perspective. The system deeply integrates the probability of a target falling into the blind zone, the uncertainty measure of blind zone judgment, and the concealment space preference based on historical data. The resulting 3D blind zone risk comprehensive index accurately characterizes the target's concealment tendency and loss risk in complex occlusion environments, thus providing a precise and forward-looking risk situation assessment for subsequent collaborative decision-making.

[0029] At the trajectory planning level, this invention innovatively introduces a comprehensive index for blind zone suppression and collaborative effectiveness. It incorporates target-level blind zone risk, insufficient coverage of key concealed spaces, and collaborative execution costs into a unified optimization framework. Through multi-step prediction and online reconfiguration mechanisms, the UAV swarm can proactively avoid blind zone resonance and suppress high-risk concealed areas. This system not only significantly improves the continuous target tracking capability in complex scenarios such as overpasses and forest canopies, but also possesses strategy self-evolution and parameter self-adaptation capabilities. It can learn from historical tasks to optimize weight settings, forming a collaborative tracking closed loop that becomes increasingly intelligent with use. Attached Figure Description

[0030] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0031] Figure 1 This is a schematic diagram of the structure of the UAV cooperative tracking system based on multimodal optoelectronic fusion according to the present invention;

[0032] Figure 2 This is a schematic diagram of the workflow of the UAV cooperative tracking system based on multimodal optoelectronic fusion according to the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1: The UAV cooperative tracking system based on multimodal optoelectronic fusion of the present invention has the following high-level workflow: Figure 2 As shown, its structure is as follows Figure 1 As shown, it includes the following modules:

[0035] The 3D situational awareness module is deployed in a multimodal optoelectronic fusion-based UAV collaborative tracking system. By establishing standard data interface coupling with the multimodal optoelectronic sensing units, flight control system, and ground control terminal on each UAV platform, it is used to perform continuous multimodal environmental perception, 3D occupancy map construction, and online modeling of the relationship between occlusion structure and view frustum in the mission area under a unified time base and coordinate system. This provides high-precision 3D occlusion field, joint view volume, and joint blind zone volume inputs for subsequent target risk quantification module and collaborative decision-making and trajectory optimization module.

[0036] In practical implementation, each UAV participating in collaborative tracking is equipped with at least a visible light camera and an infrared camera, and may also be equipped with depth cameras, lidar, and other sensors depending on mission requirements. The 3D situational awareness module acquires raw observation data collected by each sensor under a unified time base through an airborne communication link, and simultaneously obtains attitude angles, position coordinates, and velocity information at corresponding moments from the flight control system or inertial measurement unit. This module has an internal pose estimation subunit that uses visual and inertial odometry, lidar odometry, or a fusion algorithm of the two to continuously estimate the pose of each UAV in the mission coordinate system. The estimated pose is then combined with a pre-calibrated sensor extrinsic parameter matrix, and the multimodal observation data at each moment is uniformly projected onto the world coordinate system or local map coordinate system. This achieves precise alignment of observations from different UAVs and different sensors on the time and space axes, avoiding systematic deviations in subsequent occlusion judgment and field-of-view calculations caused by insufficient coordinate system unification and time synchronization.

[0037] The 3D occupancy modeling subunit in the 3D situational awareness module performs spatial discretization modeling of the task area, using a 3D grid of preset resolution to divide the task area into several voxel units. For each voxel unit, the 3D occupancy modeling subunit continuously accumulates observational evidence from multimodal sensors such as visible light, infrared, depth, and lidar. Based on the number of times the voxel is observed to have an object or be empty, the observation angle, and the sensor type, it maintains attributes such as its occupancy probability, reflection intensity, and material type. If a voxel region is consistently observed as a physical entity with a clear geometric outline at multiple times and from multiple viewpoints, it is marked as an occluded voxel.

[0038] Based on the construction and maintenance of the 3D occupancy map and the voxel distribution of occlusion objects, the 3D situational awareness module further maps the current observation capabilities of each UAV to a 3D visible volume in a unified coordinate system through the frustum modeling and visible field of view (VLA) computation subunit. This subunit establishes a corresponding 3D frustum model based on the real-time pose information of each UAV in the mission coordinate system, as well as the intrinsic parameters, installation attitude, field of view, and effective detection range of onboard visible light cameras, infrared cameras, depth cameras, and lidar sensors. It abstracts the sensing range of each sensor into a set of virtual rays radiating outwards in a cone or fan shape from the UAV's position in space. In the 3D occupancy map, the frustum modeling and VLA computation subunit traverses voxels along each virtual ray. When a virtual ray encounters a voxel marked as an occlusion object, that voxel and subsequent voxels along the ray direction are considered as areas invisible due to occlusion. Voxels before the occlusion object and within the effective detection range of the sensors are marked as the visible volume under the current frustum. This operation is performed in parallel on the view frustum models of each UAV, enabling the system to obtain the visible volume of each UAV in three-dimensional space and the invisible volume caused by occlusion behind it at each discrete time step.

[0039] Subsequently, the frustum modeling and visible field of view (VLA) subunit performs set operations on the visible volumes of all UAVs in a unified coordinate system. It calculates the union of the visible volumes corresponding to the frustums of each UAV to obtain the joint VLA volume of the entire UAV swarm within the mission area at the current moment. The physical meaning of the joint VLA volume is the spatial volume that can be directly observed by at least one UAV, considering the current attitude of all UAVs, sensor field of view, and the distribution of 3D occlusions. This subunit also defines the complement of the joint VLA volume in the mission area as the joint blind zone volume, representing the spatial region that cannot be directly observed by any UAV under the current multi-UAV layout and occlusion structure—that is, the volume range where the dynamic blind zone is located. The 3D situational awareness module outputs the aforementioned 3D occupancy map, voxel distribution of occlusion objects, view frustum models of each UAV, joint visible field volume, and joint blind zone volume in the form of a structured environment model to the target risk quantification module and the collaborative decision-making and trajectory optimization module. This enables subsequent modules to calculate the 3D blind zone risk comprehensive index and the blind zone suppression collaborative effectiveness comprehensive index based on a precise understanding of the 3D occlusion field and joint visible field distribution, thereby achieving quantitative identification and targeted suppression of dynamic blind zone resonance phenomena.

[0040] The target risk quantification module first fuses target detection and tracking results from different UAVs and perspectives through the target state estimation and probability representation subunit. Each UAV's onboard tracking algorithm outputs the target's position, velocity, and confidence level in its local coordinate system. This module, based on the UAV pose estimation results under a unified time base, transforms these state variables from their local coordinate system to the unified coordinate system of the aforementioned 3D occupancy map. It then uses filtering algorithms or equivalent state estimation methods to obtain the target's 3D position, velocity, and covariance estimates at the current moment. To accommodate subsequent spatial relationship calculations with the joint visible area volume and joint blind zone volume, this subunit further discretizes the target position distribution using particle sets or Monte Carlo samples. It generates a set of weighted hypothetical position samples within the possible spatial range of the target's occurrence, with each sample corresponding to a candidate 3D position and its occurrence probability weight, to approximate the uncertainty distribution of the target position.

[0041] After obtaining the particle-based representation of the potential target location, the target risk quantification module maps each particle sample to the joint visible field volume and joint blind zone volume output by the 3D situational awareness module through the visibility mapping and blind zone determination subunit. This subunit determines, in a unified coordinate system, whether the voxel unit containing the particle belongs to the joint visible field volume at the current moment. If it does, it is considered that under the current multi-UAV layout and 3D occlusion conditions, at least one UAV can directly observe the location. If the particle location falls into the joint blind zone volume, it is considered that under the current layout, the location cannot be directly observed by any UAV. By statistically analyzing the ratio of the sum of the weights of all particles falling into the joint blind zone volume to the sum of all sample weights, the probability measure of the target falling into the joint blind zone volume at the current moment is obtained, denoted as in this embodiment. The value of this probability quantity is preferably normalized to the range of zero to one; the larger the value, the higher the probability that the target is in an unobserved space.

[0042] To reflect the uncertainty of the system's target visibility assessment, this module further constructs a blind zone uncertainty measure based on visibility mapping and blind zone determination. In one specific implementation, the target being within the joint visible volume and the target being within the joint blind zone volume can be considered as two mutually exclusive events. and Let the probabilities of the two types of events be represented, and an entropy-based uncertainty index be constructed based on this probability distribution. Then, normalization is applied to make... The value of falls between zero and one. The Shannon entropy of this binary distribution is directly used for calculation, and its mathematical expression is: Since the maximum value of binary Shannon entropy is 1, the above calculation results are normalized to the range of 0 to 1. Value. Among them The higher the value, the less certain the system is about whether the target is in the blind zone, such as when the target is located at the boundary of the blind zone or when there is a large discrepancy between multi-source observations.

[0043] Beyond considering current geometric visibility and uncertainty, the target risk quantification module also introduces a statistical description of the target's historical evasion behavior through a concealed space prior modeling subunit. This subunit, based on multiple batches of task data accumulated during long-term system operation, performs offline mining of disappearing and reappearing segments of the target trajectory. It correlates these trajectory segments with the 3D occlusion structure and the evolution of the joint blind zone volume at the time of the task, identifying spatial areas repeatedly used by the target as evasion paths or temporary stops, such as narrow passages between tall buildings, shadowed areas under overpasses, and occlusion strips under forest canopies. For these spatial voxels exhibiting high concealment tendency in historical tasks, this module can add concealment weights to the 3D occupancy map, forming a concealed space prior field. During the online evaluation phase, the concealed space prior modeling subunit performs a weighted average of all particles based on the concealment weight of the voxel where the current particle sample is located, obtaining a comprehensive concealment score of the target's current potential position relative to the historical concealed space distribution, denoted in this embodiment as... The overall concealment score can also be normalized to between zero and one. (Concealment score) The calculation formula is: ;in, is the total number of particles; i is the particle index; The voxel associated with the i-th particle; voxels The hidden weights; The weight of the i-th particle; the hidden weight The calculation formula is: ,in For the historical mission objectives in voxels The number of times it disappears within. This represents the total number of times the objective disappeared during the historical mission.

[0044] Based on the aforementioned probability, uncertainty, and concealment components, this module introduces a three-dimensional blind spot risk comprehensive index. ,Will , and Three complementary indicators are summarized to quantitatively characterize the blind spots and risk avoidance of a single target at the current moment. In specific implementation, It can be represented as: ;in, , , These are non-negative weighting coefficients; the preferred settings satisfy... + + =1, for example =0.5, =0.3, =0.2.

[0045] The collaborative decision-making and trajectory optimization module interacts bidirectionally with the 3D situational awareness module and the target risk quantification module. It receives 3D occupancy maps, occlusion voxel distributions, view frustum models of each UAV, joint view area and blind zone volumes, as well as target state estimations and particle-based position distributions. This module incorporates prior field information on concealed spaces. Under the premise of meeting flight safety, communication, and energy constraints, it optimizes the collaborative flight path of multiple UAVs, ensuring tracking accuracy while actively suppressing the volume of potential concealed spaces and joint blind zones, and inhibiting the formation and accumulation of dynamic blind zone resonance phenomena.

[0046] In its implementation, the collaborative decision-making and trajectory optimization module includes a candidate trajectory generation and view-of-view prediction subunit. This subunit generates several feasible candidate trajectories for each UAV within a given planning time domain, based on the UAV's real-time pose, speed, remaining battery power, and the maneuverability constraints and safety interval rules provided by the flight control system, while also considering dynamic and obstacle avoidance constraints. The candidate trajectory generation and view-of-view prediction subunit employs a piecewise Bézier curve-based method to randomly sample and generate a set of smooth candidate trajectories under the UAV's dynamic constraints.

[0047] For each collaborative scheme consisting of multiple UAV candidate trajectories, the candidate trajectory generation and vision prediction subunit utilizes the 3D occupancy map and occlusion voxel distribution constructed in the 3D situational awareness module, combined with the temporal evolution of each UAV's frustum model, to predict the visible volume, joint visible volume, and joint blind zone volume at each discrete time step within the planning time domain. This yields the evolution trajectory of the 3D visible and blind zone fields over a future period under the candidate collaborative scheme. Thus, each candidate collaborative trajectory not only corresponds geometrically to a set of trajectory curves showing the UAV pose changing over time, but also, at the visibility level, corresponds to a set of temporal evolution of the joint visible volume and joint blind zone volume distribution.

[0048] To evaluate the overall performance of different candidate cooperative trajectory schemes in suppressing blind zone risk, the cooperative decision-making and trajectory optimization module uses blind zone entropy assessment and cost construction sub-units to convert the three-dimensional blind zone risk comprehensive index output by the target risk quantification module into a single unit. Joint analysis is performed with the predicted joint visible volume and occultation space prior field under the candidate schemes. For the k-th group of candidate cooperative trajectory schemes, at each discrete time step within the planning time domain, this sub-unit calculates the target under the candidate scheme based on the predicted joint visible volume and target particle position distribution. The evolutionary situation, and for all key objectives Within the planning time domain, a time-averaged and target-weighted average is performed to obtain the time-averaged average reflecting the overall blind spot risk level, denoted as . .

[0049] Meanwhile, the blind zone entropy assessment and cost construction sub-unit, based on the prior field of hidden space, statistically analyzes the spatial intersection between voxel regions with high hidden weights and the predicted joint blind zone volume at each time step within the planning time domain. This intersection is then normalized using volume proportion or prior weight integration to obtain an index characterizing the degree of insufficient coverage of key hidden spaces under this candidate scheme, denoted as... . ;in For the total volume of highly concealed voxels, The volume of highly concealed voxels in the combined blind zone; t represents the total number of time steps in the planning time domain; t is the time step index.

[0050] In addition, this subunit also constructs a comprehensive quantity of collaborative execution cost to reflect the cost of collaborative execution based on the trajectory deviation, attitude adjustment range, and formation reconstruction range of each UAV under the candidate scheme. , ;in The trajectory deviation is used to measure the degree of deviation between the candidate trajectory and the current flight path or the planned cruise path of the UAV; it is quantified by calculating the Euclidean distance difference between trajectory points, the change in path length, or the change in control commands. Attitude adjustment range measures the range of changes in the drone's body attitude required to achieve a new observation perspective, such as the required changes in pitch and roll angles. The formation reconfiguration magnitude measures the degree of change in the entire drone swarm from its current formation to the formation required by the new scheme. It is quantified by calculating the mean or variance of the position changes of all drones in the swarm. , , These are the normalized weighting coefficients. + + =1, for example =0.3, =0.4, =0.3.

[0051] After completing the construction of the above three components, this module introduces a comprehensive index of blind zone suppression synergistic effectiveness. The performance of the k-th candidate scheme in mitigating blind zone risks, suppressing key hidden spaces, and controlling execution costs is evaluated using a single scale. In specific implementation, It can be represented as: ;in, For based on The average target-level blind zone risk is summarized, reflecting the effectiveness of the scheme in reducing overall blind zone risk. It serves as a measure of insufficient coverage in key concealed spaces, reflecting the extent to which highly concealed areas remain in blind spots for extended periods. The overall cost of coordinated execution reflects the resource expenditure of the scheme in terms of energy consumption, maneuver disturbance, and formation stability. , , These are non-negative weighting coefficients used to strike a balance between reducing blind spot risk, enhancing coverage of key hidden spaces, and the cost of constraint enforcement. The preferred settings satisfy... =1, for example , , .

[0052] This module calculates all candidate solutions in each planning cycle. Values, and minimize them. To optimize the objective, the collaborative trajectory scheme with the best overall performance is selected as the execution plan. In a centralized implementation, this module can be deployed at the ground control terminal or the master UAV to uniformly evaluate and select the cluster trajectory, and distribute the desired pose sequence or control commands to each UAV's flight control system. In a distributed implementation, each UAV can independently generate its own candidate trajectory based on the globally broadcast blind zone risk field and local perception information, and converge to the overall trajectory through negotiation or auction processes. The solution that is approximately the minimum.

[0053] To adapt to rapid changes in target behavior and environmental conditions, the online reconfiguration and trigger management subunit continuously monitors the behavior of each target. Changes and current implementation plan Value. When a target is discovered. A significant increase in a short period of time, or the current plan When a value exceeds a preset threshold, an emergency or incremental planning process is triggered, rapidly reconfiguring the mission roles and flight paths of some drones without waiting for the regular planning cycle to end. For example, drones in low-risk areas may be instructed to temporarily adjust their cruise modes and head towards high-risk areas. or high New observation perspectives are established in the region. By adjusting altitude, heading, and formation, the joint visibility volume can be quickly reconstructed, compressing structural blind spots that targets may exploit.

[0054] Through the above structural setup and workflow, the collaborative decision-making and trajectory optimization module introduces a comprehensive three-dimensional blind zone risk index on top of the traditional multi-UAV collaborative planning framework. Synergistic efficacy index of blind zone suppression The core comprehensive analysis mechanism enables UAV swarms to no longer solely optimize path length or local field of view quality when performing target tracking and trajectory planning under the constraints of a three-dimensional occlusion field. Instead, it enables unified quantification and balancing of blind zone risks, concealed space, and execution costs at the planning and reconfiguration levels. This effectively suppresses the long-term existence of dynamic blind zone resonance in complex three-dimensional scenarios such as urban canyons, overpass clusters, and forest canopies, thereby improving the stability and robustness of the UAV cooperative tracking system based on multimodal optoelectronic fusion described in this invention in continuous tracking tasks.

[0055] The system strategy self-evolution module, deployed in the system backend or ground control terminal, connects to the 3D situational awareness module, target risk quantification module, and collaborative decision-making and trajectory optimization module via a standard data interface. It is used to uniformly collect, structure, archive, and correlate the multi-source operational data generated throughout the entire mission after the collaborative tracking task is completed. and Two comprehensive analytical parameters are used to adaptively correct and update the strategy parameters related to the blind zone.

[0056] During mission execution, through interfaces with various modules, the system continuously acquires the following data: 3D occupancy map version sequence in a unified coordinate system, temporal evolution of occlusion voxel markers, joint visible field and blind zone volume information at each time step; state estimation time series of each target throughout the mission, statistical characteristics of particle position distribution, and update records of the cloaked space prior field. Time curves; candidate collaborative trajectory schemes and their corresponding timelines within each planning period. The actual performance indicators of the implementation plan, the online reconfiguration trigger time, the list of drones participating in the reconfiguration, and the joint blind zone volume before and after the reconfiguration. Distribution changes.

[0057] Based on sufficient data accumulation, the system strategy self-evolution module explicitly models the relationships between environmental geometry, view frustum layout, target behavior, and comprehensive analysis quantities through a relationship modeling and pattern mining sub-unit. This sub-unit, under a unified coordinate system, overlays the 3D occupancy map of the entire task process with the joint blind zone volume evolution to construct a spatiotemporal trajectory reflecting changes in the 3D occlusion field, and displays the actual motion trajectories of each target, the particle distribution center trajectory, and the corresponding... The time series is mapped onto the aforementioned occlusion field evolution trajectory, and the frustum distribution of multiple UAVs, joint visible field volume, and the currently selected cooperative trajectory scheme are superimposed on the same time axis. Value. In this way, the system forms a multi-dimensional relationship graph at the data level, covering the entire chain of UAVs, view frustums, occlusion objects, targets, comprehensive analysis quantities, and planning decisions. This allows for quantitative analysis of a specific type of 3D occlusion pattern, a particular altitude layer configuration, or a set of planning weight settings, and... Long-term high The degree of correlation between persistently unsatisfactory results and the frequent occurrence of structurally lost events.

[0058] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0059] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0060] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0061] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0062] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-modal opto-electronic fusion based UAV cooperative tracking system, characterized in that, The method comprises the following modules: A three-dimensional situation intelligent perception module is configured to construct and dynamically update a three-dimensional occupancy map and an occlusion voxel distribution, and to establish a three-dimensional view cone model based on real-time poses and sensor parameters of each unmanned aerial vehicle, to calculate a visible volume of each unmanned aerial vehicle, and to further obtain a joint visible volume and a joint blind volume of the unmanned aerial vehicle cluster through set operation; A target risk quantification module is configured to calculate a probability of a target falling into the joint blind volume based on the joint visible volume and the joint blind volume, to construct a blind area uncertainty metric, to calculate a comprehensive score of target concealment in combination with a prior field of a hidden space, and to further generate a three-dimensional blind area risk comprehensive index by weighted fusion of the probability of the target falling into the joint blind volume, the blind area uncertainty metric, and the comprehensive score of target concealment. A cooperative decision and trajectory optimization module is configured to generate candidate cooperative trajectory schemes for the multiple unmanned aerial vehicles, to predict evolution of the joint visible volume and the joint blind volume under each scheme, to construct a blind area suppression cooperative efficiency comprehensive index for each candidate cooperative trajectory scheme, to generate the index by weighted fusion of a target-level blind area risk average, a key hidden space coverage deficiency metric, and a cooperative execution cost comprehensive quantity, and to select an optimal cooperative trajectory scheme for execution by taking minimization of the index as a target while monitoring risks and triggering online reconfiguration.

2. The multi-modal opto-electronic fusion based UAV cooperative tracking system according to claim 1, wherein: The three-dimensional situation intelligent perception module comprises a pose estimation subunit, a three-dimensional occupancy modeling subunit, and a view cone modeling and visible domain operation subunit. The pose estimation subunit is configured to continuously estimate poses of each unmanned aerial vehicle in a task coordinate system by using vision and inertial odometry, laser radar odometry, or a fusion algorithm, and to project multi-modal observation data to a world coordinate system or a local map coordinate system in combination with a pre-calibrated sensor extrinsic parameter matrix. The three-dimensional occupancy modeling subunit is configured to discretely model a task area in space by using a three-dimensional grid or a voxel structure with a preset resolution, to divide a plurality of voxel units, and to maintain occupancy probability, reflection intensity, and material category attributes of each voxel unit based on observation evidence of multi-modal sensors, and to mark occlusion voxels. The view cone modeling and visible domain operation subunit is configured to establish a three-dimensional view cone model based on real-time poses and sensor parameters of each unmanned aerial vehicle, to calculate visible volumes and invisible volumes of each unmanned aerial vehicle, and to obtain a joint visible volume and a joint blind volume of the unmanned aerial vehicle cluster through set operation. 3.The multi-modal opto-electronic fusion based UAV cooperative tracking system of claim 2, wherein: The view cone modeling and visible domain operation subunit calculates the joint visible volume and the joint blind volume by the following method: based on the three-dimensional view cone model of each unmanned aerial vehicle, traversing voxels along a virtual ray direction in the three-dimensional occupancy map, marking voxels and subsequent voxels as invisible regions when the ray encounters occlusion voxels, and marking voxels before the occlusion voxels and within an effective detection range of the sensor as visible volumes; performing set operation on the visible volumes of all unmanned aerial vehicles to obtain the joint visible volume; and defining a complement of the joint visible volume in the task area as the joint blind volume. 4.The multi-modal opto-electronic fusion based UAV cooperative tracking system of claim 1, wherein: The target risk quantification module comprises a target state estimation and probability representation subunit, a visibility mapping and blind area determination subunit, and a hidden space prior modeling subunit. The target state estimation and probability representation subunit obtains the state estimation of the target in a unified coordinate system, and discretely represents the uncertainty distribution of the target position by using a particle set. The visibility mapping and blind area determination subunit is used for mapping each particle sample into a joint visible volume and a joint blind volume, and calculating a probability metric of the target falling into the joint blind volume by statistically calculating the ratio of the sum of the particle weights falling into the joint blind volume to the sum of the total weights. The hidden space prior modeling subunit is used for constructing a hidden space prior field based on historical task data, and calculating a hidden property comprehensive score of the target by weightedly averaging the weights of all particles according to the hidden property weight of the voxel where the current particle sample is located.

5. The multi-modal opto-electronic fusion based UAV cooperative tracking system of claim 1, wherein: The blind area uncertainty metric is obtained by calculating the Shannon entropy of the binary probability distribution of the target being in the joint visible volume and in the joint blind volume. 6.The multi-modal opto-electronic fusion based UAV cooperative tracking system of claim 1, wherein: The cooperative decision and trajectory optimization module comprises a candidate track generation and visibility prediction subunit and a blind area entropy evaluation and cost construction subunit. The candidate track generation and visibility prediction subunit is used for generating a group of smooth candidate trajectories that are feasible under the constraints of dynamics and obstacle avoidance for each unmanned aerial vehicle, and predicting the evolution trajectories of the joint visible volume and the joint blind volume within the future planning time domain for each cooperative scheme composed of the candidate trajectory groups. The blind area entropy evaluation and cost construction subunit is used for calculating the target-level blind area risk average, the key hidden space coverage deficiency metric, and the cooperative execution cost comprehensive metric corresponding to each candidate cooperative trajectory scheme.

7. The multi-modal opto-electronic fusion based UAV cooperative tracking system according to claim 6, wherein: The target-level blind area risk average is obtained by time-averaging the three-dimensional blind area risk comprehensive index of each target within the planning time domain and weighting according to the importance of the target. 8.The multi-modal opto-electronic fusion based UAV cooperative tracking system of claim 6, wherein: The key hidden space coverage deficiency metric is obtained by statistically calculating the spatial intersection between the voxel region with a high hidden property weight and the predicted joint blind volume within the planning time domain, and performing normalization processing.

9. The multi-modal opto-electronic fusion based UAV cooperative tracking system of claim 6, wherein: The cooperative execution cost comprehensive metric is obtained by weightedly fusing the trajectory deviation, the attitude adjustment amplitude, and the formation reconstruction amplitude.

10. The multi-modal opto-electronic fusion based drone cooperative tracking system of claim 1, wherein: The system strategy self-evolution module is used for performing correlation analysis on the three-dimensional blind area risk comprehensive index and the blind area suppression cooperative efficiency comprehensive index.

Images