A UAV multi-target tracking method and system based on vision and millimeter-wave radar information fusion

Abstract

The present invention provides a method and system for tracking multiple targets of unmanned aerial vehicles based on the fusion of visual and millimeter-wave radar information, wherein the present invention obtains association results by utilizing a spatiotemporal consistency model, constructs an adaptive joint probability data association framework according to the association results, so as to obtain an initial target state estimation value, adopts a forward-looking path planning algorithm, obtains an optimal observation angle flight path, utilizes a Bayesian network, obtains initial observation data, generates an optimal target state estimation value based on the initial observation data, uses a distributed collaborative perception mechanism, generates an initial tracking task allocation plan, implements an intelligent resource allocation strategy, generates an initial resource configuration plan, starts a rapid response protocol, obtains an optimal tracking task allocation plan, and generates an optimal resource configuration plan based on the optimal tracking task allocation plan; the technical solution provided by the present invention improves the accuracy of target recognition, enhances environmental adaptability, improves tracking efficiency, and prolongs working time.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of unmanned aerial vehicles, and in particular to a method for tracking multiple targets of unmanned aerial vehicles based on the fusion of visual and millimeter-wave radar information. Background Art

[0002] With the development of UAV technology, its application in many fields is becoming more and more extensive, especially in multi-target tracking. The UAV multi-target tracking method needs to be able to track multiple dynamic targets in real time and accurately in complex environments, such as pedestrians, vehicles and other aircraft, including monitoring urban traffic conditions, ensuring public safety, and performing battlefield surveillance tasks to ensure the timely acquisition of tactical intelligence, and long-term monitoring of wild animals in nature reserves to protect the ecological environment. These application scenarios require UAVs to have high-precision target recognition capabilities, rapid response capabilities and efficient task allocation mechanisms to adapt to changing environmental conditions and task requirements;

[0003] At present, multi-target tracking of UAVs mainly relies on a single sensor (such as visual sensor or radar) or a simple information fusion method. Existing solutions usually use cameras to capture images and use image processing algorithms to identify and track targets. This method is suitable for environments with sufficient light and relatively simple backgrounds. Then, millimeter-wave radar is used to detect parameters such as the distance and speed of the target. It is suitable for all-weather conditions, especially in bad weather or low-light environments. Finally, the data of the visual sensor and millimeter-wave radar are preliminarily integrated, but lacks in-depth information association and intelligent processing;

[0004] Although the existing solutions have met some of the needs to a certain extent, there are still some significant defects. For example, first of all, there are limitations of a single sensor. Relying solely on visual sensors is easily affected by lighting conditions, and using millimeter-wave radar alone is difficult to provide sufficient target feature information, resulting in insufficient tracking accuracy; then there is the shallow level of information fusion. Existing information fusion methods mostly stay at the data superposition level, failing to fully utilize the advantages of the two sensors, unable to achieve deep information association and complementarity, and slow response speed. When facing new targets entering the observation range or existing targets leaving, existing technologies often cannot quickly adjust tracking strategies, resulting in low tracking efficiency and unable to respond to complex dynamic environmental changes in real time. Finally, resource allocation is unreasonable. The resource allocation strategy of existing technologies is not flexible enough, and sensor parameters cannot be dynamically adjusted according to actual task requirements, resulting in energy waste and reduced task completion quality. Summary of the invention

[0005] The embodiment of the present invention provides a method and system for tracking multiple targets of unmanned aerial vehicles based on the fusion of visual and millimeter-wave radar information, so as to solve the problems in the prior art of insufficient tracking accuracy, inability to respond to complex dynamic environmental changes in real time and to achieve deep information association and complementarity, and inability to respond to complex dynamic environmental changes in real time, resulting in energy waste and reduced task completion quality.

[0006] In a first aspect, an embodiment of the present invention provides a method for tracking multiple targets of a drone based on the fusion of visual and millimeter-wave radar information, comprising:

[0007] Using the spatiotemporal consistency model, the synchronous observation data streams from the visual sensor and millimeter-wave radar equipped by the UAV are matched and processed to obtain the correlation results between the contour changes of dynamic targets and the time series characteristics of the millimeter-wave radar reflection signal, wherein the dynamic targets include pedestrians, vehicles and other aircraft;

[0008] According to the association results of the dynamic target contour changes and the time series characteristics of the millimeter-wave radar reflection signal, an adaptive joint probability data association framework is constructed, and the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in three-dimensional space are estimated by using the adaptive joint probability data association framework to generate an initial target state estimation value. Based on the initial target state estimation value, a forward-looking path planning algorithm is used to dynamically generate and optimize the flight trajectory of the UAV to obtain a flight path with an optimal observation angle;

[0009] Using a Bayesian network, uncertainty modeling is performed on the observation error caused by environmental interference to obtain initial observation data, and the initial observation data is integrated into the initial target state estimate to generate an optimal target state estimate. Based on the optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to collaboratively process the dynamic target tracking information shared by multiple UAVs to generate an initial tracking task allocation plan.

[0010] According to the initial tracking task allocation plan and the optimal observation angle flight path, an intelligent resource allocation strategy is implemented to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter wave radar to generate an initial resource allocation plan;

[0011] Based on the initial resource allocation plan, when it is detected that a dynamic target other than the existing dynamic target enters the observation range or the current dynamic target leaves the observation range, the rapid response protocol is started, the priority under the current monitoring scenario is evaluated, and a priority list is obtained. Based on the priority list and the optimal observation angle flight path, the initial tracking task allocation plan is adjusted to obtain the optimal tracking task allocation plan, and the optimal tracking task allocation plan is fed back to the intelligent resource allocation strategy to generate the optimal resource allocation plan.

[0012] Optionally, according to the association result between the dynamic target contour change and the millimeter-wave radar reflection signal time series characteristics, an adaptive joint probability data association framework is constructed, and the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in three-dimensional space are estimated by using the adaptive joint probability data association framework to generate an initial target state estimation value, and based on the initial target state estimation value, a forward-looking path planning algorithm is used to dynamically generate and optimize the flight trajectory of the UAV to obtain the optimal observation angle flight path, including:

[0013] By using the correlation results of the dynamic target contour changes and the time series characteristics of the millimeter-wave radar reflection signal, an adaptive joint probability data association framework is constructed, and the behavior pattern of the dynamic target is predicted in combination with the deep learning algorithm. The position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in the three-dimensional space are estimated to obtain the initial target state estimation value and the initial target behavior prediction;

[0014] Based on the initial target state estimation value, a forward-looking path planning algorithm is used, combined with the terrain and obstacle information obtained in real time by the environmental perception module, to dynamically generate and optimize the flight trajectory of the UAV to obtain the initial observation angle flight path;

[0015] Feeding back the initial observation angle flight path into an adaptive joint probability data association framework, combining the initial target behavior prediction, verifying and adjusting the accuracy of the initial target state estimate, and generating an optimal target state estimate;

[0016] Based on the optimal target state estimation value, a multi-objective optimization algorithm is used to comprehensively calculate the energy consumption, flight efficiency and tracking accuracy of the UAV, and the flight trajectory of the UAV is evaluated and optimized again to obtain the flight path with the best observation angle.

[0017] Optionally, based on the initial target state estimate, a forward-looking path planning algorithm is used, combined with terrain and obstacle information obtained in real time by the environment perception module, to dynamically generate and optimize the flight trajectory of the UAV to obtain an initial observation angle flight path, including:

[0018] Using the initial target state estimation value and the terrain and obstacle information obtained in real time by the environment perception module, combined with the three-dimensional map data, a multi-level modeling process is performed on the environment around the UAV to obtain an environment model;

[0019] Based on the environmental model, a forward-looking path planning algorithm and a reinforcement learning optimizer are used to dynamically generate multiple candidate flight paths, and potential risks and observation effects on each flight path are predicted to obtain a candidate flight path set and a candidate flight path risk assessment report;

[0020] Based on the candidate flight path set and the candidate flight path risk assessment report, combined with the maneuverability limitations of the UAV, weather condition predictions, and dynamic obstacle avoidance algorithms, a safety assessment is performed on each flight path, flight paths that meet preset conditions are screened out, and an optimal flight path set is generated;

[0021] From the optimal flight path set, a multi-objective genetic algorithm is used to calculate the energy consumption, flight efficiency, tracking accuracy and flight path complexity of the UAV, and an optimal flight path is selected to obtain an initial observation angle flight path.

[0022] Optionally, based on the environmental model, a forward-looking path planning algorithm and a reinforcement learning optimizer are used to dynamically generate multiple candidate flight paths, and potential risks and observation effects on each flight path are predicted to obtain a candidate flight path set and a candidate flight path risk assessment report, including:

[0023] Using the environmental model and the real-time dynamic environment prediction module, combined with the deep reinforcement learning optimizer, the UAV flight trajectory is intelligently planned, the environmental change trend is calculated, and a preliminary flight path set is obtained;

[0024] Based on the preliminary flight path set, an adaptive multi-agent system and a swarm intelligence algorithm are used to simulate the collaborative behavior mode among multiple drones, dynamically adjust and optimize each flight path, and generate multiple candidate flight paths;

[0025] Analyze the potential risks and observation effects of the multiple candidate flight paths using three-dimensional map data, real-time perception information, and a risk assessment model based on a Bayesian network to generate a risk assessment report;

[0026] Associating the risk assessment report with multiple candidate flight paths to generate an initial candidate flight path set, and performing a virtual simulation test on the initial candidate flight path set through a digital twin platform to verify the performance of each candidate flight path and the impact of each candidate flight path on the UAV fleet, so as to obtain an optimal candidate flight path set;

[0027] The initial candidate flight path set is fed back into the safety assessment, and combined with the behavior pattern prediction of the drone cluster to obtain the initial observation angle flight path.

[0028] Optionally, a Bayesian network is used to perform uncertainty modeling on observation errors caused by environmental interference to obtain initial observation data, and the initial observation data is integrated into the initial target state estimate to generate an optimal target state estimate. Based on the optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to collaboratively process dynamic target tracking information shared by multiple UAVs to generate an initial tracking task allocation plan, including:

[0029] Using Bayesian network, uncertainty modeling is performed on observation errors caused by environmental interference to obtain initial observation data.

[0030] Integrating the initial observation data into the initial target state estimate to generate an optimal target state estimate;

[0031] Based on the optimal target state estimation value and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to combine the dynamic target tracking information shared by multiple UAVs to perform collaborative processing and generate a preliminary tracking task allocation plan;

[0032] The preliminary tracking task allocation plan is optimized and adjusted in combination with the current task load and remaining resource conditions of each UAV to generate an initial tracking task allocation plan.

[0033] Optionally, according to the initial tracking task allocation plan and the optimal observation angle flight path, an intelligent resource allocation strategy is implemented to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter wave radar to generate an initial resource allocation plan, including:

[0034] According to the initial tracking task allocation plan and the optimal observation angle flight path, the task requirements and current status of each UAV are evaluated to obtain a task requirement evaluation result;

[0035] Based on the task requirement assessment results, an intelligent resource allocation strategy is implemented to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter-wave radar to obtain the optimal sensor configuration parameters;

[0036] According to the task requirement assessment result, the sampling frequency and resolution of the visual sensor are adjusted to obtain the optimal parameter setting of the visual sensor;

[0037] According to the task requirement assessment result, the sampling frequency and resolution of the millimeter wave radar are adjusted to obtain the optimal parameter setting of the millimeter wave radar;

[0038] The optimal parameter settings of the visual sensor and the optimal parameter settings of the millimeter-wave radar are integrated to generate an initial resource configuration plan.

[0039] Optionally, when it is detected that a dynamic target other than an existing dynamic target enters the observation range or when the current dynamic target leaves the observation range, a rapid response protocol is started, the priority in the current monitoring scenario is evaluated to obtain a priority list, and based on the priority list and the optimal observation angle flight path, the initial tracking task allocation plan is adjusted to obtain an optimal tracking task allocation plan, and the optimal tracking task allocation plan is fed back to the intelligent resource allocation strategy to generate an optimal resource allocation plan, including:

[0040] Using the visual sensor and millimeter wave radar equipped by the UAV, the synchronous observation data stream from the UAV is monitored in real time to detect the dynamic targets other than the existing dynamic targets entering the observation range or the existing dynamic targets leaving the observation range, and obtain the trigger condition confirmation result;

[0041] According to the trigger condition confirmation result, a rapid response protocol is initiated to re-evaluate the current monitoring scene;

[0042] Based on the trigger condition confirmation result, combined with the real-time acquired observation data and the initial resource allocation plan, priority evaluation processing is performed on the dynamic targets in the current monitoring scenario to generate a priority list;

[0043] Using the priority list and the optimal observation angle flight path, the initial tracking task allocation scheme is adjusted to obtain the optimal tracking task allocation scheme;

[0044] The optimal tracking task allocation plan is fed back into the intelligent resource allocation strategy, and the sampling frequency and resolution of the visual sensor and the millimeter-wave radar are adjusted according to the optimal tracking task allocation plan to generate the optimal resource configuration plan.

[0045] In a second aspect, an embodiment of the present invention provides a multi-target tracking system for unmanned aerial vehicles based on vision and millimeter-wave radar information fusion, including:

[0046] A matching module uses a spatiotemporal consistency model to match the synchronous observation data streams from the visual sensor and the millimeter-wave radar equipped by the UAV to obtain the correlation results between the contour changes of dynamic targets and the time series characteristics of the millimeter-wave radar reflection signal, wherein the dynamic targets include pedestrians, vehicles and other aircraft;

[0047] A construction module is used to construct an adaptive joint probability data association framework according to the association result of the dynamic target contour change and the millimeter wave radar reflection signal time series characteristics, and use the adaptive joint probability data association framework to estimate the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter wave radar equipped by the UAV in three-dimensional space, generate an initial target state estimation value, and based on the initial target state estimation value, use a forward-looking path planning algorithm to dynamically generate and optimize the flight trajectory of the UAV to obtain the flight path with the best observation angle;

[0048] A processing module is used to use a Bayesian network to perform uncertainty modeling on observation errors caused by environmental interference to obtain initial observation data, and integrate the initial observation data into the initial target state estimate to generate an optimal target state estimate. Based on the optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to collaboratively process dynamic target tracking information shared by multiple UAVs to generate an initial tracking task allocation plan;

[0049] An adjustment module is used to implement an intelligent resource allocation strategy according to the initial tracking task allocation plan and the optimal observation angle flight path, dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter wave radar, and generate an initial resource allocation plan;

[0050] A generation module is used to start a rapid response protocol based on the initial resource allocation plan, evaluate the priority in the current monitoring scenario, obtain a priority list, adjust the initial tracking task allocation plan based on the priority list and the optimal observation angle flight path, obtain the optimal tracking task allocation plan, and feed back the optimal tracking task allocation plan to the intelligent resource allocation strategy to generate the optimal resource allocation plan. In a third aspect, an embodiment of the present invention provides a computing device, including a processor and a memory, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute any of the methods described in the first aspect.

[0051] In a fourth aspect, an embodiment of the present invention provides a computer storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement any one of the methods described in the first aspect.

[0052] In the embodiment of the present invention, the spatiotemporal consistency model is used to match the synchronous observation data streams of the visual sensor and the millimeter-wave radar equipped by the UAV, and the contour changes of dynamic targets (such as pedestrians, vehicles and other aircraft) and the time series characteristics of the millimeter-wave radar reflection signal are extracted. Based on this association result, an adaptive joint probability data association framework is constructed to estimate the position, posture and motion parameters of each dynamic target in three-dimensional space, generate an initial target state estimate, and use a forward-looking path planning algorithm to optimize the flight trajectory of the UAV to obtain the flight path with the best observation angle.

[0053] Furthermore, the Bayesian network is used to model the uncertainty of the observation error caused by environmental interference, and the corrected initial observation data is obtained, which is integrated into the initial target state estimate to generate the optimal target state estimate. Based on this estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to process the dynamic target tracking information between multiple drones to generate an initial tracking task allocation plan. Subsequently, according to the task allocation plan and the flight path, an intelligent resource allocation strategy is implemented to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter wave radar to generate an initial resource allocation plan. When a new dynamic target is detected to enter or an existing target leaves the observation range, a rapid response protocol is initiated to evaluate the priority in the monitoring scenario, adjust the tracking task allocation plan, and finally generate the optimal tracking task allocation plan and feed it back to the resource allocation strategy to generate the optimal resource allocation plan; the technical solution provided by the present invention improves the accuracy of target recognition, enhances environmental adaptability, improves tracking efficiency and quality, reduces energy consumption, and extends the working time of drones;

[0054] Furthermore, the introduction of deep learning algorithms to predict the behavior patterns of dynamic targets enhances the system's ability to predict future target movements, allowing the drone to adjust its flight path in advance to avoid potential conflicts; the use of environmental perception modules to obtain terrain and obstacle information in real time, combined with high-resolution three-dimensional map data, to build a refined environmental model, improving the safety and reliability of path planning; through forward-looking path planning algorithms and reinforcement learning optimizers, multiple candidate flight paths are dynamically generated, and the risks and observation effects of each path are evaluated to ensure the selection of the optimal path, thereby improving the flight efficiency and mission completion rate of the drone; the preliminary optimal observation angle flight path is fed back to the adaptive joint probability data association framework to verify and adjust the target state estimate, forming a closed-loop control and enhancing the stability and accuracy of the system;

[0055] Furthermore, by combining the real-time dynamic environment prediction module and high-resolution three-dimensional map data, the environment around the drone is modeled at multiple levels to generate a refined environmental model, which provides a solid foundation for subsequent path planning; the adaptive multi-agent system (MAS) and swarm intelligence algorithm (such as particle swarm optimization PSO) are used to simulate the collaborative behavior mode between multiple drones, dynamically adjust and optimize each path, and improve the efficiency of multi-drone collaboration; through the risk assessment model based on the Bayesian network, the potential risks and observation effects on each path are comprehensively analyzed, and a detailed risk assessment report is generated to ensure the safety and reliability of path selection; the digital twin platform is used to conduct virtual simulation tests to verify the performance of each path under different conditions and its impact on the drone fleet, to ensure the reliability and optimal performance of path selection, and finally, the initial candidate flight path set is generated through a distributed collaborative perception mechanism to provide a comprehensive basis for subsequent safety assessment.

[0056] These and other aspects of the present invention will become more apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A flowchart of a method for tracking multiple targets of a drone based on the fusion of visual and millimeter-wave radar information provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of the structure of a multi-target tracking system for unmanned aerial vehicles based on the fusion of visual and millimeter-wave radar information provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the structure of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0062] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0064] Figure 1 A flowchart of a method for tracking multiple targets of a UAV based on the fusion of visual and millimeter-wave radar information is provided for an embodiment of the present invention. Figure 1 As shown, the method includes:

[0065] Step 101: using a spatiotemporal consistency model, matching and processing the synchronous observation data streams from the visual sensor and the millimeter-wave radar equipped by the UAV to obtain the correlation results between the contour changes of the dynamic targets and the time series characteristics of the millimeter-wave radar reflection signals, wherein the dynamic targets include pedestrians, vehicles and other aircraft;

[0066] In this step, the spatiotemporal consistency model is a technology used to synchronize and correlate observation data from different sensors (such as visual sensors and millimeter-wave radars). The model ensures the consistency of data from different sensors in time and space, thereby improving the accuracy of multi-source data fusion. In this step, the visual sensor equipped with the drone captures image sequences, while the millimeter-wave radar provides information such as the distance, speed, and reflection intensity of the target. These data streams are matched and processed through the spatiotemporal consistency model to extract the correlation results of the dynamic target contour changes and the time series characteristics of the millimeter-wave radar reflection signal. The dynamic targets include pedestrians, vehicles, and other aircraft;

[0067] When the UAV is performing a mission, the visual sensor and the millimeter-wave radar continuously collect synchronous observation data streams in the environment. The spatiotemporal consistency model first preprocesses these data to eliminate noise and synchronize timestamps. Then, the model matches the target contour changes captured by the visual sensor with the time series characteristics of the reflected signal recorded by the millimeter-wave radar to generate correlation results.

[0068] For example, in urban security scenarios, drones fly above busy traffic intersections and use visual sensors to capture images of vehicles and pedestrians. At the same time, millimeter-wave radars record their distance and speed information. Through the spatiotemporal consistency model, they can accurately distinguish between stationary objects and moving targets and establish a relationship between the two, laying a solid foundation for subsequent target tracking.

[0069] Step 102: constructing an adaptive joint probability data association framework based on the association result between the dynamic target contour change and the millimeter-wave radar reflection signal time series characteristics, using the adaptive joint probability data association framework to estimate the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in three-dimensional space, generate an initial target state estimation value, and based on the initial target state estimation value, using a forward-looking path planning algorithm to dynamically generate and optimize the flight trajectory of the UAV to obtain a flight path with an optimal observation angle;

[0070] In this step, the adaptive joint probabilistic data association framework (JPDAF) is a method used to estimate the state of multiple dynamic targets. It combines data from multiple sensors and calculates the state parameters (such as position, attitude, and motion parameters) of each target through a probabilistic model. The forward-looking path planning algorithm dynamically generates and optimizes the flight trajectory of the drone based on these estimated values ​​to ensure that it always maintains the best observation angle;

[0071] Based on the association results obtained in step 101, the system constructs an adaptive joint probabilistic data association framework to estimate the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter wave radar equipped by the UAV in three-dimensional space, generate an initial target state estimation value, and then use a forward-looking path planning algorithm to dynamically generate and optimize the flight trajectory of the UAV according to these estimation values ​​to obtain a flight path with the best observation angle;

[0072] Continuing with the above embodiment, in the traffic intersection monitoring scenario, the system uses an adaptive joint probabilistic data association framework to estimate the position, speed and other parameters of pedestrians and vehicles in real time. Then, through a forward-looking path planning algorithm, the drone adjusts its own flight path to ensure that it is always at the best observation angle, so as to more clearly capture the target's behavior and provide support for subsequent safety analysis.

[0073] Step 103: using a Bayesian network to perform uncertainty modeling on the observation error caused by environmental interference to obtain initial observation data, and integrating the initial observation data into the initial target state estimate to generate an optimal target state estimate. Based on the optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to collaboratively process the dynamic target tracking information shared by multiple UAVs to generate an initial tracking task allocation plan.

[0074] In this step, the Bayesian network is a probabilistic graphical model used to represent the dependencies between variables and is widely used in uncertainty reasoning and decision making. In this step, the Bayesian network is used to model the observation errors caused by environmental interference, generate the corrected initial observation data, and integrate it into the initial target state estimate to generate the optimal target state estimate. The distributed collaborative perception mechanism is used to process the dynamic target tracking information shared by multiple drones and generate the initial tracking task allocation plan;

[0075] Using Bayesian networks, the system models the uncertainty of observation errors caused by environmental interference (such as weather changes, electromagnetic interference, etc.) and generates corrected initial observation data. These corrected data are integrated into the initial target state estimate to generate a more accurate optimal target state estimate. Based on this estimate and the optimal observation angle flight path, the system uses a distributed collaborative perception mechanism to process dynamic target tracking information between multiple drones and generate an initial tracking task allocation plan to ensure that the task allocation of each drone is reasonable and the collaborative work is efficient.

[0076] Continuing with the above traffic intersection monitoring scenario, when the drone encounters environmental interference such as rain or strong light, the Bayesian network helps the system correct observation errors to ensure that the target state can be accurately estimated even under harsh conditions. At the same time, multiple drones share tracking information through a distributed collaborative perception mechanism, optimize their respective flight paths and task allocation, and jointly complete effective monitoring of the entire area.

[0077] Step 104: implementing an intelligent resource allocation strategy according to the initial tracking task allocation plan and the optimal observation angle flight path, dynamically adjusting the sampling frequency and resolution of the visual sensor and the millimeter wave radar, and generating an initial resource allocation plan;

[0078] In this step, the intelligent resource allocation strategy refers to dynamically adjusting sensor parameters (such as sampling frequency and resolution) according to the current task requirements and resource conditions to achieve optimal resource utilization, ensuring that the drone can maintain high accuracy without wasting energy when performing multi-target tracking tasks;

[0079] According to the initial tracking task allocation plan generated in step 103 and the optimal observation angle flight path, the system implements an intelligent resource allocation strategy, dynamically adjusts the sampling frequency and resolution of the visual sensor and the millimeter wave radar, and generates an initial resource allocation plan. This dynamic allocation not only improves the tracking effect, but also extends the working time of the drone and improves the overall task efficiency.

[0080] In traffic intersection monitoring, the intelligent resource allocation strategy dynamically adjusts the parameter settings of the visual sensors and millimeter-wave radars of each drone according to its specific mission requirements. For example, for drones that need to track fast-moving vehicles at close range, the system will increase the sampling frequency and resolution of the sensors, while for drones responsible for large-scale monitoring, the parameters will be appropriately reduced to save energy, ensuring that all drones can efficiently complete their respective tasks.

[0081] Step 105: Based on the initial resource allocation plan, when it is detected that a dynamic target other than the existing dynamic target enters the observation range or the current dynamic target leaves the observation range, a rapid response protocol is started, the priority in the current monitoring scenario is evaluated, and a priority list is obtained. Based on the priority list and the optimal observation angle flight path, the initial tracking task allocation plan is adjusted to obtain an optimal tracking task allocation plan, and the optimal tracking task allocation plan is fed back to the intelligent resource allocation strategy to generate an optimal resource allocation plan;

[0082] In this step, the quick response protocol is a mechanism that is immediately activated when a new dynamic target is detected to enter the observation range or an existing target leaves. It is used to re-evaluate the priority in the current monitoring scenario, generate a priority list, and adjust the tracking task allocation plan accordingly. Finally, these adjustments are fed back to the intelligent resource allocation strategy to generate the best resource allocation plan to ensure the real-time and flexibility of the system.

[0083] Based on the initial resource allocation scheme generated in step 104, when the system detects that a new target other than an existing dynamic target enters the observation range or an existing target leaves, a rapid response protocol is initiated, which evaluates the priority in the current monitoring scenario and generates a priority list. Based on this list and the flight path of the best observation angle, the system adjusts the initial tracking task allocation scheme to obtain the best tracking task allocation scheme, and feeds these adjustments back to the intelligent resource allocation strategy to generate the best resource allocation scheme;

[0084] In the traffic intersection monitoring scenario, when a new vehicle or pedestrian suddenly enters the drone's observation range, or a target leaves, the rapid response protocol is immediately activated. The system quickly evaluates the priority of the current scene and adjusts the task allocation and flight path of each drone to ensure that the new target can be included in the monitoring range in a timely manner without affecting other targets being tracked. In this way, the entire drone swarm can flexibly respond to complex dynamic environments and maintain efficient monitoring and tracking capabilities.

[0085] Through the implementation of the above steps, the present invention significantly improves the intelligence level, adaptability and efficiency of the UAV multi-target tracking system. The spatiotemporal consistency model ensures the accurate matching of multi-source data. The adaptive joint probabilistic data association framework and the forward-looking path planning algorithm realize high-precision target state estimation and optimal flight path planning. The Bayesian network and distributed collaborative perception mechanism enhance the uncertainty and collaborative processing capabilities of the system. The intelligent resource allocation strategy and the rapid response protocol ensure the efficient use of resources and the real-time response of the system. It not only improves the multi-target tracking capability of the UAV in complex environments, but also greatly improves the quality and efficiency of task completion.

[0086] In order to further improve the prediction capability of the system and optimize the flight path of the UAV, as another embodiment, the step 102 constructs an adaptive joint probability data association framework based on the association result between the dynamic target contour change and the millimeter-wave radar reflection signal time series characteristics, and uses the adaptive joint probability data association framework to estimate the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in three-dimensional space, and generate an initial target state estimation value. Based on the initial target state estimation value, a forward-looking path planning algorithm is used to dynamically generate and optimize the flight trajectory of the UAV to obtain the flight path with the best observation angle,,:

[0087] An adaptive joint probability data association framework is constructed by using the association results of the dynamic target contour changes and the time series characteristics of the millimeter-wave radar reflection signal, and the behavior pattern of the dynamic target is predicted in combination with the deep learning algorithm. The position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in three-dimensional space are estimated to obtain an initial target state estimate and an initial target behavior prediction; based on the initial target state estimate, a forward-looking path planning algorithm is used, combined with the terrain and obstacle information obtained in real time by the environmental perception module, to dynamically generate and optimize the flight trajectory of the UAV to obtain an initial observation angle flight path; the initial observation angle flight path is fed back to the adaptive joint probability data association framework, combined with the initial target behavior prediction, the accuracy of the initial target state estimate is verified and adjusted, and an optimal target state estimate is generated; based on the optimal target state estimate, a multi-objective optimization algorithm is used to comprehensively calculate the energy consumption, flight efficiency and tracking accuracy of the UAV, and the flight trajectory of the UAV is evaluated and optimized again to obtain an optimal observation angle flight path;

[0088] In this embodiment, a deep learning algorithm is introduced to predict the behavior pattern of dynamic targets. This algorithm is trained with a large amount of historical data and can identify and predict the future behavior trend of targets, such as the walking path of pedestrians or the turning intention of vehicles. In addition, the terrain and obstacle information obtained in real time by the environmental perception module provides an important reference for path planning. The multi-objective optimization algorithm is used to comprehensively consider the energy consumption, flight efficiency and tracking accuracy of the UAV to ensure that the path planning is not only efficient but also energy-saving.

[0089] In the embodiment of the present application, the synchronous observation data stream from the visual sensor and millimeter-wave radar equipped by the UAV is first processed by matching the spatiotemporal consistency model to obtain the association results of the dynamic target contour change and the time series characteristics of the millimeter-wave radar reflection signal. Then, based on these association results, an adaptive joint probability data association framework is constructed, and the behavior pattern of the dynamic target is predicted in combination with the deep learning algorithm. The position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in three-dimensional space are estimated to generate an initial target state estimation value and an initial target behavior prediction. Subsequently, based on the initial target state estimation value, a forward-looking path planning algorithm is used, combined with the terrain and obstacle information obtained in real time by the environmental perception module, the flight trajectory of the UAV is dynamically generated and optimized to obtain an initial observation angle flight path, and then this path is fed back to the adaptive joint probability data association framework, combined with the initial target behavior prediction, the accuracy of the initial target state estimation value is verified and adjusted, and the optimal target state estimation value is generated. Finally, based on the optimal target state estimation value, a multi-objective optimization algorithm is used to comprehensively calculate the energy consumption, flight efficiency and tracking accuracy of the UAV, and the flight trajectory of the UAV is evaluated and optimized again to obtain the optimal observation angle flight path;

[0090] For example, in a busy urban traffic monitoring scenario, a drone performs a multi-target tracking task in the air. Using the spatiotemporal consistency model, the drone accurately captures the images and distance information of pedestrians and vehicles. The system constructs an adaptive joint probabilistic data association framework and combines deep learning algorithms to predict the walking paths of pedestrians and the driving directions of vehicles. Based on these predictions, the initial target state estimate and behavior prediction are generated. Next, a forward-looking path planning algorithm is used, combined with the terrain and obstacle information provided by the environmental perception module, to dynamically generate an optimal flight path so that the drone always maintains the best observation angle. Subsequently, this path is fed back to the adaptive joint probabilistic data association framework. Combined with the behavior prediction, the accuracy of the initial target state estimate is verified and adjusted, and the optimal target state estimate is generated. Finally, the multi-objective optimization algorithm is used to comprehensively consider the drone's energy consumption, flight efficiency and tracking accuracy, and the flight path is optimized again to ensure that the drone can efficiently complete the multi-target tracking task in a complex environment. During the whole process, the drone can not only respond quickly to the emergence of new targets, but also flexibly respond to environmental changes and maintain efficient monitoring capabilities.

[0091] In order to solve the problem of drone path planning and risk assessment in a complex environment, according to the previous embodiment, based on the initial target state estimation value, a forward-looking path planning algorithm is used, combined with the terrain and obstacle information obtained in real time by the environmental perception module, to dynamically generate and optimize the flight trajectory of the drone, and obtain the initial observation angle flight path, which specifically includes:

[0092] Using the initial target state estimation value and the terrain and obstacle information obtained in real time by the environmental perception module, combined with three-dimensional map data, multi-level modeling processing is performed on the environment around the UAV to obtain an environmental model. Based on the environmental model, a forward-looking path planning algorithm and a reinforcement learning optimizer are used to dynamically generate multiple candidate flight paths, and the potential risks and observation effects on each flight path are predicted to obtain a candidate flight path set and a candidate flight path risk assessment report. According to the candidate flight path set and the candidate flight path risk assessment report, combined with the maneuverability limitation of the UAV, weather condition prediction and dynamic obstacle avoidance algorithm, a safety assessment is performed on each flight path, and flight paths that meet preset conditions are screened out to generate an optimal flight path set. From the optimal flight path set, a multi-objective genetic algorithm is used to calculate the energy consumption, flight efficiency, tracking accuracy and flight path complexity of the UAV, and an optimal flight path is selected to obtain an initial observation angle flight path;

[0093] In this embodiment, a multi-level modeling process is introduced, that is, the terrain and obstacle information obtained in real time by the initial target state estimation value and the environment perception module is combined with the three-dimensional map data to perform multi-level modeling of the environment around the UAV to generate a detailed environmental model. This environmental model includes not only static terrain information, but also the location and movement trend of dynamic obstacles. The reinforcement learning optimizer is used to dynamically adjust the path planning strategy to ensure the optimality of path selection through continuous learning and optimization. The candidate flight path set refers to a set of possible flight paths generated by the system. Each path has been preliminarily evaluated. The candidate flight path risk assessment report records the potential risks and observation effects on each path in detail. The multi-objective genetic algorithm is an optimization algorithm used to comprehensively consider multiple goals (such as energy consumption, flight efficiency, tracking accuracy, and flight path complexity) to select the optimal flight path. The dynamic obstacle avoidance algorithm is used to detect and avoid obstacles in real time to ensure the safe flight of the UAV.

[0094] In the embodiment of the present application, firstly, the terrain and obstacle information obtained in real time by the initial target state estimation value and the environment perception module is combined with the three-dimensional map data to perform multi-level modeling processing on the environment around the UAV to obtain an environmental model. Then, based on this environmental model, a forward-looking path planning algorithm and a reinforcement learning optimizer are used to dynamically generate multiple candidate flight paths, and the potential risks and observation effects on each flight path are predicted to obtain a candidate flight path set and a candidate flight path risk assessment report. Then, based on the candidate flight path set and the candidate flight path risk assessment report, combined with the maneuverability limitations of the UAV, weather condition predictions, and dynamic obstacle avoidance algorithms, a safety assessment is performed on each flight path, and flight paths that meet the preset conditions are screened out to generate an optimal flight path set. Finally, from the optimal flight path set, a multi-objective genetic algorithm is used to calculate the energy consumption, flight efficiency, tracking accuracy, and flight path complexity of the UAV, and an optimal flight path is selected to obtain an initial observation angle flight path.

[0095] For example, in a busy urban traffic monitoring scenario, the UAV needs to perform multi-target tracking tasks in an environment with tall buildings, dense vehicles and pedestrians. First, the spatial-temporal consistency model is used to match the data of the visual sensor and the millimeter-wave radar to generate an initial target state estimate. At the same time, the environmental perception module obtains terrain and obstacle information in real time, and builds a detailed environmental model in combination with high-resolution three-dimensional map data. Based on this environmental model, the forward-looking path planning algorithm and reinforcement learning optimizer are used to dynamically generate multiple candidate flight paths, predict the potential risks and observation effects on each path, and generate a candidate flight path set and a candidate flight path risk assessment system. Assessment report, then according to the candidate flight path set and risk assessment report, combined with the UAV's maneuverability limitations, meteorological conditions forecast (such as wind speed, rainfall, etc.) and dynamic obstacle avoidance algorithm, a safety assessment was conducted on each flight path, and the flight paths that met the preset conditions were screened out, and the optimal flight path set was generated. Finally, a multi-objective genetic algorithm was used to comprehensively consider the UAV's energy consumption, flight efficiency, tracking accuracy and flight path complexity, and an optimal flight path was selected as the initial observation angle flight path. During the whole process, the UAV can not only respond quickly to the emergence of new targets, but also flexibly respond to environmental changes and maintain efficient monitoring capabilities.

[0096] In order to solve the problem of drone path planning and collaborative work in a complex environment, and to improve the intelligence level and safety of the system, according to the previous embodiment, based on the environmental model, a forward-looking path planning algorithm and a reinforcement learning optimizer are used to dynamically generate multiple candidate flight paths, and the potential risks and observation effects on each flight path are predicted to obtain a candidate flight path set and a candidate flight path risk assessment report, which specifically includes:

[0097] Using the environmental model and the real-time dynamic environmental prediction module, combined with a deep reinforcement learning optimizer, the UAV flight trajectory is intelligently planned, the environmental change trend is calculated, and a preliminary flight path set is obtained. Based on the preliminary flight path set, an adaptive multi-agent system and a swarm intelligence algorithm are used to simulate the collaborative behavior mode between multiple UAVs, and each flight path is dynamically adjusted and optimized to generate multiple candidate flight paths. The potential risks and observation effects of the multiple candidate flight paths are analyzed using three-dimensional map data, real-time perception information, and a risk assessment model based on a Bayesian network to generate a risk assessment report. The risk assessment report is associated with multiple candidate flight paths to generate an initial candidate flight path set, and the initial candidate flight path set is virtually simulated and tested through a digital twin platform to verify the performance of each candidate flight path and the impact of each candidate flight path on the UAV fleet, so as to obtain the best candidate flight path set, and the initial candidate flight path set is fed back to the safety assessment, and the initial observation angle flight path is obtained in combination with the behavior pattern prediction of the UAV cluster;

[0098] In this embodiment, a real-time dynamic environment prediction module is introduced to calculate the trend of environmental changes, and the flight trajectory of the UAV is intelligently planned in combination with a deep reinforcement learning optimizer. The preliminary flight path set refers to a set of possible flight paths generated by the system based on the environmental model and the real-time dynamic environment prediction module. The adaptive multi-agent system (MAS) and the swarm intelligence algorithm simulate the collaborative behavior mode between multiple UAVs, dynamically adjust and optimize each flight path, and generate multiple candidate flight paths. Three-dimensional map data, real-time perception information, and a risk assessment model based on a Bayesian network are used to analyze the potential risks and observation effects of multiple candidate flight paths and generate a risk assessment report. The digital twin platform is used for virtual simulation testing to verify the performance of each candidate flight path and its impact on the UAV fleet. Finally, the initial observation angle flight path is obtained by feeding back the candidate flight path set with simulation results in the safety assessment and combining the behavior pattern prediction of the UAV cluster.

[0099] In the embodiment of the present application, firstly, the environmental model and the real-time dynamic environmental prediction module are used in combination with the deep reinforcement learning optimizer to intelligently plan the flight trajectory of the UAV, calculate the environmental change trend, and obtain a preliminary flight path set. Then, based on this preliminary flight path set, an adaptive multi-agent system and a swarm intelligence algorithm are used to simulate the collaborative behavior mode between multiple UAVs, dynamically adjust and optimize each flight path, and generate multiple candidate flight paths. Then, the three-dimensional map data, real-time perception information, and a risk assessment model based on a Bayesian network are used to analyze the potential risks and observation effects of these candidate flight paths, generate a risk assessment report, associate the risk assessment report with multiple candidate flight paths, generate an initial candidate flight path set, and perform a virtual simulation test on the initial candidate flight path set through a digital twin platform to verify the performance of each candidate flight path and its impact on the UAV fleet, so as to obtain the best candidate flight path set. Finally, the candidate flight path set with the simulation results is fed back to the safety assessment, and the initial observation angle flight path is obtained in combination with the behavior pattern prediction of the UAV cluster.

[0100] For example, in a large urban security scenario, drones need to perform multi-target tracking tasks in conditions of high-rise buildings, dense vehicles and pedestrians, and constantly changing environments. First, the environmental model and real-time dynamic environmental prediction module are combined with a deep reinforcement learning optimizer to calculate environmental change trends, such as changes in traffic flow or changes in weather conditions, and generate a set of preliminary flight path sets. Next, based on this preliminary flight path set, an adaptive multi-agent system and swarm intelligence algorithm are used to simulate the collaborative behavior patterns between multiple drones, dynamically adjust and optimize each flight path, and generate multiple candidate flight paths. Then, three-dimensional map data, real-time perception information, and a risk assessment model based on a Bayesian network are used to predict the future of the project. , a detailed analysis of the potential risks and observation effects of these candidate flight paths was conducted, and a risk assessment report was generated. The risk assessment report was associated with multiple candidate flight paths, and an initial set of candidate flight paths was generated. These paths were virtually simulated and tested through the digital twin platform to verify the performance of each candidate flight path and its impact on the drone fleet. Finally, the system fed back the candidate flight path set with simulation results to the safety assessment, combined with the behavior pattern prediction of the drone cluster, and finally selected an optimal flight path as the initial observation angle flight path. During the whole process, the drone can not only respond quickly to the emergence of new targets, but also flexibly respond to environmental changes to ensure efficient monitoring and tracking capabilities.

[0101] In order to solve the observation error problem caused by environmental interference and improve the accuracy of target state estimation, as another embodiment, according to step 103, a Bayesian network is used to perform uncertainty modeling on the observation error caused by environmental interference to obtain initial observation data, and the initial observation data is integrated into the initial target state estimation value to generate an optimal target state estimation value. Based on the optimal target state estimation value and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to collaboratively process the dynamic target tracking information shared by multiple drones to generate an initial tracking task allocation plan, which specifically includes:

[0102] Using a Bayesian network, uncertainty modeling is performed on observation errors caused by environmental interference to obtain initial observation data, and the initial observation data is integrated into the initial target state estimate to generate an optimal target state estimate. Based on the optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to combine the dynamic target tracking information shared by multiple drones to perform collaborative processing to generate a preliminary tracking task allocation plan. The preliminary tracking task allocation plan is optimized and adjusted in combination with the current task load and remaining resource conditions of each drone to generate an initial tracking task allocation plan.

[0103] In this embodiment, a Bayesian network is introduced, which is a probabilistic graphical model used to represent the dependencies and uncertainties between variables. It models the observation errors caused by environmental interference (such as weather changes, electromagnetic interference, etc.) by quantifying the probability distribution of observation errors, and generates corrected initial observation data. The distributed collaborative perception mechanism refers to the sharing of sensor data and computing resources between multiple drones to achieve collaborative work and optimize the overall tracking effect. The preliminary tracking task allocation plan is a preliminary task arrangement generated based on the task requirements and current status of each drone, and the initial tracking task allocation plan is the final task allocation after further optimization and adjustment. In the embodiment of the present application, First, the Bayesian network is used to model the uncertainty of the observation error caused by environmental interference to generate the corrected initial observation data. Then, these corrected initial observation data are integrated into the initial target state estimate to generate a more accurate optimal target state estimate. Then, based on this optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to combine the dynamic target tracking information shared by multiple UAVs for collaborative processing to generate a preliminary tracking task allocation plan. Finally, the preliminary tracking task allocation plan is optimized and adjusted in combination with the current task load and remaining resources of each UAV to generate an initial tracking task allocation plan.

[0104] For example, in an urban security scenario, drones need to perform multi-target tracking tasks in a complex and ever-changing environment. First, when the drone encounters environmental interference such as rain or strong light, the system uses the Bayesian network to help correct the observation error and generate the corrected initial observation data. These corrected data are integrated into the initial target state estimate to generate the optimal target state estimate. Next, based on the optimal target state estimate and the optimal observation angle flight path, the system uses a distributed collaborative perception mechanism and combines the dynamic target tracking information shared by multiple drones for collaborative processing to generate a preliminary tracking task allocation plan. For example, at a busy traffic intersection, multiple drones jointly monitor the dynamics of pedestrians and vehicles to ensure that all targets are within the monitoring range. Then, the system optimizes and adjusts the preliminary tracking task allocation plan, taking into account the current task load and remaining resources of each drone, such as power level, sensor resources used, etc., and finally generates an initial tracking task allocation plan.

[0105] In order to solve the problem of unreasonable resource allocation, as another embodiment, as described in step 104, according to the initial tracking task allocation plan and the optimal observation angle flight path, an intelligent resource allocation strategy is implemented to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter wave radar to generate an initial resource allocation plan, which specifically includes:

[0106] According to the initial tracking task allocation plan and the optimal observation angle flight path, the task requirements and current status of each UAV are evaluated to obtain a task requirement evaluation result; based on the task requirement evaluation result, an intelligent resource allocation strategy is implemented to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter-wave radar to obtain the optimal sensor configuration parameters; according to the task requirement evaluation result, the sampling frequency and resolution of the visual sensor are adjusted to obtain the optimal parameter setting of the visual sensor; according to the task requirement evaluation result, the sampling frequency and resolution of the millimeter-wave radar are adjusted to obtain the optimal parameter setting of the millimeter-wave radar; the optimal parameter setting of the visual sensor and the optimal parameter setting of the millimeter-wave radar are integrated to generate an initial resource allocation plan;

[0107] In this embodiment, the mission requirement evaluation result is obtained by evaluating the mission requirement and current status of each UAV, and is used to guide subsequent resource allocation. The intelligent resource allocation strategy is a method for dynamically adjusting sensor parameters (such as sampling frequency and resolution) according to the mission requirement evaluation result to achieve optimal utilization of resources. The optimal sensor configuration parameters refer to the optimal settings of the visual sensor and the millimeter-wave radar under specific mission requirements, ensuring that high accuracy is maintained without wasting energy. The optimal parameter settings of the visual sensor and the optimal parameter settings of the millimeter-wave radar are the specific parameter adjustment results of the two sensors, respectively, aiming to meet their respective mission requirements;

[0108] In the embodiment of the present application, firstly, the task requirements and current status of each UAV are evaluated according to the initial tracking task allocation plan and the flight path of the optimal observation angle to obtain the task requirement evaluation result. Then, based on this evaluation result, the system implements an intelligent resource allocation strategy to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter-wave radar to obtain the optimal sensor configuration parameters. Then, according to the task requirement evaluation result, the sampling frequency and resolution of the visual sensor and the millimeter-wave radar are adjusted respectively to obtain the respective optimal parameter settings. Finally, the optimal parameter settings of the visual sensor and the optimal parameter settings of the millimeter-wave radar are integrated to generate an initial resource allocation plan. This process ensures the rationality and efficiency of resource allocation and improves the overall performance of the system.

[0109] For example, in an urban security scenario, multiple drones need to perform multi-target tracking tasks in a complex environment. First, the system evaluates the mission requirements and current status of each drone based on the initial tracking mission allocation plan and the optimal observation angle flight path. For example, one drone is responsible for close-range tracking of fast-moving vehicles, while another is responsible for large-scale monitoring of pedestrian traffic. The evaluation results show that the drone responsible for close-range tracking requires a higher sampling frequency and resolution to capture the rapidly changing target state, while the drone responsible for large-scale monitoring can work at a lower sampling frequency and resolution to save energy. Next, based on these mission requirement evaluation results, the system implements an intelligent resource allocation strategy to dynamically adjust the sampling frequency and resolution of the visual sensor and millimeter-wave radar. For the drone responsible for close-range tracking, the system increases the sampling frequency of its visual sensor to the maximum value and increases the resolution to ensure that the details of the target can be clearly captured. At the same time, the sampling frequency of the millimeter-wave radar is also increased accordingly to cope with high-speed moving targets. For the drone responsible for large-scale monitoring, the parameter settings of the visual sensor and millimeter-wave radar are appropriately reduced to extend the flight time. Finally, the system integrates these optimal parameter settings to generate an initial resource allocation plan.

[0110] In order to improve the response speed and flexibility of task allocation of the UAV in a dynamic environment, as another embodiment, as described in step 105, when it is detected that a dynamic target other than the existing dynamic target enters the observation range or the current dynamic target leaves the observation range, the rapid response protocol is started, the priority in the current monitoring scene is evaluated, and a priority list is obtained. Based on the priority list and the optimal observation angle flight path, the initial tracking task allocation plan is adjusted to obtain the optimal tracking task allocation plan, and the optimal tracking task allocation plan is fed back to the intelligent resource allocation strategy to generate the optimal resource allocation plan, which specifically includes:

[0111] Using the visual sensor and millimeter-wave radar equipped by the UAV, real-time monitoring is performed on the synchronous observation data stream from the UAV to detect the situation where dynamic targets other than existing dynamic targets enter the observation range or the existing dynamic targets leave the observation range, and a trigger condition confirmation result is obtained. According to the trigger condition confirmation result, a rapid response protocol is started to re-evaluate the current monitoring scene. Based on the trigger condition confirmation result, combined with the observation data obtained in real time and the initial resource allocation plan, priority evaluation is performed on the dynamic targets in the current monitoring scene to generate a priority list. Using the priority list and the optimal observation angle flight path, the initial tracking task allocation plan is adjusted to obtain the optimal tracking task allocation plan. The optimal tracking task allocation plan is fed back to the intelligent resource allocation strategy. According to the optimal tracking task allocation plan, the sampling frequency and resolution of the visual sensor and the millimeter-wave radar are adjusted to generate the optimal resource allocation plan.

[0112] In this embodiment, the rapid response protocol is a mechanism that is immediately activated when the above situation is detected, and is used to re-evaluate the current monitoring scene and make adjustments quickly. The priority evaluation process refers to the evaluation of the importance and urgency of the dynamic targets in the current monitoring scene based on the trigger condition confirmation result, combined with the observation data obtained in real time and the initial resource allocation plan, and the priority list is generated. The best tracking task allocation plan is the optimal task allocation plan adjusted based on the priority list and the flight path of the best observation angle, ensuring that the task arrangement of each drone is reasonable and efficient. The best resource allocation plan is the final resource allocation plan after adjusting the sampling frequency and resolution of the visual sensor and the millimeter wave radar according to the best tracking task allocation plan;

[0113] In the embodiment of the present application, the visual sensor and millimeter-wave radar equipped by the drone are first used to monitor the synchronous observation data stream from the drone in real time, detect the situation where new targets other than existing dynamic targets enter the observation range or existing targets leave, and obtain the trigger condition confirmation result. Then, based on the trigger condition confirmation result, the rapid response protocol is started to re-evaluate the current monitoring scene. Then, based on the trigger condition confirmation result, combined with the observation data obtained in real time and the initial resource allocation plan, the dynamic targets in the current monitoring scene are prioritized and evaluated to generate a priority list. Subsequently, the priority list and the optimal observation angle flight path are used to adjust the initial tracking task allocation plan to obtain the optimal tracking task allocation plan. Finally, the optimal tracking task allocation plan is fed back to the intelligent resource allocation strategy, and the sampling frequency and resolution of the visual sensor and the millimeter-wave radar are adjusted according to the optimal tracking task allocation plan to generate the optimal resource allocation plan.

[0114] For example, in a busy urban traffic monitoring scenario, multiple drones are responsible for tracking pedestrians, vehicles, and other aircraft. First, the system uses the visual sensors and millimeter-wave radars equipped on the drones to monitor the synchronous observation data streams from the drones in real time. When a new vehicle or pedestrian is detected to suddenly enter the drone's observation range, or a target leaves, the system obtains a trigger condition confirmation result. Next, based on this trigger condition confirmation result, the system immediately initiates a rapid response protocol to re-evaluate the current monitoring scenario. The system combines the real-time observation data and the initial resource allocation plan to prioritize dynamic targets in the current monitoring scenario. For example, fast-moving vehicles may be given higher priority, while slow-moving pedestrians may have lower priority. After generating the priority list, the system uses this list and the optimal observation angle flight path to adjust the initial tracking task allocation plan to obtain the optimal tracking task allocation plan. For high-priority targets that require closer tracking, the system increases the sampling frequency and resolution of the visual sensor and millimeter-wave radar; for low-priority targets, the parameter settings are appropriately lowered to save energy. Finally, the optimal tracking task allocation plan is fed back to the intelligent resource allocation strategy to adjust the sensor parameters and generate the optimal resource allocation plan.

[0115] Through the above improvements, the present invention significantly enhances the response speed of the UAV in a complex dynamic environment and the flexibility of task allocation, which not only improves the real-time and adaptability of the system, but also optimizes resource utilization, ensuring that the UAV can efficiently complete multi-target tracking tasks in a complex environment.

[0116] Figure 2 The present invention provides a schematic diagram of a multi-target tracking system for unmanned aerial vehicles based on the fusion of visual and millimeter-wave radar information. Figure 2As shown, the system includes:

[0117] The matching module 21 uses a spatiotemporal consistency model to match the synchronous observation data streams from the visual sensor equipped by the UAV and the millimeter-wave radar to obtain the correlation result between the contour change of the dynamic target and the time series characteristics of the millimeter-wave radar reflection signal, wherein the dynamic target includes: pedestrians, vehicles and other aircraft;

[0118] A construction module 22 is used to construct an adaptive joint probability data association framework according to the association result of the dynamic target contour change and the millimeter wave radar reflection signal time series characteristics, and use the adaptive joint probability data association framework to estimate the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter wave radar equipped by the UAV in three-dimensional space, generate an initial target state estimation value, and based on the initial target state estimation value, use a forward-looking path planning algorithm to dynamically generate and optimize the flight trajectory of the UAV to obtain the flight path with the best observation angle;

[0119] The processing module 23 is used to use a Bayesian network to perform uncertainty modeling on the observation error caused by environmental interference to obtain initial observation data, and integrate the initial observation data into the initial target state estimate to generate an optimal target state estimate. Based on the optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to collaboratively process the dynamic target tracking information shared by multiple UAVs to generate an initial tracking task allocation plan;

[0120] The adjustment module 24 is used to implement an intelligent resource allocation strategy according to the initial tracking task allocation plan and the optimal observation angle flight path, dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter wave radar, and generate an initial resource allocation plan;

[0121] The generation module 25 is used to start the rapid response protocol based on the initial resource allocation plan, evaluate the priority under the current monitoring scenario, obtain a priority list, adjust the initial tracking task allocation plan based on the priority list and the flight path with the best observation angle, obtain the best tracking task allocation plan, and feed the best tracking task allocation plan back to the intelligent resource allocation strategy to generate the best resource allocation plan. Figure 2 The UAV multi-target tracking system based on vision and millimeter-wave radar information fusion can perform Figure 1The implementation principle and technical effect of the method for tracking multiple targets of unmanned aerial vehicles based on the fusion of visual and millimeter-wave radar information described in the embodiment shown are not described in detail. The specific way in which each module and unit performs operations in the above embodiment of the unmanned aerial vehicle multi-target tracking system based on the fusion of visual and millimeter-wave radar information has been described in detail in the embodiment of the method, and will not be described in detail here.

[0122] In one possible design, Figure 2 The UAV multi-target tracking system based on vision and millimeter-wave radar information fusion of the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0123] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0124] The processing component 32 is used to match the synchronous observation data streams from the visual sensor and millimeter-wave radar equipped by the drone using a spatiotemporal consistency model to obtain the association results of the dynamic target contour changes and the time series characteristics of the millimeter-wave radar reflection signal, wherein the dynamic targets include: pedestrians, vehicles and other aircraft; according to the association results of the dynamic target contour changes and the time series characteristics of the millimeter-wave radar reflection signal, an adaptive joint probability data association framework is constructed, and the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the drone in three-dimensional space are estimated by using the adaptive joint probability data association framework to generate an initial target state estimation value; based on the initial target state estimation value, a forward-looking path planning algorithm is used to dynamically generate and optimize the flight trajectory of the drone to obtain the flight path with the best observation angle; using a Bayesian network, the observation error caused by environmental interference is subjected to uncertainty modeling processing to obtain initial observation data, and the initial observation data is integrated into the The method comprises the steps of: inputting the initial target state estimation value into the initial target state estimation value to generate the optimal target state estimation value; based on the optimal target state estimation value and the optimal observation angle flight path, using the distributed collaborative perception mechanism, collaboratively processing the dynamic target tracking information shared by multiple UAVs, and generating an initial tracking task allocation plan; implementing the intelligent resource allocation strategy according to the initial tracking task allocation plan and the optimal observation angle flight path, dynamically adjusting the sampling frequency and resolution of the visual sensor and the millimeter wave radar, and generating an initial resource allocation plan; based on the initial resource allocation plan, when it is detected that a dynamic target other than the existing dynamic target enters the observation range or the current dynamic target leaves the observation range, starting the rapid response protocol, evaluating the priority under the current monitoring scenario, and obtaining a priority list; adjusting the initial tracking task allocation plan based on the priority list and the optimal observation angle flight path to obtain the optimal tracking task allocation plan, and feeding the optimal tracking task allocation plan back to the intelligent resource allocation strategy to generate the optimal resource allocation plan.

[0125] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0126] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0127] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0128] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0129] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0130] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0131] The embodiment of the present invention further provides a computer storage medium storing a computer program, which can achieve the above-mentioned Figure 1 The illustrated embodiment is a method for tracking multiple targets of a UAV based on the fusion of visual and millimeter-wave radar information.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0134] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-target tracking method for unmanned aerial vehicles based on the fusion of visual and millimeter-wave radar information, characterized in that: include: Using the spatiotemporal consistency model, the synchronous observation data streams from the visual sensor and millimeter-wave radar equipped by the UAV are matched and processed to obtain the correlation results between the contour changes of dynamic targets and the time series characteristics of the millimeter-wave radar reflection signal, wherein the dynamic targets include pedestrians, vehicles and other aircraft; According to the association results of the dynamic target contour changes and the time series characteristics of the millimeter-wave radar reflection signal, an adaptive joint probability data association framework is constructed, and the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in three-dimensional space are estimated by using the adaptive joint probability data association framework to generate an initial target state estimation value. Based on the initial target state estimation value, a forward-looking path planning algorithm is used to dynamically generate and optimize the flight trajectory of the UAV to obtain a flight path with an optimal observation angle; Using a Bayesian network, uncertainty modeling is performed on the observation error caused by environmental interference to obtain initial observation data, and the initial observation data is integrated into the initial target state estimate to generate an optimal target state estimate. Based on the optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to collaboratively process the dynamic target tracking information shared by multiple UAVs to generate an initial tracking task allocation plan. According to the initial tracking task allocation plan and the optimal observation angle flight path, an intelligent resource allocation strategy is implemented to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter wave radar to generate an initial resource allocation plan; Based on the initial resource allocation plan, when it is detected that a dynamic target other than the existing dynamic target enters the observation range or the current dynamic target leaves the observation range, the rapid response protocol is started, the priority under the current monitoring scenario is evaluated, and a priority list is obtained. Based on the priority list and the optimal observation angle flight path, the initial tracking task allocation plan is adjusted to obtain the optimal tracking task allocation plan, and the optimal tracking task allocation plan is fed back to the intelligent resource allocation strategy to generate the optimal resource allocation plan.

2. The method according to claim 1, characterized in that According to the association result of the dynamic target contour change and the time series characteristics of the millimeter-wave radar reflection signal, an adaptive joint probability data association framework is constructed, and the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in three-dimensional space are estimated by using the adaptive joint probability data association framework to generate an initial target state estimation value. Based on the initial target state estimation value, a forward-looking path planning algorithm is used to dynamically generate and optimize the flight trajectory of the UAV to obtain the flight path with the best observation angle, including: By using the correlation results of the dynamic target contour changes and the time series characteristics of the millimeter-wave radar reflection signal, an adaptive joint probability data association framework is constructed, and the behavior pattern of the dynamic target is predicted in combination with the deep learning algorithm. The position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter-wave radar equipped by the UAV in the three-dimensional space are estimated to obtain the initial target state estimation value and the initial target behavior prediction; Based on the initial target state estimation value, a forward-looking path planning algorithm is used, combined with the terrain and obstacle information obtained in real time by the environmental perception module, to dynamically generate and optimize the flight trajectory of the UAV to obtain the initial observation angle flight path; Feeding back the initial observation angle flight path into an adaptive joint probability data association framework, combining the initial target behavior prediction, verifying and adjusting the accuracy of the initial target state estimate, and generating an optimal target state estimate; Based on the optimal target state estimation value, a multi-objective optimization algorithm is used to comprehensively calculate the energy consumption, flight efficiency and tracking accuracy of the UAV, and the flight trajectory of the UAV is evaluated and optimized again to obtain the flight path with the best observation angle.

3. The method according to claim 2, characterized in that Based on the initial target state estimate, a forward-looking path planning algorithm is used to dynamically generate and optimize the flight trajectory of the UAV in combination with the terrain and obstacle information obtained in real time by the environmental perception module to obtain the initial observation angle flight path, including: Using the initial target state estimation value and the terrain and obstacle information obtained in real time by the environment perception module, combined with the three-dimensional map data, a multi-level modeling process is performed on the environment around the UAV to obtain an environment model; Based on the environmental model, a forward-looking path planning algorithm and a reinforcement learning optimizer are used to dynamically generate multiple candidate flight paths, and potential risks and observation effects on each flight path are predicted to obtain a candidate flight path set and a candidate flight path risk assessment report; Based on the candidate flight path set and the candidate flight path risk assessment report, combined with the maneuverability limitations of the UAV, weather condition predictions, and dynamic obstacle avoidance algorithms, a safety assessment is performed on each flight path, flight paths that meet preset conditions are screened out, and an optimal flight path set is generated; From the optimal flight path set, a multi-objective genetic algorithm is used to calculate the energy consumption, flight efficiency, tracking accuracy and flight path complexity of the UAV, and an optimal flight path is selected to obtain an initial observation angle flight path.

4. The method according to claim 3, characterized in that Based on the environmental model, a forward-looking path planning algorithm and a reinforcement learning optimizer are used to dynamically generate multiple candidate flight paths, and the potential risks and observation effects on each flight path are predicted to obtain a candidate flight path set and a candidate flight path risk assessment report, including: Using the environmental model and the real-time dynamic environment prediction module, combined with the deep reinforcement learning optimizer, the UAV flight trajectory is intelligently planned, the environmental change trend is calculated, and a preliminary flight path set is obtained; Based on the preliminary flight path set, an adaptive multi-agent system and a swarm intelligence algorithm are used to simulate the collaborative behavior mode among multiple drones, dynamically adjust and optimize each flight path, and generate multiple candidate flight paths; Analyze the potential risks and observation effects of the multiple candidate flight paths using three-dimensional map data, real-time perception information, and a risk assessment model based on a Bayesian network to generate a risk assessment report; Associating the risk assessment report with multiple candidate flight paths to generate an initial candidate flight path set, and performing a virtual simulation test on the initial candidate flight path set through a digital twin platform to verify the performance of each candidate flight path and the impact of each candidate flight path on the UAV fleet, so as to obtain an optimal candidate flight path set; The initial candidate flight path set is fed back into the safety assessment, and combined with the behavior pattern prediction of the drone cluster to obtain the initial observation angle flight path.

5. The method according to claim 1, characterized in that Using a Bayesian network, uncertainty modeling is performed on the observation error caused by environmental interference to obtain initial observation data, and the initial observation data is integrated into the initial target state estimate to generate an optimal target state estimate. Based on the optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to collaboratively process the dynamic target tracking information shared by multiple UAVs to generate an initial tracking task allocation plan, including: Using Bayesian network, uncertainty modeling is performed on observation errors caused by environmental interference to obtain initial observation data. Integrating the initial observation data into the initial target state estimate to generate an optimal target state estimate; Based on the optimal target state estimation value and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to combine the dynamic target tracking information shared by multiple UAVs to perform collaborative processing and generate a preliminary tracking task allocation plan; The preliminary tracking task allocation plan is optimized and adjusted in combination with the current task load and remaining resource conditions of each UAV to generate an initial tracking task allocation plan.

6. The method according to claim 1, characterized in that According to the initial tracking task allocation plan and the optimal observation angle flight path, an intelligent resource allocation strategy is implemented to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter wave radar to generate an initial resource allocation plan, including: According to the initial tracking task allocation plan and the optimal observation angle flight path, the task requirements and current status of each UAV are evaluated to obtain a task requirement evaluation result; Based on the task requirement assessment results, an intelligent resource allocation strategy is implemented to dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter-wave radar to obtain the optimal sensor configuration parameters; According to the task requirement assessment result, the sampling frequency and resolution of the visual sensor are adjusted to obtain the optimal parameter setting of the visual sensor; According to the task requirement assessment result, the sampling frequency and resolution of the millimeter wave radar are adjusted to obtain the optimal parameter setting of the millimeter wave radar; The optimal parameter settings of the visual sensor and the optimal parameter settings of the millimeter-wave radar are integrated to generate an initial resource configuration plan.

7. The method according to claim 1, characterized in that When a dynamic target other than an existing dynamic target is detected to enter the observation range or the current dynamic target leaves the observation range, a rapid response protocol is initiated to evaluate the priority in the current monitoring scenario to obtain a priority list, and based on the priority list and the optimal observation angle flight path, the initial tracking task allocation plan is adjusted to obtain the optimal tracking task allocation plan, and the optimal tracking task allocation plan is fed back to the intelligent resource allocation strategy to generate the optimal resource allocation plan, including: Using the visual sensor and millimeter wave radar equipped by the UAV, the synchronous observation data stream from the UAV is monitored in real time to detect the dynamic targets other than the existing dynamic targets entering the observation range or the existing dynamic targets leaving the observation range, and obtain the trigger condition confirmation result; According to the trigger condition confirmation result, a rapid response protocol is initiated to re-evaluate the current monitoring scene; Based on the trigger condition confirmation result, combined with the real-time acquired observation data and the initial resource allocation plan, priority evaluation processing is performed on the dynamic targets in the current monitoring scenario to generate a priority list; Using the priority list and the optimal observation angle flight path, the initial tracking task allocation scheme is adjusted to obtain the optimal tracking task allocation scheme; The optimal tracking task allocation plan is fed back into the intelligent resource allocation strategy, and the sampling frequency and resolution of the visual sensor and the millimeter-wave radar are adjusted according to the optimal tracking task allocation plan to generate the optimal resource configuration plan.

8. A multi-target tracking system for unmanned aerial vehicles based on the fusion of visual and millimeter-wave radar information, characterized in that: include: A matching module uses a spatiotemporal consistency model to match the synchronous observation data streams from the visual sensor and the millimeter-wave radar equipped by the UAV to obtain the correlation results between the contour changes of dynamic targets and the time series characteristics of the millimeter-wave radar reflection signal, wherein the dynamic targets include pedestrians, vehicles and other aircraft; A construction module is used to construct an adaptive joint probability data association framework according to the association result of the dynamic target contour change and the millimeter wave radar reflection signal time series characteristics, and use the adaptive joint probability data association framework to estimate the position, posture and motion parameters of each dynamic target observed by the visual sensor and millimeter wave radar equipped by the UAV in three-dimensional space, generate an initial target state estimation value, and based on the initial target state estimation value, use a forward-looking path planning algorithm to dynamically generate and optimize the flight trajectory of the UAV to obtain the flight path with the best observation angle; A processing module is used to use a Bayesian network to perform uncertainty modeling on observation errors caused by environmental interference to obtain initial observation data, and integrate the initial observation data into the initial target state estimate to generate an optimal target state estimate. Based on the optimal target state estimate and the optimal observation angle flight path, a distributed collaborative perception mechanism is used to collaboratively process dynamic target tracking information shared by multiple UAVs to generate an initial tracking task allocation plan; An adjustment module is used to implement an intelligent resource allocation strategy according to the initial tracking task allocation plan and the optimal observation angle flight path, dynamically adjust the sampling frequency and resolution of the visual sensor and the millimeter wave radar, and generate an initial resource allocation plan; A generation module is used to start a rapid response protocol based on the initial resource allocation plan, evaluate the priority in the current monitoring scenario, obtain a priority list, adjust the initial tracking task allocation plan based on the priority list and the optimal observation angle flight path, obtain the optimal tracking task allocation plan, and feed the optimal tracking task allocation plan back to the intelligent resource allocation strategy to generate the optimal resource allocation plan.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a UAV multi-target tracking method based on vision and millimeter-wave radar information fusion as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, a method for tracking multiple targets of a drone based on the fusion of visual and millimeter-wave radar information as described in any one of claims 1 to 7 is implemented.

Images