An Unmanned Aerial Vehicle Detection and Tracking Method and System Based on the Combination of LiDAR and Vision

Abstract

The present invention provides a method and system for detecting and tracking unmanned aerial vehicles (UAVs) based on the combination of lidar and vision. Specifically, the present invention utilizes a distributed computing framework to obtain timestamp information, generates a multi-source dataset based on the timestamp information, performs data fusion using a deep reinforcement learning algorithm based on the multi-source dataset to obtain multi-source heterogeneous data, uses the multi-source heterogeneous data to obtain the UAV behavior pattern and predicted motion trajectory, analyzes the UAV behavior pattern and predicted motion trajectory through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensing resource allocation scheme, processes intelligent vision sensing nodes based on the optimized sensing resource allocation scheme to obtain an emergency correction program, integrates edge computing capabilities according to the emergency correction program, and generates UAV detection and tracking instructions; the technical solution provided by the present invention improves the quality and accuracy of data fusion and enhances the sensing ability.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of unmanned aerial vehicles, and particularly to an unmanned aerial vehicle detection and tracking method based on the combination of lidar and vision. Background Art

[0002] With the rapid development of unmanned aerial vehicle technology, unmanned aerial vehicles are increasingly widely used in fields such as military reconnaissance, logistics distribution, agricultural monitoring, and environmental monitoring. These application scenarios pose strict requirements for the detection and tracking of unmanned aerial vehicles: they need to have high-precision positioning capabilities, real-time response speeds, and stable tracking performance in complex and changing environments. Especially in dynamic environments, how to accurately identify and continuously track multiple unmanned aerial vehicles has become a technical problem that needs to be solved urgently;

[0003] Currently, the detection and tracking of unmanned aerial vehicles mainly rely on a single sensor or a combination of a limited number of sensors. For example, using a radar system can provide long-distance target detection, but its resolution is low and it is difficult to distinguish small or low-speed moving targets; while visual sensing can provide high-resolution images, but it performs poorly under complex lighting conditions or occlusion. In addition, some studies have also tried to combine lidar (Li DAR) and visual sensing to improve detection accuracy, but such methods are usually limited to target recognition in static or semi-static environments and lack effective adaptability to dynamic and changing environments;

[0004] Moreover, existing solutions have obvious deficiencies when facing complex and changing three-dimensional dynamic environments. First, a single sensor or a simple combination is difficult to simultaneously meet the requirements of high precision, real-time performance, and robustness; second, traditional data fusion methods often ignore the problem of time synchronization between different sensors, resulting in inaccurate information fusion; finally, most existing solutions fail to fully consider the importance of the behavior patterns of unmanned aerial vehicles and their predicted movement trajectories, and cannot achieve intelligent task allocation and resource optimization configuration, thus limiting the overall efficiency of the system. Summary of the Invention

[0005] The embodiments of the present invention provide an unmanned aerial vehicle detection and tracking method and system based on the combination of lidar and vision, so as to solve the problems of inaccurate information fusion, inability to achieve intelligent task allocation and resource optimization configuration, and lack of effective adaptability to dynamic and changing environments in the prior art.

[0006] In a first aspect, the embodiments of the present invention provide an unmanned aerial vehicle detection and tracking method based on the combination of lidar and vision, including:

[0007] Performing real-time communication processing on the high-resolution lidar and intelligent vision sensing nodes in the deployed hybrid sensor array by using a distributed computing framework to obtain timestamp information;

[0008] According to the timestamp information, perform spatio-temporal calibration on the angular information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent vision sensing node to generate a multi-source data set;

[0009] Based on the multi-source data set, use a deep reinforcement learning algorithm for data fusion to obtain multi-source heterogeneous data, and use the multi-source heterogeneous data to obtain the UAV behavior pattern and predicted motion trajectory through behavior pattern mining;

[0010] According to the UAV behavior pattern and predicted motion trajectory, perform analysis through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensing resource allocation scheme. Based on the optimized sensing resource allocation scheme, process the working state adjustment and tracking strategy of the intelligent vision sensing node to obtain an emergency correction program;

[0011] According to the emergency correction program, integrate edge computing capabilities into the intelligent vision sensing node to process local data and make decisions, and generate UAV detection and tracking instructions.

[0012] Optionally, according to the UAV behavior pattern and predicted motion trajectory, perform analysis through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensing resource allocation scheme. Based on the optimized sensing resource allocation scheme, process the working state adjustment and tracking strategy of the intelligent vision sensing node to obtain an emergency correction program, including:

[0013] Use the UAV behavior pattern and predicted motion trajectory to dynamically evaluate the task priority, coverage range of the intelligent vision sensing node, and the complementarity between the intelligent vision sensing node and lidar data to obtain an initial sensing resource allocation model;

[0014] Based on the initial sensing resource allocation model, apply a multi-objective optimization algorithm to adjust the working parameters of the intelligent vision sensing node, and evaluate the cooperation efficiency and energy consumption of the intelligent vision sensing node to obtain an optimized sensing resource allocation scheme;

[0015] Use the optimized sensing resource allocation scheme to perform tests under different interference conditions through simulation technology, identify potential problems, and formulate corresponding preventive measures;

[0016] Convert the preventive measures into specific operation instructions, and combine the dynamically monitored changes of the UAV to adjust the working state and tracking strategy of the intelligent vision sensing node in real time to generate an emergency correction program.

[0017] Optionally, based on the initial sensing resource configuration model, apply a multi-objective optimization algorithm to adjust the working parameters of the intelligent vision sensing nodes, and evaluate the collaboration efficiency and energy consumption of the intelligent vision sensing nodes to obtain an optimized sensing resource configuration scheme, including:

[0018] Utilize the initial sensing resource configuration model, combine the UAV behavior patterns and predicted movement trajectories, and perform initial configuration on the working parameters of the intelligent vision sensing nodes to obtain an initial parameter configuration set, where the working parameters include: scanning frequency, image resolution, viewing angle, and data transmission priority;

[0019] According to the initial parameter configuration set, apply a multi-objective optimization algorithm to evaluate the collaboration efficiency, energy consumption, and real-time response speed among the intelligent vision sensing nodes. Through simulation to simulate the interaction effects in different scenarios, dynamically iteratively adjust the working parameters of each intelligent vision sensing node to form a candidate parameter configuration scheme;

[0020] Based on the candidate parameter configuration scheme, introduce a dynamic risk assessment system, conduct risk quantification analysis on each candidate configuration scheme, calculate the impacts of external environmental changes and internal system states, and screen out high-reliability configuration schemes;

[0021] Utilize the high-reliability configuration scheme to implement an environmental adaptability evaluation mechanism, conduct simulation tests for complex electromagnetic environments and optical interference conditions, and obtain an optimized sensing resource configuration scheme.

[0022] Optionally, according to the initial parameter configuration set, apply a multi-objective optimization algorithm to evaluate the collaboration efficiency, energy consumption, and real-time response speed among the intelligent vision sensing nodes. Through simulation to simulate the interaction effects in different scenarios, dynamically iteratively adjust the working parameters of each intelligent vision sensing node to form a candidate parameter configuration scheme, including:

[0023] Utilize the initial parameter configuration set, apply a multi-objective optimization algorithm to evaluate the collaboration efficiency, energy consumption, and real-time response speed among the intelligent vision sensing nodes. Through simulation to simulate the interaction effects in different scenarios, generate a preliminary adjusted parameter configuration scheme;

[0024] Based on the preliminary adjusted parameter configuration scheme, introduce a scenario adaptability analysis module to evaluate the applicability of each intelligent vision sensing node in different task environments, and form a scenario-adapted parameter configuration scheme;

[0025] According to the scenario-adapted parameter configuration scheme, through a dynamic iterative adjustment mechanism, optimize the working parameters of each intelligent vision sensing node to obtain the best parameter configuration scheme;

[0026] Using the optimal parameter configuration scheme, combined with the multi-source information fusion technology, and integrating lidar and visual sensing data, adjust the working parameters of the intelligent visual sensing nodes to obtain a candidate parameter configuration scheme.

[0027] Optionally, according to the timestamp information, perform spatio-temporal calibration on the angle information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent visual sensing node to generate a multi-source data set, including:

[0028] Using the timestamp information combined with the environmental perception algorithm, perform precise time synchronization processing on the angle information point cloud data stream of the high-resolution lidar and the multi-view image frames captured by the intelligent visual sensing nodes to obtain time-synchronized data pairs, and record the environmental parameters;

[0029] Based on the time-synchronized data pairs, apply the adaptive space coordinate conversion algorithm, and dynamically adjust the conversion parameters in combination with the recorded environmental parameters to map the three-dimensional point cloud data of the lidar to the two-dimensional image plane of each intelligent visual sensing node to generate a spatially aligned data set;

[0030] According to the spatially aligned data set, introduce a geometric consistency verification mechanism, detect and correct geometric mismatch problems through a deep learning model, and at the same time use semantic segmentation technology to identify and eliminate data interference in non-target areas to form an initial multi-source data set;

[0031] Using the initial multi-source data set, perform spatio-temporal correlation optimization processing, fuse continuous data frames in the time dimension, and use a time series prediction model to predict the data distribution to generate a target multi-source data set.

[0032] Optionally, based on the multi-source data set, use the deep reinforcement learning algorithm for data fusion to obtain multi-source heterogeneous data, and use the multi-source heterogeneous data to obtain the UAV behavior pattern and predicted motion trajectory through behavior pattern mining, including:

[0033] Using the multi-source data set, use the deep reinforcement learning algorithm to perform joint analysis and processing on the angle information point cloud data stream of the lidar and the multi-view image frames captured by the intelligent visual sensing nodes to obtain multi-source heterogeneous data;

[0034] According to the multi-source heterogeneous data, construct a multi-modal perception model, integrate the data characteristics from different sensors, and generate a reinforced multi-modal perception model;

[0035] Based on the reinforced multi-modal perception model, apply the behavior pattern mining algorithm to extract the behavior characteristics of the UAV from the multi-source heterogeneous data to form an initial behavior pattern library, and the initial behavior pattern library includes the position, speed, direction, and behavior intention of the UAV;

[0036] Using the initial behavior pattern library, combining historical data and real-time updated perception information, predicting the motion trajectory of the UAV through a time series prediction model, generating the UAV behavior pattern and the predicted motion trajectory, wherein the real-time updated perception information includes: multi-view image frames, angular information point cloud data streams, environmental changes, UAV state updates, as well as external interference and mission instruction changes.

[0037] Optionally, according to the emergency correction procedure, integrating edge computing capabilities into the intelligent vision sensing nodes, processing local data and making decisions, generating UAV detection and tracking instructions, including:

[0038] Using the emergency correction procedure, deploying the edge computing module to the intelligent vision sensing nodes, enabling each intelligent vision sensing node to have local data processing capabilities, and obtaining a distributed computing network;

[0039] Based on the distributed computing network, performing real-time preprocessing on the local data collected by the intelligent vision sensing nodes, extracting feature information related to UAV detection and tracking, and obtaining optimized local feature data;

[0040] According to the optimized local feature data, applying an intelligent decision-making algorithm, independently executing a fast decision-making process on each intelligent vision sensing node, determining the best action plan for the current environment, and generating preliminary detection and tracking instructions;

[0041] Based on the preliminary detection and tracking instructions, performing instruction fusion through a cooperation mechanism under a self-organizing network, adjusting and optimizing each intelligent vision sensing node, and obtaining comprehensive UAV detection and tracking instructions.

[0042] In a second aspect, an embodiment of the present invention provides a UAV detection and tracking system based on the combination of lidar and vision, including:

[0043] A processing module, configured to perform real-time communication processing on the high-resolution lidar and intelligent vision sensing nodes in the deployed hybrid sensor array by using a distributed computing framework, and obtain timestamp information;

[0044] A calibration module, configured to perform spatio-temporal calibration on the angular information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent vision sensing node according to the timestamp information, and generate a multi-source data set;

[0045] A fusion module, configured to perform data fusion on the multi-source data set by using a deep reinforcement learning algorithm to obtain multi-source heterogeneous data, and use the multi-source heterogeneous data to obtain the UAV behavior pattern and the predicted motion trajectory through behavior pattern mining;

[0046] An analysis module, configured to analyze according to the UAV behavior pattern and predicted motion trajectory through an intelligent task allocation protocol under a self-organizing network, generate an optimized sensing resource allocation scheme, and process the working state adjustment and tracking strategy of the intelligent vision sensing nodes based on the optimized sensing resource allocation scheme to obtain an emergency correction program;

[0047] A formulation module, configured to integrate edge computing capabilities into the intelligent vision sensing nodes according to the emergency correction program, process local data and make decisions, and generate UAV detection and tracking instructions.

[0048] In a third aspect, an embodiment of the present invention provides a computing device, including a processor and a memory. A computer program is stored in the memory, and the processor is configured to run the computer program to execute any one of the methods for detecting and tracking a UAV based on the combination of lidar and vision in the first aspect.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, they implement any one of the methods for detecting and tracking a UAV based on the combination of lidar and vision in the first aspect.

[0050] In an embodiment of the present invention, a distributed computing framework is used to perform real-time communication processing on a high-resolution lidar and intelligent vision sensing nodes in a deployed hybrid sensor array to obtain timestamp information; according to the timestamp information, spatio-temporal calibration is performed on the angular information point cloud data stream generated by the high-resolution lidar and multi-view image frames captured by each intelligent vision sensing node to generate a multi-source data set; based on the multi-source data set, a deep reinforcement learning algorithm is used for data fusion to obtain multi-source heterogeneous data, and the multi-source heterogeneous data is used to obtain the behavior pattern and predicted movement trajectory of the drone through behavior pattern mining; according to the behavior pattern and predicted movement trajectory of the drone, analysis is performed through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensing resource allocation scheme, and based on the optimized sensing resource allocation scheme, the working state adjustment and tracking strategy of the intelligent vision sensing node are processed to obtain an emergency correction program; according to the emergency correction program, edge computing capabilities are integrated into the intelligent vision sensing node to process local data and make decisions to generate drone detection and tracking instructions; the technical solution provided by the present invention ensures the consistency of different sensor data, improves the quality and accuracy of data fusion, enables the system to more accurately identify and track targets in complex environments, enhances the perception ability of the system, can also predict the future movement path of the drone, adjust the monitoring strategy in advance, improves the prediction ability and response speed of the system, ensures the reasonable allocation of resources, reduces redundant configurations, improves the overall efficiency and flexibility of the system, improves the real-time processing ability of the system, and also enhances the speed and accuracy of local decision-making, ensuring high-efficiency detection and tracking performance even in complex environments;

[0051] Further, using the initial parameter configuration set, apply the multi-objective optimization algorithm to evaluate the collaboration efficiency, energy consumption, and real-time response speed among intelligent vision sensing nodes, and simulate the interaction effects in different scenarios through simulation to generate a preliminary adjusted parameter configuration plan. This step ensures that the working parameters of each node are scientifically evaluated to achieve the best performance; based on the preliminary adjusted parameter configuration plan, introduce a scenario adaptability analysis module to evaluate the applicability of each intelligent vision sensing node in different task environments, and form a parameter configuration plan adapted to the scenario. This approach takes into account the diversity of actual application scenarios, enabling the system to maintain good performance under various conditions; according to the parameter configuration plan adapted to the scenario, optimize the working parameters of each intelligent vision sensing node through a dynamic iterative adjustment mechanism to obtain the best parameter configuration plan. The dynamic adjustment mechanism ensures that the system can continuously optimize its performance, adapt to changing environmental conditions, and improve the stability and reliability of the system; using the best parameter configuration plan, combined with multi-source information fusion technology, combine lidar and vision sensing data to adjust the working parameters of the intelligent vision sensing nodes to obtain a candidate parameter configuration plan. By fusing data from different types of sensors, the perception ability and decision-making accuracy of the system are further improved, ensuring the successful completion of the UAV detection and tracking tasks.

[0052] These aspects or other aspects of the present invention will be more clearly understood in the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 It is a flowchart of a UAV detection and tracking method based on the combination of lidar and vision provided by an embodiment of the present invention;

[0055] Figure 2 It is a schematic structural diagram of a UAV detection and tracking system based on the combination of lidar and vision provided by an embodiment of the present invention;

[0056] Figure 3 It is a schematic structural diagram of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] In order to enable those skilled in the art to better understand the solution of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention.

[0058] In some of the processes described in the specification, claims and above-mentioned drawings of the present invention, a plurality of operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. The operation numbers such as 101, 102, etc. are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit that "first" and "second" are different types.

[0059] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0060] Figure 1 The following is a flowchart of a method for detecting and tracking an unmanned aerial vehicle based on the combination of lidar and vision provided for an embodiment of the present invention. As Figure 1 shown, the method includes:

[0061] Step 101: Use a distributed computing framework to perform real-time communication processing on the high-resolution lidar and intelligent vision sensing nodes in the deployed hybrid sensor array to obtain timestamp information.

[0062] In this step, the distributed computing framework is a software architecture that allows computers distributed in different physical locations to work together to complete common tasks. In this step, it is used to coordinate the communication between multiple high-resolution lidar and intelligent vision sensing nodes deployed in a vast area. These sensing nodes can collect environmental data in real time and transmit the data to a central or edge server through a network;

[0063] First, each sensing node is connected to the distributed computing framework through a high-speed wireless network to ensure low latency and high reliability of data transmission. When each sensing node captures data (such as the angular information point cloud data stream of the lidar and the multi-view image frames of the intelligent vision sensing node), it will attach an accurate timestamp information, which is generated by the Global Positioning System (GPS) or other synchronization mechanisms to mark the exact moment of data acquisition. This step is the basis for subsequent spatio-temporal calibration, ensuring that data from different sources can be compared and fused on the same time basis;

[0064] For example, in a drone monitoring project at the city level, multiple high-resolution lidars and intelligent vision sensing nodes installed on top of buildings form a sensing network covering the entire city. Each node is equipped with a GPS receiver to ensure that all collected data has a unified timestamp. With such a setup, even in a busy urban environment, the temporal consistency of all sensing data can be guaranteed, providing a solid foundation for subsequent drone detection and tracking.

[0065] Step 102: According to the timestamp information, perform spatio-temporal calibration on the angle information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent vision sensing node to generate a multi-source data set.

[0066] In this step, spatio-temporal calibration refers to the process of adjusting data from different sensors to make it consistent in both the time and space dimensions. In this step, the angle information point cloud data stream and the multi-view image frames are calibrated to the same time point and spatial coordinate system, thus forming a multi-source data set, providing a basis for subsequent data fusion and analysis.

[0067] Based on the timestamp information obtained in Step 101, perform spatio-temporal calibration on the angle information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by the intelligent vision sensing nodes. Specifically, each pair of point cloud data and image frames with the same timestamp will be mapped to the same three-dimensional coordinate system to eliminate the deviation caused by the difference in sensor positions. This process may involve the application of geometric transformation algorithms to ensure that data from different types of sensors can be correctly aligned in space.

[0068] For example, continuing with the above-mentioned drone monitoring project at the city level, after spatio-temporal calibration, the originally independent lidar point cloud data and visual image frames can now be displayed in a unified spatial coordinate system. For example, in an emergency response drill, it can accurately show the position of the drone relative to ground buildings and other objects, helping the command center make decisions quickly.

[0069] Step 103: Based on the multi-source data set, use the deep reinforcement learning algorithm for data fusion to obtain multi-source heterogeneous data. Utilize the multi-source heterogeneous data to obtain the drone behavior pattern and predicted movement trajectory through behavior pattern mining.

[0070] In this step, the deep reinforcement learning algorithm is a machine learning method that combines the powerful representation ability of deep neural networks and the goal - orientation of reinforcement learning. Here, it is used to extract valuable information from multi - source datasets, namely, the behavior patterns of the UAV and its predicted motion trajectories. The so - called multi - source heterogeneous data refers to data from different types of sensors, which have different formats and characteristics;

[0071] Based on the multi - source dataset after spatio - temporal calibration, the deep reinforcement learning algorithm is applied for data fusion. This algorithm not only integrates the three - dimensional spatial information provided by lidar and the two - dimensional image information given by visual sensing, but also takes into account environmental factors such as light changes and weather conditions. Through learning a large amount of historical data, the algorithm can identify various behavior patterns of the UAV, including flight paths, speed changes, and possible behavior intentions. In addition, the algorithm can also predict the motion trajectory of the UAV in the next period of time, providing guidance for the subsequent tracking strategy;

[0072] For example, in the practical application of an urban - level UAV monitoring project, the deep reinforcement learning algorithm has successfully learned the behavior rules of the UAV from a large amount of historical data. For example, during daily patrol tasks, it can anticipate the flight route of the UAV in advance and adjust the parameters of other monitoring devices accordingly to ensure continuous and effective tracking. This ability greatly improves the initiative and predictability, reducing the probability of unexpected situations.

[0073] Step 104: According to the UAV behavior patterns and predicted motion trajectories, analyze through the intelligent task allocation protocol under the self - organizing network to generate an optimized sensing resource allocation plan. Based on the optimized sensing resource allocation plan, process the working state adjustment and tracking strategy of the intelligent vision sensing nodes to obtain an emergency correction program.

[0074] In this step, the self - organizing network refers to a network structure without a central control, where nodes can autonomously adjust their behaviors according to their own states and environmental changes. The intelligent task allocation protocol is a set of rules used to guide how nodes cooperate efficiently to complete specific tasks. In this step, according to the UAV behavior patterns and predicted motion trajectories, optimize the allocation of sensing resources through the intelligent task allocation protocol under the self - organizing network;

[0075] Based on the drone behavior patterns and predicted movement trajectories obtained in step 103, optimize the allocation of sensing resources through an intelligent task allocation protocol under a self-organizing network. Specifically, the protocol will evaluate the requirements of the current task, such as the area or time period that needs to be monitored intensively, and adjust the working parameters of each intelligent vision sensing node accordingly, such as the scanning frequency, image resolution, etc. At the same time, the protocol will also formulate an emergency correction procedure to quickly respond in case of abnormal situations, such as when the drone suddenly changes its flight path or a new interference source appears;

[0076] For example, continuing with the urban-level drone monitoring project as an example, when it is detected that a certain drone deviates from the predetermined flight route, the intelligent task allocation protocol under the self-organizing network is immediately activated to reconfigure the nearby sensing nodes to monitor the area in a more intensive manner. At the same time, an emergency correction procedure is also formulated to prepare to add additional monitoring devices or adjust the working status of existing devices when necessary to ensure that the continuous tracking of the drone is not interrupted.

[0077] Step 105: Integrate edge computing capabilities into the intelligent vision sensing nodes according to the emergency correction procedure, process local data, and make decision-making to generate drone detection and tracking instructions.

[0078] In this step, edge computing refers to the technology of performing data processing at the edge of the network (i.e., close to the data source), which can reduce data transmission latency, improve response speed, and relieve the burden on the central server. In this step, edge computing capabilities are integrated into the intelligent vision sensing nodes, enabling them to process local data locally and make decisions immediately;

[0079] According to the emergency correction procedure, integrate edge computing capabilities into the intelligent vision sensing nodes so that each node can process local data locally and make decisions quickly according to the actual situation. For example, if a certain sensing node detects the approach of a drone, it can immediately adjust its working parameters, such as increasing the scanning frequency or improving the image resolution, and at the same time send signals to adjacent nodes to notify them to get ready. Finally, each node generates drone detection and tracking instructions based on the latest situation to ensure that the response is both timely and accurate;

[0080] For example, in the actual operation of the urban-level drone monitoring project, once the emergency correction procedure is triggered, the intelligent vision sensing nodes at key positions can immediately take actions. For instance, in an emergency situation, when an unauthorized drone is detected entering a restricted airspace, all nearby sensing nodes quickly adjust their working states, increase the monitoring density of the area, and generate detection and tracking instructions in a timely manner to ensure that safety measures can be implemented quickly.

[0081] Through the implementation of steps 101 to 105, the drone has achieved a series of efficient operations from data collection, spatio-temporal calibration, data fusion to sensing resource allocation and local decision-making. This not only significantly improves the accuracy and reliability of drone detection and tracking, but also enhances adaptability and response speed. In particular, by introducing edge computing capabilities, flexibility and robustness are further improved, ensuring stable and efficient performance even in complex and changing environments. The overall solution provides strong technical support for urban-level drone monitoring, ensuring the effectiveness of public safety and airspace management.

[0082] In actual operation, to address the issues of resource allocation efficiency and response speed, according to the above embodiments, in step 104, based on the drone behavior pattern and predicted movement trajectory, analysis is carried out through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensing resource allocation plan. Based on the optimized sensing resource allocation plan, the working state adjustment and tracking strategy of the intelligent vision sensing nodes are processed to obtain an emergency correction program, including:

[0083] Dynamically evaluate the task priority, coverage range of the intelligent vision sensing nodes and their complementarity with lidar data using the drone behavior pattern and predicted movement trajectory to generate an initial sensing resource allocation model; based on this model, apply a multi-objective optimization algorithm to adjust the working parameters and evaluate the cooperation efficiency and energy consumption to obtain an optimized sensing resource allocation plan; test the performance under different interference conditions through simulation technology, identify potential problems and formulate preventive measures; convert the preventive measures into operation instructions, and in combination with the real-time monitoring of the dynamic changes of the drone, adjust the working state and tracking strategy of the intelligent vision sensing nodes in real time, and finally generate an emergency correction program.

[0084] In this embodiment, the drone behavior pattern refers to the typical flight path, speed changes and other characteristics of the drone obtained through data analysis. The predicted movement trajectory is an estimate of the future movement path of the drone based on historical data and real-time information. The task priority refers to the importance and urgency of each sensing node determined according to the drone behavior pattern. The coverage range represents the spatial area that each intelligent vision sensing node can effectively monitor. Complementarity describes the mutual complementary relationship between data from different sensors (such as lidar and vision sensing) for improving the overall detection accuracy.

[0085] In the embodiments of this application, first, use the drone behavior pattern and predicted movement trajectory to dynamically evaluate the task priority, coverage range of the intelligent vision sensing nodes and their complementarity with lidar data to obtain an initial sensing resource allocation model. This process ensures that the resource allocation not only meets the current environmental requirements but also has predictability for future changes.

[0086] Then, based on the initial sensing resource configuration model, a multi-objective optimization algorithm is applied to adjust the working parameters (such as scanning frequency, image resolution, etc.) of the intelligent vision sensing nodes, and the cooperation efficiency and energy consumption among the intelligent vision sensing nodes are evaluated to obtain an optimized sensing resource configuration plan.

[0087] Next, using the optimized sensing resource configuration plan, tests are carried out through simulation technology under different interference conditions (such as electromagnetic interference, optical camouflage, etc.) to identify potential problems and formulate corresponding preventive measures. This stage ensures that the system can operate stably even in a harsh environment;

[0088] Finally, the preventive measures are converted into specific operation instructions, and combined with the dynamically changing UAVs monitored in real time, the working status and tracking strategies of the intelligent vision sensing nodes are adjusted in real time to generate an emergency correction program. This process enables the system to quickly respond to emergencies and maintain continuous and effective tracking performance.

[0089] For example, in a city-level UAV monitoring project, a hybrid sensor array composed of multiple high-resolution lidars and intelligent vision sensing nodes is deployed. When the system detects that a UAV is approaching a restricted airspace, it first predicts the future movement trajectory of the UAV based on its historical flight path and real-time position, and adjusts the task priorities of nearby sensing nodes accordingly. For example, the nodes close to the predicted path of the UAV are given higher priorities and their coverage ranges are expanded to ensure denser monitoring.

[0090] Subsequently, a multi-objective optimization algorithm is applied to adjust the working parameters of these high-priority nodes, such as increasing the scanning frequency or improving the image resolution, while considering the cooperation efficiency and energy consumption among the nodes to generate an optimized resource configuration plan. To verify the effectiveness of the plan, the system simulates various possible interference conditions, such as strong electromagnetic interference or complex light changes, through simulation technology, identifies potential risk points, and formulates corresponding preventive measures;

[0091] Once potential risks are detected, such as the UAV suddenly changing direction or a new interference source emerging, the system immediately converts the preventive measures into specific operation instructions and sends them to the relevant sensing nodes. These nodes will adjust their own working status according to the latest instructions, such as adjusting the viewing angle or activating the backup power supply, to ensure that the continuous tracking of the UAV will not be interrupted. Throughout the process, the system continuously receives real-time feedback from each node and dynamically adjusts the resource configuration to ensure the best detection and tracking effects.

[0092] To further improve the detection and tracking performance of intelligent vision sensing nodes in complex environments, as another embodiment, in step 104, based on the initial sensing resource configuration model, a multi-objective optimization algorithm is applied to adjust the working parameters of the intelligent vision sensing nodes, and the cooperation efficiency and energy consumption of the intelligent vision sensing nodes are evaluated to obtain an optimized sensing resource configuration scheme, including:

[0093] The initial sensing resource configuration model is used to initialize the working parameters (such as scanning frequency, image resolution, viewing angle, and data transmission priority) of the intelligent vision sensing nodes in combination with the UAV behavior pattern and predicted movement trajectory, obtaining an initial parameter configuration set; based on this configuration set, a multi-objective optimization algorithm is applied to evaluate the cooperation efficiency, energy consumption, and real-time response speed, and the working parameters are dynamically adjusted through simulation to simulate the interaction effects in different scenarios, forming a candidate parameter configuration scheme; a dynamic risk assessment system is introduced to screen out a highly reliable configuration scheme, and through an environmental adaptability assessment mechanism, tests are carried out for complex electromagnetic and optical interference conditions, and finally an optimized sensing resource configuration scheme is obtained.

[0094] In this embodiment, the initial sensing resource configuration model refers to a preliminary configuration scheme generated according to the UAV behavior pattern and predicted movement trajectory, which is used to guide the setting of the working parameters of the intelligent vision sensing nodes. The working parameters include scanning frequency, image resolution, viewing angle, and data transmission priority, etc., which directly affect the data acquisition efficiency and quality of the nodes. The multi-objective optimization algorithm is an algorithm that can consider multiple optimization objectives (such as cooperation efficiency, energy consumption, and real-time response speed) at the same time, and is used to find the best parameter combination. Simulation is to evaluate the interaction effects between each intelligent vision sensing node by simulating the system behavior in different scenarios through a computer. The candidate parameter configuration scheme is a set of multiple possible parameter combinations generated during the optimization process, and the optimal solution is selected from them. The dynamic risk assessment system is used to quantitatively analyze the risks of each candidate configuration scheme and evaluate the impacts of external environmental changes and internal system states. The highly reliable configuration scheme is a scheme screened out from the candidate schemes that can maintain stable performance under various conditions. The environmental adaptability assessment mechanism is used to test the performance of the highly reliable configuration scheme under complex electromagnetic environments and optical interference conditions;

[0095] In the embodiment of the present application, first, the initial sensing resource configuration model is used to initialize the working parameters (such as scanning frequency, image resolution, viewing angle, and data transmission priority) of the intelligent vision sensing nodes in combination with the UAV behavior pattern and predicted movement trajectory, obtaining an initial parameter configuration set. This process ensures that each node can be appropriately configured according to the current task requirements and expected environmental changes;

[0096] Then, based on the initial parameter configuration set, apply a multi-objective optimization algorithm to evaluate the collaboration efficiency, energy consumption, and real-time response speed among intelligent vision sensing nodes, and through simulation to simulate the interaction effects in different scenarios, dynamically iteratively adjust the working parameters of each intelligent vision sensing node to form a candidate parameter configuration scheme. This stage aims to find a set of parameter combinations that can perform well under various conditions;

[0097] Next, based on the candidate parameter configuration scheme, introduce a dynamic risk assessment system to conduct a risk quantification analysis on each candidate configuration scheme, calculate the impacts of external environmental changes and internal system states, and screen out high-reliability configuration schemes. This step ensures that the selected scheme not only performs well under ideal conditions but also can maintain the stability and reliability of the system in unforeseen situations;

[0098] Finally, utilize the high-reliability configuration scheme to implement an environmental adaptability evaluation mechanism, conduct simulation tests for complex electromagnetic environments and optical interference conditions, and obtain an optimized sensing resource configuration scheme. Through this series of strict tests and evaluations, the finally determined resource configuration scheme can provide stable and efficient detection and tracking services in various complex environments.

[0099] For example, in a city-level drone monitoring project, a hybrid sensor array composed of multiple high-resolution lidars and intelligent vision sensing nodes is deployed. When the system detects a drone approaching a restricted airspace, it first predicts the future movement trajectory of the drone based on its historical flight path and real-time position, and adjusts the task priorities of nearby sensing nodes accordingly;

[0100] The system uses the initial sensing resource configuration model to set the initial working parameters for each intelligent vision sensing node. For example, the scanning frequency is set to 30 times per second, the image resolution is 1080p, the viewing angle covers a 90-degree range, and the data transmission priority is set to the highest. These parameters ensure that the nodes can quickly capture relevant information about the drone in the initial stage;

[0101] Subsequently, the system applies a multi-objective optimization algorithm, considering factors such as the collaboration efficiency, energy consumption, and real-time response speed among nodes, and dynamically adjusts the working parameters of each node through simulation to simulate the interaction effects in different scenarios. For example, in a densely built-up area, some nodes may be required to increase the scanning frequency to cope with the complex background environment, while other nodes may reduce the image resolution to save energy;

[0102] To ensure the reliability of the scheme, the system introduces a dynamic risk assessment system to conduct a risk quantification analysis on each candidate configuration scheme. Considering factors such as possible electromagnetic interference or weather changes in the urban environment, the system selects those configuration schemes that can still maintain high performance under these conditions;

[0103] Finally, through the environmental adaptability evaluation mechanism, the system conducted simulation tests for complex electromagnetic environments and optical interference conditions. The test results showed that the selected high-reliability configuration plan could maintain stable performance in various complex environments. The system applied these optimized parameters to actual operations, ensuring continuous and effective tracking of the unmanned aerial vehicle and enabling it to quickly respond and adjust the tracking strategy in case of emergencies, thus guaranteeing public safety.

[0104] To further improve the collaboration efficiency, energy consumption optimization, and real-time response speed of intelligent vision sensing nodes in complex environments, based on the initial parameter configuration set described in the previous embodiment, a multi-objective optimization algorithm was applied to evaluate the collaboration efficiency, energy consumption, and real-time response speed among intelligent vision sensing nodes. By simulating the interactive effects in different scenarios, the working parameters of each intelligent vision sensing node were dynamically iteratively adjusted to form a candidate parameter configuration plan, including:

[0105] Using the initial parameter configuration set, a multi-objective optimization algorithm was applied to evaluate the collaboration efficiency, energy consumption, and real-time response speed among intelligent vision sensing nodes. By simulating the interactive effects in different scenarios, a preliminary adjusted parameter configuration plan was generated. Based on the preliminary adjusted parameter configuration plan, a scene adaptability analysis module was introduced to evaluate the applicability of each intelligent vision sensing node in different task environments, forming a scene-adapted parameter configuration plan. According to the scene-adapted parameter configuration plan, through a dynamic iterative adjustment mechanism, the working parameters of each intelligent vision sensing node were optimized to obtain the best parameter configuration plan. Using the best parameter configuration plan and combining multi-source information fusion technology, by integrating lidar and vision sensing data, the working parameters of the intelligent vision sensing node were adjusted to obtain a candidate parameter configuration plan.

[0106] In this embodiment, the multi-objective optimization algorithm is an algorithm that can simultaneously consider multiple optimization objectives (such as collaborative efficiency, energy consumption, and real-time response speed) to find the best parameter combination. Simulation is to simulate the system behavior in different scenarios by computer and evaluate the interaction between each intelligent visual sensor node. The parameter configuration scheme after preliminary adjustment is the parameter combination generated during the initial optimization process, which provides a basis for subsequent more refined adjustments. The scene adaptability analysis module is a specially designed tool or process for evaluating the applicability and performance of each intelligent visual sensor node in different task environments. The dynamic iterative adjustment mechanism refers to the process of gradually optimizing the working parameters through continuous feedback and learning in a constantly changing environment. The optimal parameter configuration scheme is the optimal parameter combination obtained from a series of adjustments, which ensures that the system can perform well under various conditions. Multi-source information fusion technology combines data from different types of sensors (such as lidar and visual sensing) to improve data quality and decision accuracy.

[0107] In the embodiment of the present application, firstly, the initial parameter configuration set is used to apply a multi-objective optimization algorithm to evaluate the collaboration efficiency, energy consumption, and real-time response speed between intelligent visual sensor nodes, and the interaction effects under different scenarios are simulated to generate a preliminary adjusted parameter configuration scheme. This step aims to explore a set of parameter combinations that can perform well under a variety of conditions as a basis for further optimization;

[0108] Next, based on the parameter configuration scheme after the preliminary adjustment, the scene adaptability analysis module is introduced to evaluate the applicability of each intelligent visual sensor node in different task environments and form a scene-adaptive parameter configuration scheme. This stage ensures that each node can play its maximum performance in its specific application scenario. For example, a higher scanning frequency may be required in a high-density building area, while the image resolution can be reduced in an open area to save energy;

[0109] Then, according to the parameter configuration scheme adapted to the scenario, the working parameters of each intelligent visual sensor node are optimized through a dynamic iterative adjustment mechanism to obtain the best parameter configuration scheme. This process involves continuous monitoring and feedback, allowing the system to flexibly adjust according to actual conditions, ensuring that resource configuration is always in the optimal state, and improving the overall performance and stability of the system;

[0110] Finally, the optimal parameter configuration scheme is used, combined with multi-source information fusion technology, and combined with lidar and visual sensor data to make final adjustments to the working parameters of the intelligent visual sensor node to obtain a candidate parameter configuration scheme. By integrating data from different types of sensors, the system can more accurately capture and understand environmental changes, thereby making more informed decisions and enhancing the accuracy and reliability of detection and tracking;

[0111] For example, in a drone monitoring project at the city level, a hybrid sensor array composed of multiple high-resolution lidars and intelligent vision sensing nodes is deployed. When the system detects that a drone is approaching a restricted airspace, it first predicts the future movement trajectory of the drone based on its historical flight path and real-time position, and adjusts the task priorities of nearby sensing nodes accordingly;

[0112] The system uses the initial parameter configuration set to set the initial working parameters for each intelligent vision sensing node. For example, the scanning frequency is set to 30 times per second, the image resolution is 1080p, the viewing angle covers a range of 90 degrees, and the data transmission priority is set to the highest. These parameters ensure that the nodes can quickly capture relevant information about the drone in the initial stage;

[0113] Subsequently, the system applies a multi-objective optimization algorithm, considering factors such as the cooperation efficiency, energy consumption, and real-time response speed among nodes, and generates a preliminary adjusted parameter configuration plan by simulating the interaction effects in different scenarios. For example, in a densely built-up area, some nodes may be required to increase the scanning frequency to cope with the complex background environment, while other nodes may reduce the image resolution to save energy;

[0114] To ensure that the plan is suitable for the actual task environment, the system introduces a scenario adaptability analysis module to evaluate the applicability of each intelligent vision sensing node under different conditions. For example, in a low-light environment, the system will adjust the image resolution of some nodes to ensure sufficient brightness capture ability; while in an area with strong electromagnetic interference, the data transmission priority will be optimized to ensure communication quality;

[0115] Next, the system continuously optimizes the working parameters of each intelligent vision sensing node through a dynamic iterative adjustment mechanism. For example, if a node detects that the drone suddenly changes its flight path, it will immediately adjust its own working parameters, such as increasing the scanning frequency or improving the image resolution, and at the same time send a signal to adjacent nodes to notify them to get ready;

[0116] Finally, the system uses the best parameter configuration plan, combines multi-source information fusion technology, combines lidar and vision sensing data, and makes a final adjustment to the working parameters of the intelligent vision sensing nodes to obtain a candidate parameter configuration plan. By integrating the accurate distance information provided by the lidar and the rich image details captured by the vision sensing, the system can more comprehensively understand and respond to environmental changes, improving the accuracy and reliability of drone detection and tracking.

[0117] To solve the problem of data synchronization and spatial alignment between high-resolution lidar and intelligent vision sensing nodes, as another embodiment, as described in step 102, according to the timestamp information, perform spatio-temporal calibration on the angular information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent vision sensing node to generate a multi-source dataset, including:

[0118] Using the timestamp information in combination with the environmental perception algorithm, perform precise time synchronization processing on the angular information point cloud data stream of the high-resolution lidar and the multi-view image frames captured by the intelligent vision sensing nodes to obtain time-synchronized data pairs, and record environmental parameters. Based on the time-synchronized data pairs, apply the adaptive spatial coordinate transformation algorithm, and dynamically adjust the transformation parameters in combination with the recorded environmental parameters to map the three-dimensional point cloud data of the lidar to the two-dimensional image plane of each intelligent vision sensing node to generate a spatially aligned dataset. According to the spatially aligned dataset, introduce a geometric consistency verification mechanism, detect and correct geometric mismatch problems through a deep learning model, and at the same time use semantic segmentation technology to identify and eliminate data interference in non-target areas to form an initial multi-source dataset. Using the initial multi-source dataset, perform spatio-temporal correlation optimization processing, fuse continuous data frames in the time dimension, and use a time series prediction model to predict data distribution to generate a target multi-source dataset.

[0119] In this embodiment, the environmental perception algorithm refers to the technology of identifying and understanding the surrounding environment by analyzing the data collected by sensors. Time synchronization processing is a process of ensuring that data from different sensors is consistent in the time dimension for subsequent processing. The time-synchronized data pair is paired data composed of lidar point cloud data and visual image frames that have undergone time synchronization processing. Environmental parameters include factors such as temperature, humidity, and light intensity that may affect sensor performance, and these parameters are used to adjust the parameters in the data conversion process. The adaptive spatial coordinate transformation algorithm is an algorithm that can dynamically adjust the transformation parameters according to the actual situation and is used to map three-dimensional point cloud data to a two-dimensional image plane. The geometric consistency verification mechanism is used to detect and correct geometric mismatch problems between point cloud data and images. Semantic segmentation technology is an image processing method that can identify different objects in an image and eliminate data interference in non-target areas. The initial multi-source dataset is a data set formed after geometric consistency verification and semantic segmentation. Spatio-temporal correlation optimization processing is a process of optimizing by combining continuous data frames in the time and space dimensions. The time series prediction model is a model that predicts future data distribution based on historical data.

[0120] In the embodiments of the present application, first, using the timestamp information in combination with the environment perception algorithm, precise time synchronization processing is performed on the angular information point cloud data stream of the high-resolution lidar and the multi-view image frames captured by the intelligent vision sensing nodes to obtain data pairs after time synchronization, and the environmental parameters are recorded. This step ensures that data from different sensors can be compared and fused on the same time basis, while taking into account the impact of environmental changes on the sensors; Next, based on the data pairs after time synchronization, an adaptive spatial coordinate conversion algorithm is applied, and the conversion parameters are dynamically adjusted in combination with the recorded environmental parameters to map the three-dimensional point cloud data of the lidar to the two-dimensional image planes of the respective intelligent vision sensing nodes, generating a spatially aligned data set. This process solves the problem of inconsistent spatial coordinates caused by differences in sensor positions, enabling different types of data to be represented in a unified coordinate system; Then, according to the spatially aligned data set, a geometric consistency verification mechanism is introduced, the geometric mismatch problem is detected and corrected through a deep learning model, and at the same time, semantic segmentation technology is used to identify and eliminate data interference in non-target areas, forming an initial multi-source data set. This step improves the quality and accuracy of the data, reduces the interference of noise and irrelevant information, and provides a more reliable basis for subsequent analysis; Finally, using the initial multi-source data set, spatio-temporal correlation optimization processing is implemented, continuous data frames in the time dimension are fused, and a time series prediction model is used to predict the data distribution, generating a target multi-source data set. This step not only enhances the temporal coherence of the data but also estimates the future data change trend, further improving the prediction ability and response speed of the system.

[0121] For example, in a drone monitoring project at the city level, a hybrid sensor array composed of multiple high-resolution lidars and intelligent vision sensing nodes is deployed. When the system starts running, each sensor node will collect a large amount of environmental data, such as the point cloud data stream of angle information provided by the lidar and the multi-view image frames captured by the intelligent vision sensing nodes; the system first uses the timestamp information combined with the environmental perception algorithm to perform precise time synchronization processing on these data to ensure that all data are on the same time basis. For example, in an emergency response drill, even if the sensor nodes are located in different geographical locations, the system can accurately synchronize the data they collect to the same moment and record the environmental parameters at that time, such as temperature, humidity, and light conditions; next, based on the time-synchronized data pairs, the system applies an adaptive spatial coordinate transformation algorithm to map the three-dimensional point cloud data of the lidar onto the two-dimensional image plane of the intelligent vision sensing node. In this process, the system will dynamically adjust the transformation parameters according to the recorded environmental parameters to compensate for the errors caused by weather or light changes, thereby generating a spatially aligned data set; to improve the data quality and accuracy, the system introduces a geometric consistency verification mechanism to detect and correct any geometric mismatch problems through a deep learning model. For example, if there is a deviation between the image frame of a sensor node and the lidar point cloud data, the system will automatically adjust to make the two more matching. In addition, the system also uses semantic segmentation technology to identify and exclude data interference from non-target areas, such as background buildings or other irrelevant objects, thereby forming an initial multi-source data set; finally, the system uses the initial multi-source data set to implement spatio-temporal correlation optimization processing, fuse consecutive data frames in the time dimension, and use a time series prediction model to predict the data distribution in the future for a period of time. For example, before the drone enters the restricted airspace, the system can predict its future flight path based on its current behavior pattern and adjust the working state of the nearby sensing nodes in advance to ensure continuous and effective tracking.

[0122] To solve the problems of insufficient data fusion, inaccurate behavior pattern recognition, and inaccurate predicted motion trajectory in the prior art, as another embodiment, according to what is described in step 103, based on the multi-source data set, a deep reinforcement learning algorithm is used for data fusion to obtain multi-source heterogeneous data, and using the multi-source heterogeneous data, through behavior pattern mining, the drone behavior pattern and predicted motion trajectory are obtained, including:

[0123] Using the multi-source dataset, the angle information point cloud data stream of the lidar and the multi-view image frames captured by the intelligent vision sensing nodes are jointly analyzed and processed by using a deep reinforcement learning algorithm to obtain multi-source heterogeneous data. According to the multi-source heterogeneous data, a multi-modal perception model is constructed to integrate the data characteristics from different sensors, and a reinforced multi-modal perception model is generated. Based on the reinforced multi-modal perception model, a behavior pattern mining algorithm is applied to extract the behavior characteristics of the UAV from the multi-source heterogeneous data to form an initial behavior pattern library. The initial behavior pattern library includes the position, speed, direction, and behavior intention of the UAV. Using the initial behavior pattern library, combined with historical data and real-time updated perception information, the motion trajectory of the UAV is predicted through a time series prediction model to generate the UAV behavior pattern and the predicted motion trajectory. Among them, the real-time updated perception information includes: multi-view image frames, angle information point cloud data stream, environmental changes, UAV state updates, external interference, and mission instruction changes.

[0124] In this embodiment, the deep reinforcement learning algorithm is a machine learning method that combines the powerful representation ability of a deep neural network and the goal-directedness of reinforcement learning. Multi-source heterogeneous data refers to data from different types of sensors, and these data have different formats and characteristics, such as the angle information point cloud data stream of the lidar and the multi-view image frames captured by the intelligent vision sensing nodes. The multi-modal perception model refers to a model that can integrate the data characteristics from different sensors to more comprehensively describe the environment and target state. The behavior pattern mining algorithm is used to extract the behavior characteristics of the UAV from the multi-source heterogeneous data to form an initial behavior pattern library, which includes information such as the position, speed, direction, and behavior intention of the UAV. The time series prediction model is a model that predicts the future data distribution based on historical data and is used to estimate the future motion trajectory of the UAV.

[0125] In the embodiments of the present application, first, using the multi-source data set, the depth reinforcement learning algorithm is adopted to jointly analyze and process the angle information point cloud data stream of the lidar and the multi-view image frames captured by the intelligent vision sensing nodes, so as to obtain multi-source heterogeneous data. This step ensures that different types of data can be effectively integrated under a unified framework, improving the quality and accuracy of data fusion; Next, according to the multi-source heterogeneous data, a multi-modal perception model is constructed, integrating the data characteristics from different sensors to generate a reinforced multi-modal perception model. This process not only enhances the ability to understand the environment but also improves the adaptability to complex scenarios, enabling the system to better handle detection and tracking tasks in various situations; Then, based on the reinforced multi-modal perception model, the behavior pattern mining algorithm is applied to extract the behavior characteristics of the UAV from the multi-source heterogeneous data to form an initial behavior pattern library. This library contains information such as the position, speed, direction, and behavior intention of the UAV, providing a solid foundation for subsequent behavior pattern analysis and motion trajectory prediction; Finally, using the initial behavior pattern library, combined with historical data and real-time updated perception information, the motion trajectory of the UAV is predicted through a time series prediction model to generate the UAV behavior pattern and predicted motion trajectory. The real-time updated perception information includes multi-view image frames, angle information point cloud data streams, environmental changes, UAV state updates, as well as external interference and mission instruction changes, ensuring the timeliness and accuracy of the prediction.

[0126] For example, in a drone monitoring project at the city level, a hybrid sensor array composed of multiple high-resolution lidars and intelligent vision sensing nodes is deployed. When the system starts running, each sensor node continuously collects a large amount of environmental data, such as the point cloud data stream of angle information provided by the lidar and the multi-view image frames captured by the intelligent vision sensing nodes; the system first uses the multi-source data set and adopts a deep reinforcement learning algorithm to jointly analyze and process these data to obtain multi-source heterogeneous data. For example, in a daily patrol mission, the system combines the three-dimensional spatial information of the lidar with the two-dimensional image information of the vision sensing to form a richer and more detailed environmental description; next, based on the multi-source heterogeneous data, the system constructs a multi-modal perception model and further generates a reinforced multi-modal perception model. This model can more comprehensively understand the objects in the environment and their dynamic changes, improving the system's environmental perception ability. For example, it can distinguish between static buildings and moving vehicles and simultaneously track the different behaviors of multiple drones; then, based on the reinforced multi-modal perception model, the system applies a behavior pattern mining algorithm to extract the behavior characteristics of the drones from the multi-source heterogeneous data to form an initial behavior pattern library. This library records the position, speed, direction, and possible behavior intentions of the drones, providing a basis for subsequent analysis. For example, the system can identify whether a certain drone is on a regular patrol or performing a special task; finally, using the initial behavior pattern library, the system combines historical data and real-time updated perception information to predict the movement trajectory of the drones through a time series prediction model. The real-time updated information includes the latest multi-view image frames, the point cloud data stream of angle information, environmental changes (such as weather conditions), drone status updates (such as speed changes), external interferences (such as electromagnetic interference), and task instruction changes. These information help the system more accurately predict the future movement path of the drones, adjust the working parameters of other monitoring devices in advance, and ensure continuous and effective tracking.

[0127] To solve the problems of high data processing latency, slow decision response, and low resource utilization efficiency in the prior art, as another embodiment, as described in step 105, according to the emergency correction program, edge computing capabilities are integrated into the intelligent vision sensing nodes to process local data and make decisions, generating drone detection and tracking instructions, including:

[0128] Using the emergency correction procedure, deploy the edge computing module to the intelligent vision sensing nodes, enabling each intelligent vision sensing node to have local data processing capabilities, obtaining a distributed computing network. Based on the distributed computing network, perform real-time preprocessing on the local data collected by the intelligent vision sensing nodes, extract feature information related to UAV detection and tracking, and obtain optimized local feature data. According to the optimized local feature data, apply the intelligent decision-making algorithm to independently execute the fast decision-making process on each intelligent vision sensing node, determine the best action plan for the current environment, generate preliminary detection and tracking instructions, and based on the preliminary detection and tracking instructions, perform instruction fusion through the collaboration mechanism under the self-organizing network, adjust and optimize each intelligent vision sensing node, and obtain comprehensive UAV detection and tracking instructions.

[0129] In this embodiment, the emergency correction procedure refers to the process in which the system quickly adjusts its working state and takes corresponding measures when an abnormal situation or emergency is detected. The edge computing module is a technical unit that can perform data processing near the data source (i.e., the network edge), which reduces data transmission latency, improves the response speed, and alleviates the burden on the central server. The distributed computing network refers to a network structure composed of multiple nodes with local data processing capabilities, and each node can independently complete specific tasks. Local data refers to the data that only involves the current node and its neighboring nodes and is used for fast decision-making. Feature information is the key data extracted from the original data that can describe the characteristics of the target object. The optimized local feature data is more accurate and effective local data after preprocessing. The intelligent decision-making algorithm is a set of rules or models used to guide how nodes cooperate efficiently to complete specific tasks. The collaboration mechanism under the self-organizing network is a network architecture without central control that allows nodes to autonomously coordinate resource allocation and task execution.

[0130] In the embodiments of the present application, first, the edge computing module is deployed to the intelligent vision sensing nodes by using the emergency correction program, so that each intelligent vision sensing node has the local data processing ability, and a distributed computing network is obtained. This step ensures that each node can process data locally, reduces the dependence on the central server, and improves the response speed and flexibility of the system; Next, based on the distributed computing network, the local data collected by the intelligent vision sensing nodes is preprocessed in real time, and the feature information related to the detection and tracking of the unmanned aerial vehicle is extracted to obtain the optimized local feature data. Through this preprocessing, the amount of data transmitted to the central server can be significantly reduced, and the accuracy of subsequent analysis is improved at the same time; Then, according to the optimized local feature data, an intelligent decision-making algorithm is applied to independently execute a fast decision-making process on each intelligent vision sensing node to determine the best action plan for the current environment and generate preliminary detection and tracking instructions. This stage enables each node to quickly respond according to the latest local data, enhancing the real-time performance and adaptability of the system; Finally, based on the preliminary detection and tracking instructions, instruction fusion is performed through the cooperation mechanism under the self-organizing network, and each intelligent vision sensing node is adjusted and optimized to obtain comprehensive unmanned aerial vehicle detection and tracking instructions. This process ensures the collaborative work among all nodes, avoids the limitations that may be brought by the decision-making of a single node, and improves the performance and reliability of the overall system.

[0131] For example, in a drone monitoring project at the city level, when the system detects an unauthorized drone entering the restricted airspace, it immediately triggers the emergency correction procedure. At this time, the edge computing module is quickly deployed to the intelligent vision sensing nodes, endowing these nodes with the ability of local data processing, forming a distributed computing network; each intelligent vision sensing node begins to perform real-time preprocessing on the collected local data, extracting feature information related to drone detection and tracking, such as the position, speed and direction of the drone. Through this preprocessing, the node can identify which data is crucial for the current task and use these optimized local feature data for further analysis; next, each node applies an intelligent decision-making algorithm to independently execute a fast decision-making process to determine the best action plan for the current environment. For example, if a node detects that the drone is approaching rapidly, it can immediately adjust its scanning frequency or increase the image resolution to obtain a clearer target image. At the same time, the node will also send signals to adjacent nodes to notify them to get ready; finally, based on these preliminary detection and tracking instructions, the system performs instruction fusion through the collaboration mechanism under the self-organizing network, adjusting and optimizing the working parameters of each intelligent vision sensing node. For example, if multiple nodes all detect the same drone, the system will coordinate their working states to ensure the maximization of the coverage area and the minimization of redundancy. Finally, the system generates comprehensive drone detection and tracking instructions, ensuring continuous and effective tracking of the drone and being able to respond quickly even in case of emergencies, thus ensuring public safety.

[0132] Figure 2 The structural schematic diagram of a drone detection and tracking system based on the combination of lidar and vision provided by an embodiment of the present invention is as Figure 2 shown, and the system includes:

[0133] A processing module 21, configured to perform real-time communication processing on the high-resolution lidar and intelligent vision sensing nodes in the deployed hybrid sensor array by using a distributed computing framework to obtain timestamp information;

[0134] A calibration module 22, configured to perform spatio-temporal calibration on the angle information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent vision sensing node according to the timestamp information to generate a multi-source data set;

[0135] A fusion module 23, configured to perform data fusion by using a deep reinforcement learning algorithm based on the multi-source data set to obtain multi-source heterogeneous data, and use the multi-source heterogeneous data to obtain the drone behavior pattern and predicted motion trajectory through behavior pattern mining;

[0136] An analysis module 24, configured to analyze according to the UAV behavior pattern and predicted motion trajectory through an intelligent task allocation protocol under a self-organizing network, generate an optimized sensing resource allocation scheme, and process the working state adjustment and tracking strategy of the intelligent vision sensing nodes based on the optimized sensing resource allocation scheme to obtain an emergency correction program;

[0137] A formulation module 25, configured to integrate edge computing capabilities into the intelligent vision sensing nodes according to the emergency correction program, process local data and make decisions, and generate UAV detection and tracking instructions.

[0138] Figure 2 The described UAV detection and tracking system based on the combination of lidar and vision can execute Figure 1 The described UAV detection and tracking method based on the combination of lidar and vision in the illustrated embodiment, its implementation principle and technical effects will not be elaborated. For the UAV detection and tracking system based on the combination of lidar and vision in the above embodiment, the specific ways for each module and unit to execute operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0139] In a possible design, Figure 2 The UAV detection and tracking system based on the combination of lidar and vision in the illustrated embodiment can be implemented as a computing device, such as Figure 3 shown, the computing device may include a storage component 31 and a processing component 32;

[0140] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32.

[0141] The processing component 32 is used for: performing real-time communication processing on the high-resolution lidar and intelligent vision sensing nodes in the deployed hybrid sensor array by using a distributed computing framework to obtain timestamp information; performing spatio-temporal calibration on the angle information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent vision sensing node according to the timestamp information to generate a multi-source data set; performing data fusion by using a deep reinforcement learning algorithm based on the multi-source data set to obtain multi-source heterogeneous data, and using the multi-source heterogeneous data to obtain the UAV behavior pattern and predicted motion trajectory through behavior pattern mining; analyzing through an intelligent task allocation protocol under a self-organizing network according to the UAV behavior pattern and predicted motion trajectory to generate an optimized sensing resource allocation scheme, and processing the working state adjustment and tracking strategy of the intelligent vision sensing node based on the optimized sensing resource allocation scheme to obtain an emergency correction program; integrating edge computing capabilities into the intelligent vision sensing node according to the emergency correction program to process local data and make decisions to generate UAV detection and tracking instructions.

[0142] Among them, the processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components for executing the above method.

[0143] The storage component 31 is configured to store various types of data to support the operation of the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0144] Of course, the computing device may also necessarily include other components, such as an input / output interface, a display component, a communication component, etc.

[0145] The input / output interface provides an interface between the processing component and the peripheral interface module, and the above peripheral interface module may be an output device, an input device, etc.

[0146] The communication component is configured to facilitate communication between the computing device and other devices in a wired or wireless manner, etc.

[0147] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform. At this time, the computing device can refer to a cloud server. The above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from a cloud computing platform.

[0148] An embodiment of the present invention also provides a computer storage medium storing a computer program, which when executed by a computer can implement the above-mentioned Figure 1 a method for detecting and tracking an unmanned aerial vehicle based on the combination of lidar and vision shown in the embodiment.

[0149] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting and tracking an unmanned aerial vehicle based on the combination of lidar and vision, characterized in that, Including: Performing real-time communication processing on the high-resolution lidar and intelligent vision sensing nodes in the deployed hybrid sensor array using a distributed computing framework to obtain timestamp information; According to the timestamp information, performing spatio-temporal calibration on the angular information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent vision sensing node to generate a multi-source data set; Based on the multi-source data set, using a deep reinforcement learning algorithm for data fusion to obtain multi-source heterogeneous data, and using the multi-source heterogeneous data to obtain the UAV behavior pattern and predicted motion trajectory through behavior pattern mining; Using the UAV behavior pattern and predicted motion trajectory to dynamically evaluate the task priority, coverage range of the intelligent vision sensing node, and the complementarity between the intelligent vision sensing node and lidar data to obtain an initial sensing resource configuration model; Based on the initial sensing resource configuration model, applying a multi-objective optimization algorithm to adjust the working parameters of the intelligent vision sensing node, and evaluating the cooperation efficiency and energy consumption of the intelligent vision sensing node to obtain an optimized sensing resource configuration plan; Using the optimized sensing resource configuration plan, testing under different interference conditions through simulation technology to identify potential problems and formulating corresponding preventive measures; Transforming the preventive measures into specific operation instructions, and combining with the real-time monitored dynamic changes of the UAV to adjust the working state and tracking strategy of the intelligent vision sensing node in real time to generate an emergency correction program; According to the emergency correction program, integrating edge computing capabilities into the intelligent vision sensing node to process local data and make decisions to generate UAV detection and tracking instructions.

2. The method according to claim 1, wherein Based on the initial sensing resource configuration model, applying a multi-objective optimization algorithm to adjust the working parameters of the intelligent vision sensing node, and evaluating the cooperation efficiency and energy consumption of the intelligent vision sensing node to obtain an optimized sensing resource configuration plan, including: Using the initial sensing resource configuration model, combining the UAV behavior pattern and predicted motion trajectory, to initialize the configuration of the working parameters of the intelligent vision sensing node to obtain an initial parameter configuration set, where the working parameters include: scanning frequency, image resolution, and viewing angle and data transmission priority; According to the initial parameter configuration set, applying a multi-objective optimization algorithm to evaluate the cooperation efficiency, energy consumption, and real-time response speed between intelligent vision sensing nodes, and dynamically iteratively adjusting the working parameters of each intelligent vision sensing node by simulating the interaction effects in different scenarios to form a candidate parameter configuration plan; Based on the candidate parameter configuration plan, introducing a dynamic risk assessment system to perform risk quantification analysis on each candidate configuration plan, calculating the impacts of external environmental changes and internal system states, and screening out high-reliability configuration plans; Using the high-reliability configuration plan, implementing an environmental adaptability evaluation mechanism to conduct simulation tests for complex electromagnetic environments and optical interference conditions to obtain an optimized sensing resource configuration plan.

3. The method according to claim 2, characterized in that, According to the set of initial parameter configurations, apply a multi-objective optimization algorithm to evaluate the collaboration efficiency, energy consumption, and real-time response speed among intelligent vision sensing nodes. Through simulation to simulate the interaction effects in different scenarios, dynamically iteratively adjust the working parameters of each intelligent vision sensing node to form a candidate parameter configuration scheme, including: Using the set of initial parameter configurations, apply a multi-objective optimization algorithm to evaluate the collaboration efficiency, energy consumption, and real-time response speed among intelligent vision sensing nodes. Through simulation to simulate the interaction effects in different scenarios, generate a preliminary adjusted parameter configuration scheme; Based on the preliminary adjusted parameter configuration scheme, introduce a scenario adaptability analysis module to evaluate the applicability of each intelligent vision sensing node in different task environments, and form a scenario-adapted parameter configuration scheme; According to the scenario-adapted parameter configuration scheme, through a dynamic iterative adjustment mechanism, optimize the working parameters of each intelligent vision sensing node to obtain an optimal parameter configuration scheme; Using the optimal parameter configuration scheme, combined with multi-source information fusion technology, combine lidar and visual sensing data to adjust the working parameters of intelligent vision sensing nodes to obtain a candidate parameter configuration scheme.

4. The method according to claim 1, characterized in that According to the timestamp information, perform spatio-temporal calibration on the angle information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent vision sensing node to generate a multi-source data set, including: Using the timestamp information combined with an environmental perception algorithm, perform precise time synchronization processing on the angle information point cloud data stream of the high-resolution lidar and the multi-view image frames captured by the intelligent vision sensing node to obtain a pair of time-synchronized data, and record the environmental parameters; Based on the pair of time-synchronized data, apply an adaptive spatial coordinate conversion algorithm, dynamically adjust the conversion parameters in combination with the recorded environmental parameters, and map the three-dimensional point cloud data of the lidar to the two-dimensional image plane of each intelligent vision sensing node to generate a spatially aligned data set; According to the spatially aligned data set, introduce a geometric consistency verification mechanism, detect and correct geometric mismatch problems through a deep learning model, and at the same time use semantic segmentation technology to identify and eliminate data interference in non-target areas to form an initial multi-source data set; Using the initial multi-source data set, implement spatio-temporal correlation optimization processing, fuse consecutive data frames in the time dimension, and use a time series prediction model to predict data distribution to generate a target multi-source data set.

5. The method according to claim 1, wherein Based on the multi-source data set, use a deep reinforcement learning algorithm for data fusion to obtain multi-source heterogeneous data. Using the multi-source heterogeneous data, through behavior pattern mining, obtain the UAV behavior pattern and predicted motion trajectory, including: Using the multi-source data set, use a deep reinforcement learning algorithm to jointly analyze and process the angle information point cloud data stream of the lidar and the multi-view image frames captured by the intelligent vision sensing node to obtain multi-source heterogeneous data; According to the multi-source heterogeneous data, construct a multi-modal perception model, integrate the data characteristics from different sensors, and generate a strengthened multi-modal perception model; Based on the enhanced multi-modal perception model, apply the behavior pattern mining algorithm to extract the behavior characteristics of the UAV from multi-source heterogeneous data, forming an initial behavior pattern library, which includes the position, speed, direction, and behavior intention of the UAV. Utilize the initial behavior pattern library, combine historical data and real-time updated perception information, and predict the motion trajectory of the UAV through a time series prediction model to generate the UAV behavior pattern and the predicted motion trajectory. Among them, the real-time updated perception information includes: multi-view image frames, angular information point cloud data streams, environmental changes, UAV state updates, external interference, and mission instruction changes.

6. The method according to claim 1, characterized in that, According to the emergency correction procedure, integrate the edge computing capability into the intelligent vision sensing node to process local data and make decisions, generating UAV detection and tracking instructions, including: Utilize the emergency correction procedure to deploy the edge computing module to the intelligent vision sensing node, enabling each intelligent vision sensing node to have local data processing capabilities, resulting in a distributed computing network. Based on the distributed computing network, perform real-time preprocessing on the local data collected by the intelligent vision sensing node, extract the feature information related to UAV detection and tracking, and obtain the optimized local feature data. According to the optimized local feature data, apply the intelligent decision-making algorithm to independently execute a fast decision-making process on each intelligent vision sensing node, determine the best action plan for the current environment, and generate preliminary detection and tracking instructions. Based on the preliminary detection and tracking instructions, perform instruction fusion through the cooperation mechanism under the self-organizing network, adjust and optimize each intelligent vision sensing node, and obtain comprehensive UAV detection and tracking instructions.

7. An unmanned aerial vehicle detection and tracking system based on the combination of lidar and vision, characterized in that, Including: Utilize the distributed computing framework to perform real-time communication processing on the high-resolution lidar and intelligent vision sensing nodes in the deployed hybrid sensor array to obtain timestamp information. According to the timestamp information, perform spatio-temporal calibration on the angular information point cloud data stream generated by the high-resolution lidar and the multi-view image frames captured by each intelligent vision sensing node to generate a multi-source data set. Based on the multi-source data set, adopt the deep reinforcement learning algorithm for data fusion to obtain multi-source heterogeneous data. Utilize the multi-source heterogeneous data and through behavior pattern mining, obtain the UAV behavior pattern and the predicted motion trajectory. Dynamically evaluate the task priority, coverage range of intelligent vision sensing nodes, and the complementarity between intelligent vision sensing nodes and lidar data by using the behavior patterns and predicted movement trajectories of drones to obtain an initial sensing resource allocation model; based on the initial sensing resource allocation model, apply a multi-objective optimization algorithm to adjust the working parameters of intelligent vision sensing nodes, and evaluate the cooperation efficiency and energy consumption of intelligent vision sensing nodes to obtain an optimized sensing resource allocation plan; use the optimized sensing resource allocation plan to conduct tests under different interference conditions through simulation technology, identify potential problems, and formulate corresponding preventive measures; convert the preventive measures into specific operation instructions, and combine the dynamically monitored changes of drones to adjust the working state and tracking strategy of intelligent vision sensing nodes in real time to generate an emergency correction program; According to the emergency correction program, integrate edge computing capabilities into intelligent vision sensing nodes to process local data and make decisions, and generate drone detection and tracking instructions.

8. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a drone detection and tracking method based on the combination of lidar and vision according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that, There is a computer program stored, and when the computer program is executed by a computer, it implements a drone detection and tracking method based on the combination of lidar and vision according to any one of claims 1 to 6.

Images