Unmanned aerial vehicle detection tracking method and system based on combination of laser radar and vision

Abstract

The invention provides an unmanned aerial vehicle detection tracking method and system based on combination of a laser radar and vision, and the method comprises the steps: obtaining timestamp information through employing a distributed calculation framework, generating a multi-source data set according to the timestamp information, carrying out the data fusion based on the multi-source data set through employing a deep reinforcement learning algorithm, and obtaining multi-source heterogeneous data, the method comprises the steps of obtaining an unmanned aerial vehicle behavior mode and a predicted motion trail by using multi-source heterogeneous data, performing analysis through an intelligent task allocation protocol under a self-organizing network according to the unmanned aerial vehicle behavior mode and the predicted motion trail, generating an optimal sensing resource allocation scheme, and performing resource allocation on the basis of the optimal sensing resource allocation scheme. Processing the intelligent visual sensing node to obtain an emergency correction program, and according to the emergency correction program, integrating edge computing power to generate an unmanned aerial vehicle detection and tracking instruction; according to the technical scheme provided by the invention, the quality and accuracy of data fusion are improved, and the perception capability is enhanced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of unmanned aerial vehicles, and in particular to a method for detecting and tracking unmanned aerial vehicles based on the combination of laser radar and vision. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in military reconnaissance, logistics distribution, agricultural monitoring, environmental monitoring and other fields. These application scenarios have put forward strict requirements for drone detection and tracking: high-precision positioning capabilities, real-time response speed, and stable tracking performance in complex and changing environments. Especially in dynamic environments, how to accurately identify and continuously track multiple drones has become a technical problem that needs to be solved urgently;

[0003] At present, drone detection and tracking mainly rely on a single sensor or a limited combination of sensors. For example, the use of radar systems can provide long-range target detection, but its resolution is low and it is difficult to distinguish small or slow-moving targets; while visual sensing can provide high-resolution images, it performs poorly in complex lighting conditions or occlusion. In addition, some studies have attempted to combine LiDAR with visual sensing to improve detection accuracy, but such methods are usually limited to target recognition in static or semi-static environments and lack effective adaptability to dynamically changing environments;

[0004] In addition, existing solutions have obvious shortcomings when facing complex and ever-changing three-dimensional dynamic environments. First, a single sensor or a simple combination is difficult to meet the requirements of high precision, real-time and robustness at the same time; second, traditional data fusion methods often ignore the time synchronization problem between different sensors, resulting in inaccurate information fusion; finally, most existing solutions fail to fully consider the importance of drone behavior patterns and their predicted motion trajectories, and cannot achieve intelligent task allocation and resource optimization, thus limiting the overall effectiveness of the system. Summary of the invention

[0005] The embodiment of the present invention provides a drone detection and tracking method and system based on the combination of laser radar and vision, which is used to solve the problems in the prior art of inaccurate information fusion, inability to achieve intelligent task allocation and resource optimization configuration, and lack of effective adaptability to dynamically changing environments.

[0006] In a first aspect, an embodiment of the present invention provides a method for detecting and tracking a drone based on a combination of laser radar and vision, comprising:

[0007] Use a distributed computing framework to process real-time communications between high-resolution lidar and intelligent visual sensor nodes in a deployed hybrid sensor array to obtain timestamp information.

[0008] According to the timestamp information, the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node are time-space calibrated to generate a multi-source data set;

[0009] Based on the multi-source data set, a deep reinforcement learning algorithm is used to perform data fusion to obtain multi-source heterogeneous data, and the multi-source heterogeneous data is used to obtain the drone behavior pattern and predicted motion trajectory through behavior pattern mining;

[0010] According to the UAV behavior pattern and predicted motion trajectory, an analysis is performed through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensor resource configuration plan. Based on the optimized sensor resource configuration plan, the working state adjustment and tracking strategy of the intelligent visual sensor node are processed to obtain an emergency correction program;

[0011] According to the emergency correction procedure, edge computing capabilities are integrated into the intelligent visual sensor nodes to process local data and make decisions to generate drone detection and tracking instructions.

[0012] Optionally, according to the drone behavior pattern and predicted motion trajectory, an analysis is performed through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensor resource configuration scheme, and based on the optimized sensor resource configuration scheme, the working state adjustment and tracking strategy of the intelligent visual sensor node are processed to obtain an emergency correction program, including:

[0013] Using the behavior patterns and predicted motion trajectories of drones, the task priority, coverage range, and complementarity of intelligent visual sensor nodes and lidar data are dynamically evaluated to obtain an initial sensor resource configuration model.

[0014] Based on the initial sensor resource configuration model, a multi-objective optimization algorithm is applied to adjust the working parameters of the intelligent visual sensor nodes, and the collaboration efficiency and energy consumption of the intelligent visual sensor nodes are evaluated to obtain an optimized sensor resource configuration scheme;

[0015] Using the optimized sensor resource configuration scheme, testing is performed under different interference conditions through simulation technology to identify potential problems and formulate corresponding preventive measures;

[0016] The preventive measures are converted into specific operating instructions, and combined with the dynamic changes of the drone monitored in real time, the working status and tracking strategy of the intelligent visual sensor node are adjusted in real time to generate an emergency correction program.

[0017] Optionally, based on the initial sensor resource configuration model, a multi-objective optimization algorithm is applied to adjust the working parameters of the intelligent visual sensor nodes, and the collaboration efficiency and energy consumption of the intelligent visual sensor nodes are evaluated to obtain an optimized sensor resource configuration scheme, including:

[0018] Using the initial sensor resource configuration model, combined with the drone behavior pattern and predicted motion trajectory, the working parameters of the intelligent visual sensor node are initialized and configured to obtain an initial parameter configuration set, wherein the working parameters include: scanning frequency, image resolution, viewing angle and data transmission priority;

[0019] According to the initial parameter configuration set, a multi-objective optimization algorithm is applied to evaluate the collaboration efficiency, energy consumption and real-time response speed between the intelligent visual sensor nodes, and the interactive impact under different scenarios is simulated to dynamically iteratively adjust the working parameters of each intelligent visual sensor node to form a candidate parameter configuration scheme;

[0020] Based on the candidate parameter configuration schemes, a dynamic risk assessment system is introduced to perform risk quantification analysis on each candidate configuration scheme, calculate the impact of external environmental changes and internal system states, and screen out high-reliability configuration schemes;

[0021] By utilizing the high reliability configuration scheme, an environmental adaptability evaluation mechanism is implemented, and simulation tests are performed on complex electromagnetic environments and optical interference conditions to obtain an optimized sensor resource configuration scheme.

[0022] Optionally, according to the initial parameter configuration set, a multi-objective optimization algorithm is applied to evaluate the collaboration efficiency, energy consumption and real-time response speed between the intelligent visual sensor nodes, and the interactive impact under different scenarios is simulated to dynamically iteratively adjust the working parameters of each intelligent visual sensor node to form a candidate parameter configuration scheme, including:

[0023] Using the initial parameter configuration set, applying a multi-objective optimization algorithm, evaluating the collaboration efficiency, energy consumption, and real-time response speed between intelligent visual sensor nodes, and generating a preliminary adjusted parameter configuration scheme by simulating the interaction effects under different scenarios;

[0024] Based on the parameter configuration scheme after the preliminary adjustment, a scene adaptability analysis module is introduced to evaluate the applicability of each intelligent visual sensor node in different task environments to form a scene-adaptive parameter configuration scheme;

[0025] According to the parameter configuration scheme adapted to the scenario, the working parameters of each intelligent visual sensor node are optimized through a dynamic iterative adjustment mechanism to obtain the best parameter configuration scheme;

[0026] By using the optimal parameter configuration scheme, combined with multi-source information fusion technology, and combined with laser radar and visual sensor data, the working parameters of the intelligent visual sensor node are adjusted to obtain a candidate parameter configuration scheme.

[0027] Optionally, according to the timestamp information, the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node are time-space calibrated to generate a multi-source data set, including:

[0028] The timestamp information is combined with an environmental perception algorithm to accurately synchronize the angle information point cloud data stream of the high-resolution laser radar with the multi-view image frames captured by the intelligent visual sensor node to obtain a time-synchronized data pair and record environmental parameters;

[0029] Based on the time-synchronized data pairs, an adaptive spatial coordinate conversion algorithm is applied, and conversion parameters are dynamically adjusted in combination with recorded environmental parameters to map the three-dimensional point cloud data of the laser radar to the two-dimensional image plane of each intelligent visual sensor node to generate a spatially aligned data set;

[0030] According to the spatially aligned dataset, a geometric consistency verification mechanism is introduced to detect and correct geometric mismatch problems through a deep learning model, and semantic segmentation technology is used to identify and exclude data interference in non-target areas to form an initial multi-source dataset;

[0031] The initial multi-source data set is used to implement spatiotemporal correlation optimization processing, fuse continuous data frames in the time dimension, and use a time series prediction model to predict data distribution to generate a target multi-source data set.

[0032] Optionally, based on the multi-source data set, a deep reinforcement learning algorithm is used to perform data fusion to obtain multi-source heterogeneous data, and the multi-source heterogeneous data is used to obtain the drone behavior pattern and predicted motion trajectory through behavior pattern mining, including:

[0033] Utilizing the multi-source data set, a deep reinforcement learning algorithm is used to jointly analyze and process the angle information point cloud data stream of the laser radar and the multi-view image frames captured by the intelligent visual sensor node to obtain multi-source heterogeneous data;

[0034] According to the multi-source heterogeneous data, a multimodal perception model is constructed, data characteristics from different sensors are integrated, and an enhanced multimodal perception model is generated;

[0035] Based on the enhanced multimodal perception model, a behavior pattern mining algorithm is applied to extract the behavior characteristics of the drone from multi-source heterogeneous data to form an initial behavior pattern library, which includes the position, speed, direction and behavior intention of the drone;

[0036] The initial behavior pattern library is used in combination with historical data and real-time updated perception information to predict the motion trajectory of the UAV through a time series prediction model to generate UAV behavior patterns and predicted motion trajectories, wherein the real-time updated perception information includes: multi-perspective image frames, angle information point cloud data streams, environmental changes, UAV status updates, and external interference and mission instruction changes.

[0037] Optionally, according to the emergency correction procedure, edge computing capabilities are integrated into intelligent visual sensor nodes to process local data and make decisions to generate drone detection and tracking instructions, including:

[0038] Using the emergency correction program, the edge computing module is deployed to the intelligent visual sensor node, so that each intelligent visual sensor node has local data processing capabilities and obtains a distributed computing network;

[0039] Based on the distributed computing network, local data collected by the intelligent visual sensor nodes are preprocessed in real time to extract feature information related to drone detection and tracking to obtain optimized local feature data;

[0040] Based on the optimized local feature data, an intelligent decision-making algorithm is applied to independently execute a fast decision-making process on each intelligent visual sensor node to determine the best action plan for the current environment and generate preliminary detection and tracking instructions;

[0041] Based on the preliminary detection and tracking instructions, the collaborative mechanism under the self-organizing network is used to fuse the instructions, adjust and optimize each intelligent visual sensor node, and obtain comprehensive UAV detection and tracking instructions.

[0042] In a second aspect, an embodiment of the present invention provides a drone detection and tracking system based on a combination of laser radar and vision, comprising:

[0043] A processing module, for performing real-time communication processing on high-resolution lidar and intelligent visual sensor nodes in a deployed hybrid sensor array using a distributed computing framework to obtain timestamp information;

[0044] A calibration module, used to perform spatiotemporal calibration on the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node according to the timestamp information, so as to generate a multi-source data set;

[0045] A fusion module is used to perform data fusion based on the multi-source data set using a deep reinforcement learning algorithm to obtain multi-source heterogeneous data, and obtain the drone behavior pattern and predicted motion trajectory by using the multi-source heterogeneous data and mining the behavior pattern;

[0046] An analysis module is used to analyze the UAV behavior pattern and predicted motion trajectory through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensor resource configuration plan, and based on the optimized sensor resource configuration plan, process the working state adjustment and tracking strategy of the intelligent visual sensor node to obtain an emergency correction program;

[0047] A formulation module is used to integrate edge computing capabilities into intelligent visual sensor nodes, process local data and make decisions, and generate drone detection and tracking instructions according to the emergency correction procedure.

[0048] In a third aspect, an embodiment of the present invention provides a computing device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute a drone detection and tracking method based on the combination of lidar and vision as described in any one of the first aspects.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement a drone detection and tracking method based on the combination of lidar and vision as described in any one of the first aspects.

[0050] In an embodiment of the present invention, a distributed computing framework is used to perform real-time communication processing on high-resolution laser radars and intelligent visual sensor nodes in a deployed hybrid sensor array to obtain timestamp information; based on the timestamp information, the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node are time-space calibrated to generate a multi-source data set; based on the multi-source data set, a deep reinforcement learning algorithm is used to perform data fusion to obtain multi-source heterogeneous data, and the multi-source heterogeneous data is used to obtain the behavior pattern and predicted motion trajectory of the unmanned aerial vehicle through behavior pattern mining; based on the behavior pattern and predicted motion trajectory of the unmanned aerial vehicle, an intelligent task allocation protocol under a self-organizing network is used to analyze and generate an optimized sensor resource configuration scheme, and based on the optimized sensor resource configuration scheme, the intelligent visual sensor The working state adjustment and tracking strategy of the node are processed to obtain an emergency correction program; according to the emergency correction program, the edge computing capability is integrated into the intelligent visual sensor node, local data is processed and decision-making is made, and drone detection and tracking instructions are generated; the technical solution provided by the present invention ensures the consistency of data from different sensors, improves the quality and accuracy of data fusion, enables the system to more accurately identify and track targets in complex environments, enhances the perception ability of the system, and can also predict the future movement path of the drone, adjust the monitoring strategy in advance, improve the prediction ability and response speed of the system, ensure the reasonable allocation of resources, reduce redundant configuration, improve the overall efficiency and flexibility of the system, improve the real-time processing capability of the system, and enhance the speed and accuracy of local decision-making, ensuring that efficient detection and tracking performance can be maintained even in complex environments;

[0051] Furthermore, using the initial parameter configuration set, a multi-objective optimization algorithm is applied to evaluate the collaborative efficiency, energy consumption and real-time response speed between intelligent visual sensor nodes, and the interaction effects in different scenarios are simulated to generate a preliminary adjusted parameter configuration scheme. This step ensures that the working parameters of each node are scientifically evaluated to achieve the best performance; based on the preliminary adjusted parameter configuration scheme, a scene adaptability analysis module is introduced to evaluate the applicability of each intelligent visual sensor node in different task environments to form a scene-adapted parameter configuration scheme. This approach takes into account the diversity of actual application scenarios, enabling the system to maintain good performance under various conditions; according to the scene-adapted parameter configuration scheme, the working parameters of each intelligent visual sensor node are optimized through a dynamic iterative adjustment mechanism to obtain the optimal parameter configuration scheme. The dynamic adjustment mechanism ensures that the system can continuously optimize its own performance, adapt to changing environmental conditions, and improve the stability and reliability of the system; using the optimal parameter configuration scheme, combined with multi-source information fusion technology, combined with lidar and visual sensor data, the working parameters of the intelligent visual sensor node are adjusted to obtain a candidate parameter configuration scheme. By fusing data from different types of sensors, the system's perception capability and decision-making accuracy are further improved, ensuring the successful completion of drone detection and tracking tasks.

[0052] These and other aspects of the present invention will become more apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A flowchart of a drone detection and tracking method based on the combination of laser radar and vision provided by an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the structure of a drone detection and tracking system based on the combination of laser radar and vision provided by an embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the structure of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0058] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] Figure 1 A flowchart of a method for detecting and tracking a drone based on a combination of laser radar and vision is provided for an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0061] Step 101: Use a distributed computing framework to perform real-time communication processing on high-resolution lidar and intelligent visual sensor nodes in a deployed hybrid sensor array to obtain timestamp information.

[0062] In this step, the distributed computing framework is a software architecture that allows computers distributed in different physical locations to work together to complete a common task. In this step, it is used to coordinate the communication between multiple high-resolution lidar and intelligent visual sensor nodes deployed in a wide area. These sensor nodes can collect environmental data in real time and transmit the data to the central or edge server through the network;

[0063] First, each sensor node is connected to a distributed computing framework through a high-speed wireless network to ensure low latency and high reliability of data transmission. When each sensor node captures data (such as the angle information point cloud data stream of the lidar and the multi-view image frame of the intelligent visual sensor node), they will be accompanied by an accurate timestamp information. The timestamp is generated by the global positioning system (GPS) or other synchronization mechanisms to mark the exact moment of data collection. This step is the basis for subsequent spatiotemporal calibration, ensuring that data from different sources can be compared and fused on the same time basis.

[0064] For example, in a city-level drone monitoring project, multiple high-resolution lidar and intelligent visual sensor nodes installed on the top of buildings formed a sensor network covering the entire city. Each node was equipped with a GPS receiver to ensure that all collected data had a unified timestamp. Through such a setting, the time consistency of all sensor data can be guaranteed even in a busy urban environment, providing a solid foundation for subsequent drone detection and tracking.

[0065] Step 102: Based on the timestamp information, perform spatiotemporal calibration on the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node to generate a multi-source data set.

[0066] In this step, time-space calibration refers to the process of adjusting the data from different sensors to make them consistent in both time and space. In this step, the angle information point cloud data stream and the multi-view image frame are calibrated to the same time point and spatial coordinate system, thus forming a multi-source data set, which provides a basis for subsequent data fusion and analysis.

[0067] Based on the timestamp information obtained in step 101, the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by the intelligent visual sensor node are calibrated in time and space. Specifically, each pair of point cloud data and image frames with the same timestamp is mapped to the same three-dimensional coordinate system to eliminate the deviation caused by the difference in sensor positions. This process may involve the application of a geometric transformation algorithm to ensure that different types of sensor data can be correctly aligned in space.

[0068] For example, continuing with the above-mentioned city-level drone monitoring project, after time and space calibration, the originally independent lidar point cloud data and visual image frames can now be displayed in a unified spatial coordinate system. For example, in an emergency response drill, the position of the drone relative to ground buildings and other objects can be accurately displayed, helping the command center make quick decisions.

[0069] Step 103: Based on the multi-source data set, a deep reinforcement learning algorithm is used to perform data fusion to obtain multi-source heterogeneous data. The multi-source heterogeneous data is used to mine the behavior pattern to obtain the drone behavior pattern and predict the motion trajectory.

[0070] In this step, the deep reinforcement learning algorithm is a machine learning method that combines the powerful representation ability of deep neural networks with the goal-oriented nature of reinforcement learning. Here, it is used to extract valuable information from multi-source data sets, namely the behavior patterns of drones and their predicted motion trajectories. The so-called multi-source heterogeneous data refers to data from different types of sensors with different formats and characteristics.

[0071] Based on the multi-source data set after time and space calibration, a deep reinforcement learning algorithm is applied for data fusion. This algorithm not only integrates the three-dimensional spatial information provided by the lidar and the two-dimensional image information given by the visual sensor, but also considers environmental factors such as lighting changes and weather conditions. By learning from a large amount of historical data, the algorithm can identify various behavior patterns of drones, including flight paths, speed changes, and possible behavioral intentions. In addition, the algorithm can also predict the movement trajectory of drones in the future and provide guidance for the subsequent tracking strategy.

[0072] For example, in the actual application of city-level drone monitoring projects, deep reinforcement learning algorithms have successfully learned the behavioral patterns of drones from a large amount of historical data. For example, in daily patrol missions, the flight route of the drone can be predicted in advance, and the parameters of other monitoring equipment can be adjusted accordingly to ensure continuous and effective tracking. This capability greatly improves initiative and foresight, and reduces the chance of accidents.

[0073] Step 104: According to the drone behavior pattern and predicted motion trajectory, an analysis is performed through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensor resource configuration plan. Based on the optimized sensor resource configuration plan, the working state adjustment and tracking strategy of the intelligent visual sensor node are processed to obtain an emergency correction program.

[0074] In this step, a self-organizing network refers to a network structure that does not require central control, in which nodes can autonomously adjust their behavior according to their own status and environmental changes. The intelligent task allocation protocol is a set of rules that guide nodes on how to efficiently collaborate to complete specific tasks. In this step, the configuration of sensor resources is optimized through the intelligent task allocation protocol under the self-organizing network based on the behavior pattern and predicted motion trajectory of the drone;

[0075] According to the drone behavior pattern and predicted motion trajectory obtained in step 103, the sensor resources are optimally configured through the intelligent task allocation protocol under the self-organizing network. Specifically, the protocol will evaluate the needs of the current task, such as the area or time period that needs to be monitored, and adjust the working parameters of each intelligent visual sensor node accordingly, such as scanning frequency, image resolution, etc. At the same time, the protocol will also formulate an emergency correction procedure to respond quickly when encountering abnormal situations, such as the drone suddenly changes its flight path or a new interference source appears;

[0076] For example, continuing with the city-level drone monitoring project, when a drone is detected to have deviated from its scheduled flight route, the intelligent task allocation protocol under the self-organizing network is immediately activated, and nearby sensor nodes are reconfigured to monitor the area in a more intensive manner. At the same time, an emergency correction procedure is also formulated to prepare to add additional monitoring equipment or adjust the working status of existing equipment when necessary to ensure that the continuous tracking of the drone is not interrupted.

[0077] Step 105: According to the emergency correction procedure, edge computing capabilities are integrated into the intelligent visual sensor node to process local data and make decisions, and generate drone detection and tracking instructions.

[0078] In this step, edge computing refers to the technology of processing data at the edge of the network (i.e., close to the data source), which can reduce data transmission delays, improve response speed, and reduce the burden on central servers. In this step, edge computing capabilities are integrated into smart visual sensor nodes, enabling them to process local data locally and make decisions instantly;

[0079] According to the emergency correction procedure, edge computing capabilities are integrated into intelligent visual sensor nodes, so that each node can process local data locally and make decisions quickly based on the actual situation. For example, if a sensor node detects that a drone is approaching, it can immediately adjust its working parameters, such as increasing the scanning frequency or improving the image resolution, and send signals to adjacent nodes to notify them to prepare as well. Finally, each node generates drone detection and tracking instructions based on the latest situation to ensure that the response is both timely and accurate.

[0080] For example, in the actual operation of the city-level drone monitoring project, once the emergency correction procedure is triggered, the intelligent visual sensor nodes located in key positions can take immediate action. For example, in an emergency, when an unauthorized drone is found to enter the restricted airspace, all nearby sensor nodes quickly adjust their working status, increase the monitoring density of the area, and generate detection and tracking instructions in a timely manner to ensure that safety measures can be implemented quickly.

[0081] Through the implementation of steps 101 to 105, the UAV has achieved a series of efficient operations from data collection, spatiotemporal calibration, data fusion to sensor resource allocation and local decision-making, which not only significantly improved the accuracy and reliability of UAV detection and tracking, but also enhanced adaptability and response speed. In particular, by introducing edge computing capabilities, the flexibility and robustness are further improved, ensuring stable and efficient performance even in complex and changing environments. The overall solution provides strong technical support for city-level UAV monitoring and ensures the effectiveness of public safety and airspace management.

[0082] In actual operation, in order to solve the problem of resource allocation efficiency and response speed, according to the above embodiment, in step 104, according to the drone behavior pattern and predicted motion trajectory, the intelligent task allocation protocol under the self-organizing network is analyzed to generate an optimized sensor resource configuration scheme. Based on the optimized sensor resource configuration scheme, the working state adjustment and tracking strategy of the intelligent visual sensor node are processed to obtain an emergency correction program, including:

[0083] The UAV behavior patterns and predicted motion trajectories are used to dynamically evaluate the task priority, coverage range and complementarity of the intelligent visual sensor nodes with lidar data, and generate an initial sensor resource allocation model. Based on this model, a multi-objective optimization algorithm is applied to adjust the working parameters and evaluate the collaboration efficiency and energy consumption to obtain the optimized sensor resource allocation plan. The performance under different interference conditions is tested through simulation technology, potential problems are identified and preventive measures are formulated. The preventive measures are converted into operating instructions, and the working status and tracking strategy of the intelligent visual sensor nodes are adjusted in real time in combination with the dynamic changes of the UAV under real-time monitoring, and finally an emergency correction program is generated.

[0084] In this embodiment, the drone behavior pattern refers to the typical flight path, speed change and other characteristics of the drone obtained through data analysis. The predicted motion trajectory is an estimate of the future movement path of the drone based on historical data and real-time information. Task priority refers to the importance and urgency of each sensor node determined based on the drone's behavior pattern. Coverage represents the spatial area that each intelligent visual sensor node can effectively monitor. Complementarity describes the complementary relationship between data from different sensors (such as lidar and visual sensing) to improve overall detection accuracy.

[0085] In the embodiment of the present application, the UAV behavior pattern and predicted motion trajectory are first used to dynamically evaluate the task priority, coverage and complementarity of the intelligent visual sensor node with the lidar data to obtain the initial sensor resource allocation model. This process ensures that the resource allocation meets the current environmental requirements and has the foresight for future changes.

[0086] Then, based on the initial sensor resource configuration model, a multi-objective optimization algorithm is applied to adjust the working parameters (such as scanning frequency, image resolution, etc.) of the intelligent visual sensor nodes, and the collaboration efficiency and energy consumption between the intelligent visual sensor nodes are evaluated to obtain an optimized sensor resource configuration scheme.

[0087] Next, using the optimized sensor resource configuration scheme, we conduct tests under different interference conditions (such as electromagnetic interference, optical camouflage, etc.) through simulation technology to identify potential problems and formulate corresponding preventive measures. This stage ensures that the system can operate stably even in harsh environments;

[0088] Finally, the preventive measures are converted into specific operational instructions, and combined with the real-time monitoring of the dynamic changes of the drone, the working status and tracking strategy of the intelligent visual sensor node are adjusted in real time to generate an emergency correction program. This process enables the system to respond quickly to emergencies and maintain continuous and effective tracking performance.

[0089] For example, in a city-level drone monitoring project, a hybrid sensor array consisting of multiple high-resolution lidar and intelligent visual sensor nodes was deployed. When the system detects that a drone is approaching restricted airspace, it first predicts its future motion trajectory based on the drone's historical flight path and real-time location, and adjusts the task priorities of nearby sensor nodes accordingly. For example, nodes close to the drone's predicted path are given higher priority and their coverage is expanded to ensure more intensive monitoring.

[0090] Subsequently, a multi-objective optimization algorithm is applied to adjust the working parameters of these high-priority nodes, such as increasing the scanning frequency or improving the image resolution, while taking into account the collaboration efficiency and energy consumption between nodes to generate the most optimized resource allocation plan. In order to verify the effectiveness of the plan, the system simulates various possible interference conditions, such as strong electromagnetic interference or complex lighting changes, through simulation technology, identifies potential risk points, and formulates corresponding preventive measures;

[0091] Once a potential risk is detected, such as a sudden change in direction of the drone or the appearance of a new source of interference, the system will immediately convert preventive measures into specific operational instructions and send them to the relevant sensor nodes. These nodes will adjust their working status according to the latest instructions, such as adjusting the viewing angle or starting the backup power supply to ensure continuous tracking of the drone. Throughout the process, the system continuously receives real-time feedback from each node and dynamically adjusts resource allocation to ensure the best detection and tracking effect.

[0092] In order to further improve the detection and tracking performance of the intelligent visual sensor nodes in complex environments, as another embodiment, in step 104, based on the initial sensor resource configuration model, a multi-objective optimization algorithm is applied to adjust the working parameters of the intelligent visual sensor nodes, and the collaboration efficiency and energy consumption of the intelligent visual sensor nodes are evaluated to obtain an optimized sensor resource configuration scheme, including:

[0093] The initial sensor resource configuration model is combined with the drone behavior pattern and predicted motion trajectory to initialize the working parameters of the intelligent visual sensor node (such as scanning frequency, image resolution, viewing angle and data transmission priority) to obtain the initial parameter configuration set; based on this configuration set, a multi-objective optimization algorithm is applied to evaluate the collaboration efficiency, energy consumption and real-time response speed, and the working parameters are dynamically adjusted by simulating the interaction effects in different scenarios to form candidate parameter configuration schemes; a dynamic risk assessment system is introduced to screen out high-reliability configuration schemes, and the environmental adaptability evaluation mechanism is used to test complex electromagnetic and optical interference conditions, and finally the optimized sensor resource configuration scheme is obtained.

[0094] In this embodiment, the initial sensor resource configuration model refers to a preliminary configuration scheme generated according to the drone behavior pattern and predicted motion trajectory, which is used to guide the working parameter setting of the intelligent visual sensor node. The working parameters include scanning frequency, image resolution, viewing angle, and data transmission priority, etc. These parameters directly affect the data acquisition efficiency and quality of the node. The multi-objective optimization algorithm is an algorithm that can simultaneously consider multiple optimization objectives (such as collaborative efficiency, energy consumption, and real-time response speed) to find the best parameter combination. Simulation is to simulate the system behavior in different scenarios by computer and evaluate the interaction between each intelligent visual sensor node. The candidate parameter configuration scheme is a plurality of possible parameter combinations generated during the optimization process, from which the optimal solution is selected. The dynamic risk assessment system is used to perform risk quantification analysis on each candidate configuration scheme and evaluate the impact of external environmental changes and internal system status. The high-reliability configuration scheme is a scheme that can maintain stable performance under various conditions selected from the candidate scheme. The environmental adaptability assessment mechanism is used to test the performance of the high-reliability configuration scheme under complex electromagnetic environments and optical interference conditions;

[0095] In the embodiment of the present application, firstly, the working parameters of the intelligent visual sensor node (such as scanning frequency, image resolution, viewing angle and data transmission priority) are initialized and configured using the initial sensor resource configuration model combined with the drone behavior pattern and predicted motion trajectory to obtain an initial parameter configuration set. This process ensures that each node can be properly configured according to the current task requirements and expected environmental changes;

[0096] Then, based on the initial parameter configuration set, a multi-objective optimization algorithm is applied to evaluate the collaboration efficiency, energy consumption, and real-time response speed between intelligent visual sensor nodes, and the interactive impact under different scenarios is simulated to dynamically iterate and adjust the working parameters of each intelligent visual sensor node to form a candidate parameter configuration scheme. This stage aims to find a set of parameter combinations that can perform well under a variety of conditions;

[0097] Next, based on the candidate parameter configuration schemes, a dynamic risk assessment system is introduced to conduct risk quantification analysis on each candidate configuration scheme, calculate the impact of external environmental changes and internal system status, and screen out high-reliability configuration schemes. This step ensures that the selected scheme not only performs well under ideal conditions, but also maintains the stability and reliability of the system under unforeseen circumstances;

[0098] Finally, using the high-reliability configuration scheme, the environmental adaptability evaluation mechanism was implemented, and simulation tests were conducted on complex electromagnetic environments and optical interference conditions to obtain the optimized sensor resource configuration scheme. Through this series of rigorous tests and evaluations, the final resource configuration scheme can provide stable and efficient detection and tracking services in various complex environments.

[0099] For example, in a city-level drone monitoring project, a hybrid sensor array consisting of multiple high-resolution lidar and intelligent visual sensor nodes was deployed. When the system detects a drone approaching restricted airspace, it first predicts its future motion trajectory based on the drone’s historical flight path and real-time location, and adjusts the task priority of nearby sensor nodes accordingly;

[0100] The system uses the initial sensor resource configuration model to set initial working parameters for each intelligent visual sensor node, such as the scanning frequency is set to 30 times per second, the image resolution is 1080p, the viewing angle covers a range of 90 degrees, and the data transmission priority is set to the highest. These parameters ensure that the node can quickly capture relevant information about the drone in the early stage;

[0101] Subsequently, the system applies a multi-objective optimization algorithm, taking into account factors such as the cooperation efficiency, energy consumption, and real-time response speed between nodes, and dynamically adjusts the working parameters of each node by simulating the interaction effects in different scenarios. For example, in densely built-up areas, some nodes may be required to increase the scanning frequency to cope with complex background environments, while other nodes may reduce the image resolution to save energy;

[0102] To ensure the reliability of the solution, the system introduced a dynamic risk assessment system to conduct a risk quantification analysis on each candidate configuration solution. Taking into account factors such as electromagnetic interference or weather changes that may exist in urban environments, the system selects configuration solutions that can still maintain high performance under these conditions;

[0103] Finally, through the environmental adaptability assessment mechanism, the system conducted simulation tests on complex electromagnetic environments and optical interference conditions. The test results show that the selected high-reliability configuration scheme can maintain stable performance in various complex environments. The system applies these optimized parameters to actual operations, ensuring continuous and effective tracking of drones, and can respond quickly and adjust tracking strategies even in emergencies to ensure public safety.

[0104] In order to further improve the collaboration efficiency, energy consumption optimization and real-time response speed of intelligent visual sensor nodes in complex environments, according to the initial parameter configuration set described in the previous embodiment, a multi-objective optimization algorithm is applied to evaluate the collaboration efficiency, energy consumption and real-time response speed between intelligent visual sensor nodes, and the interactive impact under different scenarios is simulated. The working parameters of each intelligent visual sensor node are dynamically iteratively adjusted to form a candidate parameter configuration scheme, including:

[0105] The initial parameter configuration set is used to apply a multi-objective optimization algorithm to evaluate the collaborative efficiency, energy consumption and real-time response speed between intelligent visual sensor nodes. The interaction effects in different scenarios are simulated to generate a preliminary adjusted parameter configuration scheme. Based on the preliminary adjusted parameter configuration scheme, a scene adaptability analysis module is introduced to evaluate the applicability of each intelligent visual sensor node in different task environments to form a scene-adaptive parameter configuration scheme. According to the scene-adaptive parameter configuration scheme, the working parameters of each intelligent visual sensor node are optimized through a dynamic iterative adjustment mechanism to obtain the optimal parameter configuration scheme. The optimal parameter configuration scheme is used to adjust the working parameters of the intelligent visual sensor nodes in combination with multi-source information fusion technology and laser radar and visual sensor data to obtain candidate parameter configuration schemes.

[0106] In this embodiment, the multi-objective optimization algorithm is an algorithm that can simultaneously consider multiple optimization objectives (such as collaborative efficiency, energy consumption, and real-time response speed) to find the best parameter combination. Simulation is to simulate the system behavior in different scenarios by computer and evaluate the interaction between each intelligent visual sensor node. The parameter configuration scheme after preliminary adjustment is the parameter combination generated during the initial optimization process, which provides a basis for subsequent more refined adjustments. The scene adaptability analysis module is a specially designed tool or process for evaluating the applicability and performance of each intelligent visual sensor node in different task environments. The dynamic iterative adjustment mechanism refers to the process of gradually optimizing the working parameters through continuous feedback and learning in a constantly changing environment. The optimal parameter configuration scheme is the optimal parameter combination obtained from a series of adjustments, which ensures that the system can perform well under various conditions. Multi-source information fusion technology combines data from different types of sensors (such as lidar and visual sensing) to improve data quality and decision accuracy.

[0107] In the embodiment of the present application, firstly, the initial parameter configuration set is used to apply a multi-objective optimization algorithm to evaluate the collaboration efficiency, energy consumption, and real-time response speed between intelligent visual sensor nodes, and the interaction effects under different scenarios are simulated to generate a preliminary adjusted parameter configuration scheme. This step aims to explore a set of parameter combinations that can perform well under a variety of conditions as a basis for further optimization;

[0108] Next, based on the parameter configuration scheme after the preliminary adjustment, the scene adaptability analysis module is introduced to evaluate the applicability of each intelligent visual sensor node in different task environments and form a scene-adaptive parameter configuration scheme. This stage ensures that each node can play its maximum performance in its specific application scenario. For example, a higher scanning frequency may be required in a high-density building area, while the image resolution can be reduced in an open area to save energy;

[0109] Then, according to the parameter configuration scheme adapted to the scenario, the working parameters of each intelligent visual sensor node are optimized through a dynamic iterative adjustment mechanism to obtain the best parameter configuration scheme. This process involves continuous monitoring and feedback, allowing the system to flexibly adjust according to actual conditions, ensuring that resource configuration is always in the optimal state, and improving the overall performance and stability of the system;

[0110] Finally, the optimal parameter configuration scheme is used, combined with multi-source information fusion technology, and combined with lidar and visual sensor data to make final adjustments to the working parameters of the intelligent visual sensor node to obtain a candidate parameter configuration scheme. By integrating data from different types of sensors, the system can more accurately capture and understand environmental changes, thereby making more informed decisions and enhancing the accuracy and reliability of detection and tracking;

[0111] For example, in a city-level drone monitoring project, a hybrid sensor array consisting of multiple high-resolution lidar and intelligent visual sensor nodes was deployed. When the system detects a drone approaching restricted airspace, it first predicts its future motion trajectory based on the drone’s historical flight path and real-time location, and adjusts the task priority of nearby sensor nodes accordingly;

[0112] The system uses the initial parameter configuration set to set initial working parameters for each intelligent visual sensor node, such as the scanning frequency is set to 30 times per second, the image resolution is 1080p, the viewing angle covers a range of 90 degrees, and the data transmission priority is set to the highest. These parameters ensure that the node can quickly capture relevant information about the drone in the early stage;

[0113] Subsequently, the system applies a multi-objective optimization algorithm, taking into account factors such as the cooperation efficiency, energy consumption, and real-time response speed between nodes, and generates a preliminary adjusted parameter configuration plan by simulating the interaction effects in different scenarios. For example, in densely built-up areas, some nodes may be required to increase the scanning frequency to cope with complex background environments, while other nodes may reduce the image resolution to save energy;

[0114] In order to ensure that the solution is suitable for the actual mission environment, the system introduces a scene adaptability analysis module to evaluate the applicability of each intelligent visual sensor node under different conditions. For example, in low-light environments, the system will adjust the image resolution of certain nodes to ensure sufficient brightness capture capabilities; in areas with strong electromagnetic interference, the data transmission priority will be optimized to ensure communication quality;

[0115] Next, the system continuously optimizes the working parameters of each intelligent visual sensing node through a dynamic iterative adjustment mechanism. For example, if a node detects that a drone suddenly changes its flight path, it will immediately adjust its own working parameters, such as increasing the scanning frequency or improving the image resolution, and send signals to neighboring nodes to notify them to get ready as well;

[0116] Finally, the system uses the best parameter configuration scheme, combined with multi-source information fusion technology, combined with lidar and visual sensor data, to make final adjustments to the working parameters of the intelligent visual sensor node and obtain candidate parameter configuration schemes. By integrating the precise distance information provided by lidar and the rich image details captured by visual sensing, the system can more comprehensively understand and respond to environmental changes, improving the accuracy and reliability of drone detection and tracking.

[0117] In order to solve the problem of data synchronization and spatial alignment between the high-resolution laser radar and the intelligent visual sensor node, as another embodiment, according to step 102, the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node are time-space calibrated according to the timestamp information to generate a multi-source data set, including:

[0118] The timestamp information is combined with the environmental perception algorithm to perform precise time synchronization processing on the angle information point cloud data stream of the high-resolution laser radar and the multi-view image frames captured by the intelligent visual sensor nodes to obtain a time-synchronized data pair, and the environmental parameters are recorded. Based on the time-synchronized data pair, an adaptive spatial coordinate conversion algorithm is applied, and the conversion parameters are dynamically adjusted in combination with the recorded environmental parameters to map the three-dimensional point cloud data of the laser radar to the two-dimensional image plane of each intelligent visual sensor node to generate a spatially aligned data set. Based on the spatially aligned data set, a geometric consistency verification mechanism is introduced to detect and correct geometric mismatch problems through a deep learning model. At the same time, semantic segmentation technology is used to identify and eliminate data interference in non-target areas to form an initial multi-source data set. Using the initial multi-source data set, spatiotemporal correlation optimization processing is implemented to fuse continuous data frames in the time dimension, and a time series prediction model is used to predict data distribution to generate a target multi-source data set.

[0119] In this embodiment, the environment perception algorithm refers to a technology that identifies and understands the surrounding environment by analyzing the data collected by the sensor. Time synchronization processing is a process that ensures that the data from different sensors are consistent in the time dimension for subsequent processing. The time-synchronized data pair is a pair of data consisting of lidar point cloud data and visual image frames that have been time-synchronized. Environmental parameters include factors such as temperature, humidity, and light intensity that may affect sensor performance, and these parameters are used to adjust the parameters in the data conversion process. The adaptive spatial coordinate conversion algorithm is an algorithm that can dynamically adjust the conversion parameters according to actual conditions, and is used to map three-dimensional point cloud data to a two-dimensional image plane. The geometric consistency check mechanism is used to detect and correct the geometric mismatch problem between point cloud data and images. Semantic segmentation technology is an image processing method that can identify different objects in an image and exclude data interference in non-target areas. The initial multi-source data set is a data set formed after geometric consistency check and semantic segmentation. Spatiotemporal correlation optimization processing is a process of optimizing continuous data frames in the time and space dimensions. A time series prediction model is a model that predicts future data distribution based on historical data.

[0120] In the embodiment of the present application, firstly, the timestamp information is combined with the environmental perception algorithm to accurately synchronize the angle information point cloud data stream of the high-resolution laser radar with the multi-view image frames captured by the intelligent visual sensor node, obtain the time-synchronized data pair, and record the environmental parameters. This step ensures that the data from different sensors can be compared and fused on the same time basis, while taking into account the impact of environmental changes on the sensor; next, based on the time-synchronized data pair, an adaptive spatial coordinate conversion algorithm is applied, and the conversion parameters are dynamically adjusted in combination with the recorded environmental parameters, and the three-dimensional point cloud data of the laser radar is mapped to the two-dimensional image plane of each intelligent visual sensor node to generate a spatial alignment data set. This process solves the problem of inconsistent spatial coordinates caused by differences in sensor positions, so that different types of data can be represented in a unified coordinate system; then, according to the spatial alignment data set, a geometric consistency verification mechanism is introduced, and the geometric mismatch problem is detected and corrected through a deep learning model, and the semantic segmentation technology is used to identify and exclude data interference in non-target areas to form an initial multi-source data set. This step improves the quality and accuracy of the data, reduces the interference of noise and irrelevant information, and provides a more reliable basis for subsequent analysis; finally, the initial multi-source data set is used to implement spatiotemporal correlation optimization processing, fuse continuous data frames in the time dimension, and use a time series prediction model to predict data distribution and generate a target multi-source data set. This step not only enhances the temporal coherence of the data, but also predicts the trend of future data changes, further improving the system's prediction ability and response speed.

[0121] For example, in a city-level drone monitoring project, a hybrid sensor array consisting of multiple high-resolution lidars and intelligent visual sensor nodes was deployed. When the system starts running, each sensor node will collect a large amount of environmental data, such as the angle information point cloud data stream provided by the lidar and the multi-view image frames captured by the intelligent visual sensor node; the system first uses the timestamp information combined with the environmental perception algorithm to accurately synchronize the data to ensure that all data are on the same time basis. For example, in an emergency response drill, even if the sensor nodes are located in different geographical locations, the system can accurately synchronize the data they collect to the same time and record the environmental parameters at that time, such as temperature, humidity and light conditions; next, based on the time-synchronized data pairs, the system applies an adaptive spatial coordinate transformation algorithm to map the three-dimensional point cloud data of the lidar to the two-dimensional image plane of the intelligent visual sensor node. In this process, the system dynamically adjusts the transformation parameters according to the recorded environmental parameters to compensate for errors caused by weather or light changes, thereby generating a spatially aligned data set; in order to improve data quality and accuracy, the system introduces a geometric consistency verification mechanism to detect and correct any geometric mismatch problems through a deep learning model. For example, if there is a deviation between the image frame of a sensor node and the lidar point cloud data, the system will automatically adjust to make the two more matched. In addition, the system also uses semantic segmentation technology to identify and exclude data interference in non-target areas, such as background buildings or other irrelevant objects, to form an initial multi-source data set; finally, the system uses the initial multi-source data set to implement spatiotemporal correlation optimization processing, fuse continuous data frames in the time dimension, and use a time series prediction model to predict the data distribution in the future. For example, before a drone enters restricted airspace, the system can predict the future flight path based on its current behavior pattern and adjust the working status of nearby sensor nodes in advance to ensure continuous and effective tracking.

[0122] In order to solve the problems of insufficient data fusion, inaccurate behavior pattern recognition, and inaccurate prediction of motion trajectory in the prior art, as another embodiment, according to step 103, based on the multi-source data set, a deep reinforcement learning algorithm is used to perform data fusion to obtain multi-source heterogeneous data, and the multi-source heterogeneous data is used to obtain the drone behavior pattern and predicted motion trajectory through behavior pattern mining, including:

[0123] Utilizing the multi-source data set, a deep reinforcement learning algorithm is used to jointly analyze and process the angle information point cloud data stream of the laser radar and the multi-view image frames captured by the intelligent visual sensor node to obtain multi-source heterogeneous data. According to the multi-source heterogeneous data, a multimodal perception model is constructed to integrate the data characteristics from different sensors to generate an enhanced multimodal perception model. Based on the enhanced multimodal perception model, a behavior pattern mining algorithm is applied to extract the behavior characteristics of the drone from the multi-source heterogeneous data to form an initial behavior pattern library, which includes the position, speed, direction and behavior intention of the drone. Utilizing the initial behavior pattern library, combined with historical data and real-time updated perception information, the motion trajectory of the drone is predicted through a time series prediction model to generate a drone behavior pattern and predicted motion trajectory, wherein the real-time updated perception information includes: multi-view image frames, angle information point cloud data streams, environmental changes, drone status updates, external interference and task instruction changes.

[0124] In this embodiment, the deep reinforcement learning algorithm is a machine learning method that combines the powerful representation ability of deep neural networks with the goal-oriented nature of reinforcement learning. Multi-source heterogeneous data refers to data from different types of sensors with different formats and characteristics, such as the angle information point cloud data stream of lidar and the multi-view image frames captured by intelligent visual sensor nodes. A multimodal perception model refers to a model that can integrate data characteristics from different sensors to more comprehensively describe the environment and target state. The behavioral pattern mining algorithm is used to extract the behavioral characteristics of the drone from multi-source heterogeneous data to form an initial behavioral pattern library, which contains information such as the drone's position, speed, direction, and behavioral intention. The time series prediction model is a model that predicts the distribution of future data based on historical data, and is used to estimate the future motion trajectory of the drone.

[0125] In the embodiment of the present application, firstly, the multi-source data set is used to jointly analyze and process the angle information point cloud data stream of the laser radar and the multi-view image frames captured by the intelligent visual sensor node using a deep reinforcement learning algorithm to obtain multi-source heterogeneous data. This step ensures that different types of data can be effectively integrated under a unified framework, improving the quality and accuracy of data fusion; next, a multimodal perception model is constructed based on the multi-source heterogeneous data, integrating the data characteristics from different sensors to generate an enhanced multimodal perception model. This process not only enhances the ability to understand the environment, but also improves the adaptability to complex scenes, so that the system can better cope with detection and tracking tasks in various situations; then, based on the enhanced multimodal perception model, a behavior pattern mining algorithm is applied to extract the behavior characteristics of the drone from the multi-source heterogeneous data to form an initial behavior pattern library. The library contains information such as the position, speed, direction and behavior intention of the drone, which provides a solid foundation for subsequent behavior pattern analysis and motion trajectory prediction; finally, the initial behavior pattern library is used to predict the motion trajectory of the drone through a time series prediction model in combination with historical data and real-time updated perception information, and the drone behavior pattern and predicted motion trajectory are generated. The real-time updated perception information includes multi-view image frames, angle information point cloud data streams, environmental changes, drone status updates, as well as external interference and mission instruction changes, ensuring the timeliness and accuracy of predictions.

[0126] For example, in a city-level drone monitoring project, a hybrid sensor array consisting of multiple high-resolution laser radars and intelligent visual sensor nodes was deployed. When the system starts running, each sensor node will continuously collect a large amount of environmental data, such as the angle information point cloud data stream provided by the laser radar and the multi-view image frames captured by the intelligent visual sensor node; the system first uses the multi-source data set and uses a deep reinforcement learning algorithm to jointly analyze and process these data to obtain multi-source heterogeneous data. For example, in a daily patrol mission, the system combines the three-dimensional spatial information of the laser radar with the two-dimensional image information of the visual sensor to form a richer and more detailed description of the environment; next, based on the multi-source heterogeneous data, the system constructs a multimodal perception model and further generates an enhanced multimodal perception model. This model can more comprehensively understand the objects in the environment and their dynamic changes, and improve the system's environmental perception ability. For example, it can distinguish between static buildings and moving vehicles, and track the different behaviors of multiple drones at the same time; then, based on the enhanced multimodal perception model, the system applies a behavioral pattern mining algorithm to extract the behavioral features of drones from multi-source heterogeneous data to form an initial behavioral pattern library. The library records the position, speed, direction and possible behavioral intentions of the drone, providing a basis for subsequent analysis. For example, the system can identify whether a drone is conducting a routine patrol or performing a special mission; finally, using the initial behavior pattern library, the system combines historical data and real-time updated perception information to predict the motion trajectory of the drone through a time series prediction model. Real-time updated information includes the latest multi-view image frames, angle information point cloud data streams, environmental changes (such as weather conditions), drone status updates (such as speed changes), and external interference (such as electromagnetic interference) and mission instruction changes. This information helps the system predict the future motion path of the drone more accurately, adjust the working parameters of other monitoring equipment in advance, and ensure continuous and effective tracking.

[0127] In order to solve the problems of high data processing delay, slow decision response and low resource utilization efficiency in the prior art, as another embodiment, according to step 105, according to the emergency correction procedure, the edge computing capability is integrated into the intelligent visual sensor node, local data is processed and decision-made, and drone detection and tracking instructions are generated, including:

[0128] By using the emergency correction program, the edge computing module is deployed to the intelligent visual sensor node, so that each intelligent visual sensor node has local data processing capabilities, and a distributed computing network is obtained. Based on the distributed computing network, the local data collected by the intelligent visual sensor node is preprocessed in real time, and the feature information related to drone detection and tracking is extracted to obtain optimized local feature data. According to the optimized local feature data, the intelligent decision-making algorithm is applied to independently execute the fast decision-making process on each intelligent visual sensor node, determine the best action plan for the current environment, and generate preliminary detection and tracking instructions. Based on the preliminary detection and tracking instructions, the instructions are fused through the collaborative mechanism under the self-organizing network, and each intelligent visual sensor node is adjusted and optimized to obtain comprehensive drone detection and tracking instructions.

[0129] In this embodiment, the emergency correction procedure refers to the process by which the system quickly adjusts the working state and takes corresponding measures when an abnormal situation or emergency is detected. The edge computing module is a technical unit that can process data close to the data source (i.e., the edge of the network), which reduces data transmission delays, improves response speed, and reduces the burden on the central server. A distributed computing network refers to a network structure composed of multiple nodes with local data processing capabilities, each of which can independently complete a specific task. Local data refers to data that only involves the current node and its neighboring nodes, which is used for rapid decision making. Feature information is key data extracted from the original data that can describe the characteristics of the target object. The optimized local feature data is more accurate and effective local data after preprocessing. The intelligent decision algorithm is a set of rules or models that guide nodes on how to collaborate efficiently to complete specific tasks. The collaboration mechanism under the self-organizing network is a network architecture that does not require central control and allows nodes to autonomously coordinate resource allocation and task execution.

[0130] In the embodiment of the present application, first, the emergency correction program is used to deploy the edge computing module to the intelligent visual sensor node, so that each intelligent visual sensor node has local data processing capabilities, and a distributed computing network is obtained. This step ensures that each node can process data locally, reduces dependence on the central server, and improves the response speed and flexibility of the system; Next, based on the distributed computing network, the local data collected by the intelligent visual sensor node is preprocessed in real time, and the feature information related to the detection and tracking of the drone is extracted to obtain the optimized local feature data. Through this preprocessing, the amount of data transmitted to the central server can be significantly reduced, and the accuracy of subsequent analysis can be improved; Then, according to the optimized local feature data, the intelligent decision-making algorithm is applied to independently execute the fast decision-making process on each intelligent visual sensor node, determine the best action plan for the current environment, and generate preliminary detection and tracking instructions. This stage allows each node to respond quickly according to the latest local data, enhancing the real-time and adaptability of the system; Finally, based on the preliminary detection and tracking instructions, the instructions are fused through the collaborative mechanism under the self-organizing network, and each intelligent visual sensor node is adjusted and optimized to obtain a comprehensive drone detection and tracking instruction. This process ensures the collaborative work among all nodes, avoids the limitations that may be caused by the decision-making of a single node, and improves the performance and reliability of the overall system.

[0131] For example, in a city-level drone monitoring project, when the system detected an unauthorized drone entering restricted airspace, an emergency correction procedure was immediately triggered. At this point, the edge computing module was quickly deployed to the intelligent visual sensor nodes, giving these nodes the ability to process local data and forming a distributed computing network; each intelligent visual sensor node began to preprocess the collected local data in real time, extracting feature information related to drone detection and tracking, such as the location, speed, and direction of the drone. Through this preprocessing, the node can identify which data is critical to the current task and use these optimized local feature data for further analysis; next, each node applies an intelligent decision-making algorithm to independently execute a fast decision-making process to determine the best course of action for the current environment. For example, if a node detects that a drone is approaching quickly, it can immediately adjust its scanning frequency or increase the image resolution to obtain a clearer target image. At the same time, the node will also send a signal to the adjacent nodes to notify them to be ready; finally, based on these preliminary detection and tracking instructions, the system performs command fusion through a collaborative mechanism under the self-organizing network to adjust and optimize the working parameters of each intelligent visual sensor node. For example, if multiple nodes detect the same drone, the system coordinates their working status to ensure maximum coverage and minimum redundancy. Ultimately, the system generates comprehensive drone detection and tracking instructions to ensure continuous and effective tracking of drones, and can respond quickly to emergencies to ensure public safety.

[0132] Figure 2 The present invention provides a schematic diagram of a UAV detection and tracking system based on a combination of laser radar and vision, as shown in FIG. Figure 2 As shown, the system includes:

[0133] The processing module 21 is used to perform real-time communication processing on the high-resolution laser radar and intelligent visual sensor nodes in the deployed hybrid sensor array using a distributed computing framework to obtain timestamp information;

[0134] A calibration module 22 is used to perform spatiotemporal calibration on the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node according to the timestamp information to generate a multi-source data set;

[0135] A fusion module 23 is used to perform data fusion based on the multi-source data set using a deep reinforcement learning algorithm to obtain multi-source heterogeneous data, and obtain the drone behavior pattern and predicted motion trajectory by using the multi-source heterogeneous data and mining the behavior pattern;

[0136] The analysis module 24 is used to analyze the UAV behavior pattern and predicted motion trajectory through the intelligent task allocation protocol under the self-organizing network to generate an optimized sensor resource configuration plan, and based on the optimized sensor resource configuration plan, process the working state adjustment and tracking strategy of the intelligent visual sensor node to obtain an emergency correction program;

[0137] The formulation module 25 is used to integrate edge computing capabilities into intelligent visual sensor nodes according to the emergency correction procedure, process local data and make decisions, and generate drone detection and tracking instructions.

[0138] Figure 2 The UAV detection and tracking system based on the combination of laser radar and vision can be performed Figure 1 The implementation principle and technical effect of the drone detection and tracking method based on the combination of laser radar and vision described in the embodiment shown are not repeated here. The specific way in which each module and unit performs operations in the drone detection and tracking system based on the combination of laser radar and vision in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0139] In one possible design, Figure 2 The drone detection and tracking system based on the combination of laser radar and vision in the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0140] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0141] The processing component 32 is used to: use a distributed computing framework to perform real-time communication processing on the high-resolution laser radar and intelligent visual sensor nodes in the deployed hybrid sensor array to obtain timestamp information; based on the timestamp information, perform spatiotemporal calibration on the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node to generate a multi-source data set; based on the multi-source data set, use a deep reinforcement learning algorithm to perform data fusion to obtain multi-source heterogeneous data, and use the multi-source heterogeneous data to obtain the drone behavior pattern and predicted motion trajectory through behavioral pattern mining; based on the drone behavior pattern and predicted motion trajectory, analyze through the intelligent task allocation protocol under the self-organizing network to generate an optimized sensor resource configuration plan, based on the optimized sensor resource configuration plan, process the working state adjustment and tracking strategy of the intelligent visual sensor node to obtain an emergency correction program; according to the emergency correction program, integrate edge computing capabilities into the intelligent visual sensor node, process and make decisions on local data, and generate drone detection and tracking instructions.

[0142] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0143] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0144] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0145] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0146] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0147] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0148] The embodiment of the present invention further provides a computer storage medium storing a computer program, which can achieve the above-mentioned Figure 1 The illustrated embodiment is a method for detecting and tracking a drone based on a combination of laser radar and vision.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A drone detection and tracking method based on the combination of laser radar and vision, characterized in that: include: Use a distributed computing framework to process real-time communications between high-resolution lidar and intelligent visual sensor nodes in a deployed hybrid sensor array to obtain timestamp information. According to the timestamp information, the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node are time-space calibrated to generate a multi-source data set; Based on the multi-source data set, a deep reinforcement learning algorithm is used to perform data fusion to obtain multi-source heterogeneous data, and the multi-source heterogeneous data is used to obtain the drone behavior pattern and predicted motion trajectory through behavior pattern mining; According to the UAV behavior pattern and predicted motion trajectory, an analysis is performed through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensor resource configuration plan. Based on the optimized sensor resource configuration plan, the working state adjustment and tracking strategy of the intelligent visual sensor node are processed to obtain an emergency correction program; According to the emergency correction procedure, edge computing capabilities are integrated into the intelligent visual sensor nodes to process local data and make decisions to generate drone detection and tracking instructions.

2. The method according to claim 1, characterized in that According to the UAV behavior pattern and predicted motion trajectory, an analysis is performed through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensor resource configuration scheme. Based on the optimized sensor resource configuration scheme, the working state adjustment and tracking strategy of the intelligent visual sensor node are processed to obtain an emergency correction program, including: Using the behavior patterns and predicted motion trajectories of drones, the task priority, coverage range, and complementarity of intelligent visual sensor nodes and lidar data are dynamically evaluated to obtain an initial sensor resource configuration model. Based on the initial sensor resource configuration model, a multi-objective optimization algorithm is applied to adjust the working parameters of the intelligent visual sensor nodes, and the collaboration efficiency and energy consumption of the intelligent visual sensor nodes are evaluated to obtain an optimized sensor resource configuration scheme; Using the optimized sensor resource configuration scheme, testing is performed under different interference conditions through simulation technology to identify potential problems and formulate corresponding preventive measures; The preventive measures are converted into specific operating instructions, and combined with the dynamic changes of the drone monitored in real time, the working status and tracking strategy of the intelligent visual sensor node are adjusted in real time to generate an emergency correction program.

3. The method according to claim 2, characterized in that Based on the initial sensor resource configuration model, a multi-objective optimization algorithm is applied to adjust the working parameters of the intelligent visual sensor nodes, and the collaboration efficiency and energy consumption of the intelligent visual sensor nodes are evaluated to obtain an optimized sensor resource configuration scheme, including: Using the initial sensor resource configuration model, combined with the drone behavior pattern and predicted motion trajectory, the working parameters of the intelligent visual sensor node are initialized and configured to obtain an initial parameter configuration set, wherein the working parameters include: scanning frequency, image resolution, viewing angle and data transmission priority; According to the initial parameter configuration set, a multi-objective optimization algorithm is applied to evaluate the collaboration efficiency, energy consumption and real-time response speed between the intelligent visual sensor nodes, and the interactive impact under different scenarios is simulated to dynamically iteratively adjust the working parameters of each intelligent visual sensor node to form a candidate parameter configuration scheme; Based on the candidate parameter configuration schemes, a dynamic risk assessment system is introduced to perform risk quantification analysis on each candidate configuration scheme, calculate the impact of external environmental changes and internal system states, and screen out high-reliability configuration schemes; By utilizing the high reliability configuration scheme, an environmental adaptability evaluation mechanism is implemented, and simulation tests are performed on complex electromagnetic environments and optical interference conditions to obtain an optimized sensor resource configuration scheme.

4. The method according to claim 3, characterized in that According to the initial parameter configuration set, a multi-objective optimization algorithm is applied to evaluate the collaboration efficiency, energy consumption and real-time response speed between the intelligent visual sensor nodes, and the interactive impact under different scenarios is simulated to dynamically iteratively adjust the working parameters of each intelligent visual sensor node to form a candidate parameter configuration scheme, including: Using the initial parameter configuration set, applying a multi-objective optimization algorithm, evaluating the collaboration efficiency, energy consumption, and real-time response speed between intelligent visual sensor nodes, and generating a preliminary adjusted parameter configuration scheme by simulating the interaction effects under different scenarios; Based on the parameter configuration scheme after the preliminary adjustment, a scene adaptability analysis module is introduced to evaluate the applicability of each intelligent visual sensor node in different task environments to form a scene-adaptive parameter configuration scheme; According to the parameter configuration scheme adapted to the scenario, the working parameters of each intelligent visual sensor node are optimized through a dynamic iterative adjustment mechanism to obtain the best parameter configuration scheme; By using the optimal parameter configuration scheme, combined with multi-source information fusion technology, and combined with laser radar and visual sensor data, the working parameters of the intelligent visual sensor node are adjusted to obtain a candidate parameter configuration scheme.

5. The method according to claim 1, characterized in that According to the timestamp information, the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node are time-space calibrated to generate a multi-source data set, including: The timestamp information is combined with an environmental perception algorithm to accurately synchronize the angle information point cloud data stream of the high-resolution laser radar with the multi-view image frames captured by the intelligent visual sensor node to obtain a time-synchronized data pair and record environmental parameters; Based on the time-synchronized data pairs, an adaptive spatial coordinate conversion algorithm is applied, and conversion parameters are dynamically adjusted in combination with recorded environmental parameters to map the three-dimensional point cloud data of the laser radar to the two-dimensional image plane of each intelligent visual sensor node to generate a spatially aligned data set; According to the spatially aligned dataset, a geometric consistency verification mechanism is introduced to detect and correct geometric mismatch problems through a deep learning model, and semantic segmentation technology is used to identify and exclude data interference in non-target areas to form an initial multi-source dataset; The initial multi-source data set is used to implement spatiotemporal correlation optimization processing, fuse continuous data frames in the time dimension, and use a time series prediction model to predict data distribution to generate a target multi-source data set.

6. The method according to claim 1, characterized in that Based on the multi-source data set, a deep reinforcement learning algorithm is used to perform data fusion to obtain multi-source heterogeneous data. The multi-source heterogeneous data is used to mine the behavior pattern to obtain the drone behavior pattern and predict the motion trajectory, including: Utilizing the multi-source data set, a deep reinforcement learning algorithm is used to jointly analyze and process the angle information point cloud data stream of the laser radar and the multi-view image frames captured by the intelligent visual sensor node to obtain multi-source heterogeneous data; According to the multi-source heterogeneous data, a multimodal perception model is constructed, data characteristics from different sensors are integrated, and an enhanced multimodal perception model is generated; Based on the enhanced multimodal perception model, a behavior pattern mining algorithm is applied to extract the behavior characteristics of the drone from multi-source heterogeneous data to form an initial behavior pattern library, which includes the position, speed, direction and behavior intention of the drone; The initial behavior pattern library is used in combination with historical data and real-time updated perception information to predict the motion trajectory of the UAV through a time series prediction model to generate UAV behavior patterns and predicted motion trajectories, wherein the real-time updated perception information includes: multi-perspective image frames, angle information point cloud data streams, environmental changes, UAV status updates, and external interference and mission instruction changes.

7. The method according to claim 1, characterized in that According to the emergency correction procedure, edge computing capabilities are integrated into intelligent visual sensor nodes to process local data and make decisions to generate drone detection and tracking instructions, including: Using the emergency correction program, the edge computing module is deployed to the intelligent visual sensor node, so that each intelligent visual sensor node has local data processing capabilities and obtains a distributed computing network; Based on the distributed computing network, local data collected by the intelligent visual sensor nodes are preprocessed in real time to extract feature information related to drone detection and tracking to obtain optimized local feature data; Based on the optimized local feature data, an intelligent decision-making algorithm is applied to independently execute a fast decision-making process on each intelligent visual sensor node to determine the best action plan for the current environment and generate preliminary detection and tracking instructions; Based on the preliminary detection and tracking instructions, the collaborative mechanism under the self-organizing network is used to fuse the instructions, adjust and optimize each intelligent visual sensor node, and obtain comprehensive UAV detection and tracking instructions.

8. A drone detection and tracking system based on the combination of laser radar and vision, characterized in that: include: Use a distributed computing framework to process real-time communications between high-resolution lidar and intelligent visual sensor nodes in a deployed hybrid sensor array to obtain timestamp information. According to the timestamp information, the angle information point cloud data stream generated by the high-resolution laser radar and the multi-view image frames captured by each intelligent visual sensor node are time-space calibrated to generate a multi-source data set; Based on the multi-source data set, a deep reinforcement learning algorithm is used to perform data fusion to obtain multi-source heterogeneous data, and the multi-source heterogeneous data is used to obtain the drone behavior pattern and predicted motion trajectory through behavior pattern mining; According to the UAV behavior pattern and predicted motion trajectory, an analysis is performed through an intelligent task allocation protocol under a self-organizing network to generate an optimized sensor resource configuration plan. Based on the optimized sensor resource configuration plan, the working state adjustment and tracking strategy of the intelligent visual sensor node are processed to obtain an emergency correction program; According to the emergency correction procedure, edge computing capabilities are integrated into the intelligent visual sensor nodes to process local data and make decisions to generate drone detection and tracking instructions.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a drone detection and tracking method based on the combination of laser radar and vision as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a computer, a drone detection and tracking method based on the combination of laser radar and vision as described in any one of claims 1 to 7 is implemented.

Images