Unmanned aerial vehicle countering method and system based on image depth recognition

Abstract

The invention relates to the technical field of unmanned aerial vehicle countering, and discloses an unmanned aerial vehicle countering method and system based on image depth recognition, and the method comprises the steps: carrying out the real-time monitoring of a target airspace through a camera device, and obtaining a monitoring image; identifying the monitoring image according to the target identification model, judging whether an unmanned aerial vehicle exists, and if so, analyzing the monitoring image according to a feature matching algorithm, and judging whether the unmanned aerial vehicle is a suspicious unmanned aerial vehicle; if the unmanned aerial vehicle is a suspicious unmanned aerial vehicle, calculating a three-dimensional coordinate of the suspicious unmanned aerial vehicle by adopting a triangulation method, and predicting a flight path of the suspicious unmanned aerial vehicle according to the three-dimensional coordinate to obtain a predicted flight path; and performing threat assessment on the suspicious unmanned aerial vehicle according to the predicted flight path, determining a threat level, and triggering a corresponding countering strategy. The unmanned aerial vehicle identification and positioning accuracy is effectively improved, the suspicious unmanned aerial vehicle is accurately hit, and an effective solution is provided for airspace safety protection of an important area.

Description

Technical Field

[0001] The present invention relates to the technical field of UAV countermeasures, and particularly to a UAV countermeasure method and system based on image depth recognition. Background Art

[0002] With the rapid development of UAV technology and the reduction of costs, UAVs may interfere with the normal operation of important facilities.

[0003] Currently, traditional countermeasure methods mainly use radio interference, GPS spoofing, acoustic suppression, etc. However, these methods all have certain limitations. On the one hand, these methods rely on external signals and require prior knowledge of the UAV communication frequency band or rely on GPS signals, resulting in poor effects on UAVs using encryption protocols or autonomous navigation. On the other hand, radar systems have insufficient detection capabilities for low-altitude, slow-speed, and small targets. In addition, electromagnetic interference will also interfere with the environment and may accidentally damage legitimate devices, while the range of acoustic suppression is limited. Currently, there are also countermeasure technologies based on image recognition to overcome the dependence on external signals, but the existing image depth recognition technologies still have problems such as being unable to adapt to complex environments and having low recognition and positioning accuracy in actual deployment. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a UAV countermeasure method and system based on image depth recognition, so as to solve the problems of traditional countermeasure methods relying on signals and poor environmental adaptability, and achieve the effects of improving the accuracy of UAV recognition and positioning and realizing precise countermeasures against UAVs.

[0005] In a first aspect, the present invention provides a UAV countermeasure method based on image depth recognition, the method comprising:

[0006] Real-time monitoring of a target airspace by a pre-deployed camera device and obtaining a monitoring image;

[0007] Identifying the monitoring image according to a preset target recognition model, and determining whether there is a UAV in the monitoring image. If there is a UAV, analyzing the monitoring image according to a feature matching algorithm to determine whether the UAV is a suspicious UAV;

[0008] If it is a suspicious UAV, calculating the three-dimensional coordinates of the suspicious UAV using the triangulation method, and predicting the flight trajectory of the suspicious UAV according to the three-dimensional coordinates to obtain a predicted flight trajectory;

[0009] Performing a threat assessment on the suspicious UAV according to the predicted flight trajectory, determining the threat level of the suspicious UAV, and triggering a corresponding countermeasure strategy according to the threat level.

[0010] Further, the imaging device includes three groups, and each group of imaging devices includes a plurality of cameras. Among them, the first group of imaging devices is used for mid-low altitude coverage monitoring, the second group of imaging devices is used for high altitude coverage monitoring, and the third group of imaging devices is used for lateral coverage monitoring;

[0011] The field of view overlap rate of adjacent cameras in each group of imaging devices is not less than the overlap rate threshold, and at least two cameras are not within the direct sunlight range in each time period.

[0012] Further, the step of performing real-time monitoring on the target airspace through the pre-deployed imaging device and obtaining the monitoring image includes:

[0013] Performing real-time monitoring on the target airspace through the pre-deployed imaging device;

[0014] When any camera in the imaging device detects an airborne object, call other adjacent cameras not interfered by sunlight for collaborative observation to obtain multiple target images, and the multiple target images are multiple target images captured by a plurality of cameras.

[0015] Further, the step of identifying the monitoring image according to a preset target recognition model and determining whether there is a drone in the monitoring image includes:

[0016] Preprocess the multiple target images and input them into the target recognition model to determine whether the airborne object is a drone;

[0017] Among them, the target recognition model uses a residual network as the backbone network, and includes a residual network, a feature pyramid network, a region proposal network, and a feature extraction and classification network connected in sequence;

[0018] A channel attention module is set in each residual block of the residual network, and a convolutional block attention module is set after each output layer of the feature pyramid network;

[0019] The residual network is used to extract feature maps of different stages of the multiple target images, the feature pyramid network is used to fuse each feature map to generate enhanced feature maps, the region proposal network is used to anchor the enhanced feature maps to generate candidate regions, and the feature extraction and separation network is used to extract features and classify the candidate regions to obtain the predicted target type.

[0020] Further, the step of analyzing the monitoring image according to the feature matching algorithm and determining whether the drone is a suspicious drone includes:

[0021] Extract the contour of the drone from the monitoring image according to the edge detection algorithm, and calculate the geometric features of the drone according to the extracted contour;

[0022] Perform color space conversion and histogram calculation on the UAV image in the surveillance image according to the extracted contour to obtain the color characteristics of the UAV;

[0023] Analyze the surveillance image according to the optical flow method, calculate the motion vector of the UAV, and obtain the motion characteristics of the UAV according to the motion vector;

[0024] Use cosine similarity to calculate the similarity between the geometric characteristics and color characteristics and the corresponding characteristics in the feature database respectively, use the dynamic time warping algorithm to calculate the similarity between the motion characteristics and the motion characteristics in the feature database, and perform weighted summation on the calculated similarities to obtain the matching degree of the UAV;

[0025] Judge whether the UAV is a suspicious UAV according to the matching degree.

[0026] Further, the steps of calculating the three-dimensional coordinates of the suspicious UAV by using the triangulation method include:

[0027] Perform target detection on any one of the multi-channel target images to determine the position of the suspicious UAV in the image, and estimate the initial height of the suspicious UAV according to the proportional relationship between the position and the reference object size;

[0028] Use the feature point detection algorithm to extract target feature points from the multi-channel target images, and calculate the three-dimensional coordinates of the suspicious UAV by using the triangulation method according to the initial height, the target feature points and the calibration parameters of each camera;

[0029] Perform smoothing processing on the three-dimensional coordinates by using the Kalman filter to obtain smooth three-dimensional coordinates.

[0030] Further, the steps of predicting the flight trajectory of the suspicious UAV according to the three-dimensional coordinates to obtain the predicted flight trajectory include:

[0031] Obtain the three-dimensional coordinate sequence and weather environment parameters within a preset time period, and the weather environment parameters include wind speed, air flow intensity and air pressure;

[0032] Use the three-dimensional coordinate sequence and weather environment parameters as input data, input them into the pre-constructed trajectory prediction model, and obtain the predicted flight trajectory of the suspicious UAV. The trajectory prediction model is constructed based on the long short-term memory network, and the confidence interval of the trajectory prediction model is calculated based on the uncertainty propagation of Monte Carlo.

[0033] Further, the steps of performing threat assessment on the suspicious UAV according to the predicted flight trajectory, determining the threat level of the suspicious UAV, and triggering corresponding countermeasure strategies according to the threat level include:

[0034] Obtain the drone model and usage frequency of the suspicious drone, and determine the threat level of the suspicious drone according to the predicted flight trajectory, the drone model, and the usage frequency;

[0035] When the threat level is a low threat level, execute the target tracking strategy and continuously monitor the flight path of the suspicious drone through a camera device;

[0036] When the threat level is a medium threat level, execute the signal interference strategy and interfere with the suspicious drone by activating a signal jammer;

[0037] When the threat level is a high threat level, execute the drone countermeasure strategy and interfere with and intercept the suspicious drone by dispatching countermeasure drones.

[0038] Furthermore, the step of triggering the corresponding countermeasure strategy according to the threat level further includes:

[0039] When there are multiple suspicious drones with a high threat level, start multiple countermeasure drones to conduct collaborative countermeasures against the multiple suspicious drones. Among them, the collaborative operation trajectories of the multiple countermeasure drones are set with the shortest time to reach the positions of all suspicious drones as the constraint objective function.

[0040] In a second aspect, the present invention provides a drone countermeasure system based on image depth recognition. The system includes:

[0041] A target monitoring module, configured to perform real-time monitoring on a target airspace through a pre-deployed camera device and obtain monitoring images;

[0042] A target recognition module, configured to recognize the monitoring images according to a preset target recognition model, determine whether there is a drone in the monitoring images. If there is a drone, analyze the monitoring images according to a feature matching algorithm to determine whether the drone is a suspicious drone;

[0043] A trajectory prediction module, configured to, if it is a suspicious drone, calculate the three-dimensional coordinates of the suspicious drone using the triangulation method, and predict the flight trajectory of the suspicious drone according to the three-dimensional coordinates to obtain a predicted flight trajectory;

[0044] A decision-making countermeasure module, configured to perform threat assessment on the suspicious drone according to the predicted flight trajectory, determine the threat level of the suspicious drone, and trigger the corresponding countermeasure strategy according to the threat level.

[0045] The present invention provides an anti-drone method and system based on image depth recognition. The present invention achieves full coverage of the airspace through a hierarchical camera array, realizes the precise recognition and positioning of drones through target detection algorithms and multi-source information fusion technologies, and automatically evaluates the threat level and takes corresponding countermeasures through an intelligent decision-making mechanism, achieving precise strikes against suspicious drones. The present invention can effectively ensure the safety of the airspace, provides an effective solution for the airspace security protection of important areas, and has good practicability and popularizability. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a schematic flow chart of the anti-drone method based on image depth recognition in an embodiment of the present invention;

[0047] Figure 2 is a schematic structural diagram of the anti-drone system based on image depth recognition in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0049] Please refer to Figure 1 , an anti-drone method based on image depth recognition proposed in the first embodiment of the present invention, which includes steps S10 to S40:

[0050] Step S10, real-time monitor the target airspace through a pre-deployed imaging device and obtain monitoring images;

[0051] Step S20, identify the monitoring images according to a preset target recognition model, determine whether there is a drone in the monitoring images, and if there is a drone, analyze the monitoring images according to a feature matching algorithm to determine whether the drone is a suspicious drone;

[0052] Step S30, if it is a suspicious drone, use the triangulation method to calculate the three-dimensional coordinates of the suspicious drone, and predict the flight trajectory of the suspicious drone according to the three-dimensional coordinates to obtain a predicted flight trajectory;

[0053] Step S40, conduct a threat assessment on the suspicious drone according to the predicted flight trajectory, determine the threat level of the suspicious drone, and trigger a corresponding countermeasure according to the threat level.

[0054] In the present invention, relevant camera devices are pre-deployed in the target airspace to be monitored. These camera devices are used to monitor the target airspace. To ensure that the monitoring range can cover the entire airspace, in a preferred embodiment, the present invention provides a hierarchical camera array deployment scheme. The camera devices are divided into three groups. The first group is used for medium and low altitude coverage monitoring, the second group is used for high altitude coverage monitoring, and the third group is used for lateral coverage monitoring, so as to achieve multi-level and all-round monitoring of the target airspace. Specifically, for the first group of cameras for medium and low altitude coverage, they can be deployed within 3 meters above the ground, with an elevation angle of 30° - 60°, used to monitor the airspace within 0 - 300 meters, and are also equipped with a 2.8 - 12mm zoom lens; for the second group of cameras for high altitude coverage, they can be deployed on the ground or low-rise buildings, with a large elevation angle of 60° - 85°, used to monitor the high altitude area above 300 meters, and are equipped with a 12 - 50mm telephoto lens with optical image stabilization function; for the third group of cameras for lateral coverage, they can be deployed at high points, such as the top of buildings, signal towers, etc., with an adjustable elevation angle of -5° to 30°, used to monitor the surrounding lateral airspace within 0 - 1000 meters, and are equipped with a 5 - 100mm ultra-telephoto zoom lens; in addition, the field of view overlap rate of adjacent cameras in each group of camera devices is not less than 25%. High-speed pan-tilt heads with automatic exposure control and wide dynamic range functions can be used, and the horizontal rotation speed is not lower than 180° / second. Further, an automatic control system can also be used to control the camera devices. According to the real-time solar position information, the pan-tilt head of the camera is adjusted, and the orientation and duration of the camera are dynamically adjusted to ensure that at least the fields of view of two cameras avoid direct sunlight at all times of the day, so as to overcome the influence of light on the monitoring images.

[0055] Further, a preferred scheme for the deployment of a camera device also includes that at each deployment position of each group of camera devices, a plurality of cameras, that is, cameras, are equally spaced in the vertical direction. Calculate the vertical field of view overlap rate of two adjacent cameras above and below at the same deployment position, and calculate the horizontal field of view overlap rate of two adjacent cameras at the same height at adjacent deployment positions. When both the vertical field of view overlap rate and the horizontal field of view overlap rate are greater than 80%, the deployment of the camera device is completed.

[0056] Through the deployed camera devices, all-round real-time monitoring of the target airspace is carried out. When any camera in the camera device detects an airborne object, adjacent other cameras not interfered by sunlight are called for collaborative observation to obtain multiple target images. Here, the multiple target images refer to multiple target images captured by multiple cameras.

[0057] For the surveillance images captured by the imaging device, the target recognition model is used to determine whether the aerial object is a drone. In this embodiment, first, multiple target images are preprocessed, and then the target recognition model is used to determine whether the objects in these target images are drones. Here, the data preprocessing includes image enhancement through adaptive histogram equalization, noise suppression using Gaussian filtering, image correction through perspective transformation to eliminate perspective distortion, processing of overexposed and underexposed areas based on HDR technology, and elimination of the influence of camera jitter using a motion compensation algorithm, etc. The specific preprocessing steps can refer to the conventional preprocessing steps, and the corresponding preprocessing methods can be flexibly selected according to the actual situation of the target images, which will not be elaborated here one by one.

[0058] For the preprocessed target images, a target recognition model constructed based on a residual network is used for drone determination. Among them, the backbone network of the target recognition model is a residual network, and preferably the ResNet-50 model is used for feature extraction. A Feature Pyramid Network (FPN), a Region Proposal Network (RPN), and a feature extraction and classification network are also connected behind the residual network. In the target recognition model, a multi-level feature of the target image is extracted through the residual network. And in order to enhance the response of important channels, a Squeeze-and-Excitation (SE) module is embedded in each residual block of the residual network to dynamically weight the channel features through the SE module. For the extracted multi-level feature maps, based on the balance between performance and computing power, in this embodiment, three levels of feature maps, namely C3, C4, and C5, are selected and input into the Feature Pyramid Network (FPN). Through the FPN network, multi-scale feature fusion is performed, that is, C3 - C5 are fused through a top-down path and lateral connections to generate enhanced feature pyramid feature maps P3 - P7, where P6 and P7 are obtained by downsampling P5. And a Convolutional Block Attention Module (CBAM) is added after each output layer (P3 - P7) of the FPN network to perform both channel and spatial attention weighting simultaneously, improving the feature discrimination of small and large targets. The Region Proposal Network (RPN) sets multi-scale anchor boxes for each output layer of the FPN network to generate anchor points, generate candidate regions and their confidence scores. Preferably, a spatial attention module can also be added after the convolutional layer of the RPN to make the RPN focus on the foreground area and reduce background interference. The feature extraction and classification network is set behind the Region Proposal Network (RPN). Adopting a dynamic allocation strategy, according to the size of the candidate region, it is allocated to the corresponding level of the FPN. For example, small targets are allocated to P3, and large targets to P5. Fixed-size features are extracted from the specified level, and the precise position information is retained. Finally, a fully connected layer is used for target classification and bounding box fine-tuning to achieve target classification. Preferably, a non-local attention module can also be added before the fully connected layer of the feature extraction and classification network to capture long-range dependencies and improve the classification accuracy in complex scenarios. Through the target recognition model of this embodiment, accurate recognition of drones can be achieved.

[0059] For the identified drones, the next step is to determine whether the drone is a suspicious drone or a trusted drone. In this embodiment, a feature matching algorithm is used to determine suspicious drones, and the specific steps include:

[0060] Extract the contour of the drone from the surveillance image according to the edge detection algorithm, and calculate the geometric features of the drone based on the extracted contour;

[0061] Perform color space conversion and histogram calculation on the drone image in the surveillance image according to the extracted contour to obtain the color features of the drone;

[0062] Analyze the surveillance image according to the optical flow method, calculate the motion vector of the drone, and obtain the motion features of the drone based on the motion vector;

[0063] Use the cosine similarity to calculate the similarity between the geometric features and color features and the corresponding features in the feature database respectively, use the dynamic time warping algorithm to calculate the similarity between the motion features and the motion features in the feature database, and perform weighted summation on the calculated similarities to obtain the matching degree of the drone;

[0064] Judge whether the drone is a suspicious drone according to the matching degree.

[0065] In this embodiment, first, the contour of the drone is extracted through an edge detection algorithm such as the Canny or Sobel operator, that is, the main structural features of the drone such as the wings or fuselage are extracted through polygon fitting or Hough transformation, thereby obtaining a contour point set and a structural feature description; then, according to the extracted drone contour, the geometric parameters of the drone are calculated, including the aspect ratio, area, perimeter, and circularity, etc. Among them, the aspect ratio is determined by calculating the aspect ratio of the minimum bounding rectangle of the drone contour, the area is determined by calculating the area of the drone contour, and finally, a geometric parameter vector is output; at the same time, according to the extracted contour, the drone image in the image is determined, and color conversion and histogram calculation are performed, including converting the image from RGB to the HSV or Lab space, then calculating the hue (H), saturation (S), and value (V) histograms of the drone area, and statistically analyzing the spatial distribution of the main colors such as the color centroid, and finally, a color histogram and color distribution characteristics are output; for the motion characteristics of the drone, they are extracted by analyzing the surveillance images of the drone through the optical flow method, including calculating the surveillance image video sequence using sparse optical flow such as Lucas-Kanade or dense optical flow such as Farneback to obtain the motion vector of the drone, and then, according to the motion vector, the speed and acceleration of the drone are further deduced by calculating the displacement of the drone between adjacent frames, that is, by differentiating the displacement in time to obtain the estimated values of the speed and acceleration, and finally, the attitude change of the drone is estimated through feature point tracking or a deep learning model, that is, after obtaining the speed and acceleration information of the drone, this information is used to estimate the attitude change of the drone, including using the feature point tracking method to estimate the attitude of the drone by tracking the key points in the image, and using the deep learning model to learn and classify features such as the optical flow map to more accurately estimate the attitude change of the drone, such as the pitch angle, yaw angle, etc., and finally, the motion characteristics of the drone are obtained, including speed, acceleration, pitch angle, yaw angle, etc.

[0066] The geometric features, color features, and motion features of the UAV are obtained through the above steps. Then, based on these features, it can be determined whether the UAV is a suspicious UAV. In this embodiment, a feature database is pre-established, and the feature vectors of suspicious UAVs are stored in the database. By calculating the similarity between the extracted UAV features and the feature vectors stored in the database, it is determined whether the UAV is a suspicious UAV. Among them, the cosine similarity or Euclidean distance can be used to calculate the similarity of geometric features and color features, and the dynamic time warping algorithm can be used to calculate the similarity of motion features. Then, according to the importance of each feature, feature weights are pre-allocated, and the calculated similarities are weighted and summed to calculate the comprehensive matching score. Finally, the attribute of the UAV is determined based on the calculated matching degree. In this embodiment, the matching degree threshold can be determined through experiments or statistical methods. If the matching score is higher than the threshold, it is determined as a suspicious UAV; otherwise, it is determined as a normal UAV.

[0067] After determining that the monitored UAV is a suspicious UAV, it is also necessary to calculate the three-dimensional coordinates of the UAV to predict its flight trajectory. Among them, the calculation steps of the three-dimensional coordinates include:

[0068] Perform object detection on any one of the multiple target images to determine the position of the suspicious UAV in the image, and estimate the initial height of the suspicious UAV according to the proportional relationship between the position and the reference object size;

[0069] Use the feature point detection algorithm to extract target feature points from the multiple target images, and use the triangulation method to calculate the three-dimensional coordinates of the suspicious UAV according to the initial height, the target feature points, and the calibration parameters of each camera;

[0070] Use the Kalman filter to smooth the three-dimensional coordinates to obtain the smoothed three-dimensional coordinates.

[0071] In this embodiment, first, the initial height of the suspicious drone is estimated by analyzing the image captured by a single camera. Here, a proportional estimation method based on the perspective projection model of monocular vision is adopted. Specifically, the bounding box of the drone in the image is detected through an object detection algorithm to determine the position of the drone in the image. Then, according to the known size of the reference object, such as the actual size of the drone model, the initial height of the drone is estimated by combining the proportional relationship between the pixel size of the drone in the image and the actual size, the focal length of the camera, and the imaging model: z = ((f × L) / (p × Δp)), where f is the focal length, p is the pixel size, Δp is the pixel measurement value, and L is the diagonal length of the drone. Of course, to improve the accuracy of the initial height, height error compensation can also be performed, including introducing IMU data to compensate for the lens pitch angle error and using polynomial fitting to optimize the scale nonlinear distortion at close range (<50m). Finally, the initial height estimate value of the drone is obtained.

[0072] Then, the multi-channel target images are analyzed through a feature point detection algorithm. In this embodiment, the multi-channel target images are at least multi-angle images captured by three cameras. Preferably, these three cameras come from different groups of imaging devices, and the accuracy of the calculation results is improved by analyzing the images from different angles. When analyzing the multi-channel target images, for each perspective image, first, the key feature points of the drone in the image are extracted using a feature point detection algorithm, then the same feature points are matched in the images from different perspectives using a FLANN or BFMatcher matching algorithm, and finally, the matched feature point pairs in the multi-perspective images are output. Of course, deep learning features such as SuperPoint can also be used to improve the robustness of feature point extraction and matching. Here, only a preferred method is provided and there is no excessive limitation.

[0073] Based on the initial height and target feature points obtained in the above steps, and combined with the calibration parameters of each camera, the three-dimensional coordinates of the suspicious drone are calculated through triangulation. Among them, the calibration parameters of the camera include internal parameter calibration and external parameter calibration. The internal parameters include internal parameters such as the focal length, principal point coordinates, and distortion coefficients of each camera, and the external parameters include the relative position and attitude between the cameras, such as the rotation matrix and translation vector.

[0074] In this embodiment, the initial height Z serves as the Z-axis in the three-dimensional coordinate system. Combining the pixel coordinates in the monocular image and the internal parameters of the camera, the horizontal positions, namely the X-axis and Y-axis, are calculated through the perspective projection model. Assuming that the internal parameters of the camera include the focal length f and the principal point (u0, v0), and the pixel coordinates in the image are (u, v), the horizontal positions can be expressed as: X = f(u - u0) / Z, Y = f(v - v0) / Z. In this step, the calculation of the three-dimensional coordinates is mainly constrained by the initial height, and by establishing a proportional relationship, the two-dimensional pixel coordinates in the image are converted into three-dimensional space coordinates.

[0075] For the target feature points, using the epipolar geometry constraint, calculate the possible positions of the feature points in the three-dimensional space, that is, the pixel coordinates of the feature points. Then, according to the pixel coordinates of the feature points, the external parameters of each camera, and the above three-dimensional space coordinate constraint conditions, the precise three-dimensional coordinates of the suspicious drone are obtained through triangulation. The specific steps for coordinate calculation based on triangulation can refer to the conventional calculation steps and will not be repeated here.

[0076] In order to make the three-dimensional coordinates more accurate, in this embodiment, Kalman filtering is used to smooth the three-dimensional coordinates. This includes taking the position and velocity of the drone as state variables, establishing a state equation, taking the three-dimensional coordinates obtained by triangulation as the observed values, and through the Kalman filtering algorithm, combining the state equation and the observed values to smooth the position data of the drone. The Kalman filtering in this embodiment uses adaptive filtering, dynamically scales the Q matrix according to the motion acceleration to dynamically adjust the noise, and finally outputs the smoothed three-dimensional coordinates of the drone. Through Kalman filtering, noise and jitter can be effectively reduced, making the calculated three-dimensional coordinates of the drone more accurate. In this embodiment, through the target detection algorithm and multi-source information fusion technology, the surveillance image is analyzed, thus realizing the precise identification and positioning of the drone.

[0077] After determining the three-dimensional coordinates of the suspicious drone, the flight trajectory of the suspicious drone can be predicted. The specific steps include:

[0078] Obtain the three-dimensional coordinate sequence and weather environment parameters within a preset time period. The weather environment parameters include wind speed, air flow intensity, and air pressure;

[0079] Take the three-dimensional coordinate sequence and weather environment parameters as input data, input them into a pre-constructed trajectory prediction model, and obtain the predicted flight trajectory of the suspicious drone. The trajectory prediction model is constructed based on the long short-term memory network, and the confidence interval of the trajectory prediction model is calculated based on Monte Carlo uncertainty propagation.

[0080] In this embodiment, a trajectory prediction model based on a long short-term memory neural network model is adopted to predict the flight trajectory. The trajectory prediction model is trained using the position data of historical suspicious drones and historical environmental parameters. Here, environmental parameters such as wind speed and air flow are considered. The purpose of taking wind speed and air flow as additional input features is to enhance the adaptability of the model to the dynamic environment. The wind speed can be obtained through measurement. For the influence of air flow, the spatial distribution of air flow is modeled through Gaussian process regression, and the air flow prediction result is used as the auxiliary input of the LSTM. Of course, weather parameters such as temperature and humidity can also be referred to, and there are no excessive restrictions here.

[0081] These historical data and environmental data need to be preprocessed before training the model, such as aligning timestamps, normalizing, window sliding sampling, and data augmentation. Specifically, the position data and environmental data are aligned by time, and the position and environmental data are standardized. Then, the data is divided into time series of fixed length, and data diversity is increased through random noise injection and interpolation. During model training, the mean squared error is used as the loss function, AdamW is used as the optimizer, and the cosine annealing algorithm is used for learning rate scheduling. For the trained model, a three-dimensional coordinate sequence within a certain time period, such as a three-dimensional coordinate sequence within 10 seconds and weather environment parameters, is input into the trajectory prediction model to output the predicted flight trajectory in the future. In this model, in order to improve the prediction accuracy, the confidence interval is calculated using Monte Carlo-based uncertainty propagation. Specifically, the Dropout layer, that is, the random inactivation layer, is enabled in the inference stage of the model, and multiple samplings are performed, that is, multiple forward propagations are performed on the same input. The mean and variance are calculated based on the prediction results of each time, and then the uncertainty propagation parameter: σ 2 total =σ 2 model +σ 2 env is calculated, where σ 2 model represents the uncertainty parameter of the model, that is, the variance quantified through the Monte Carlo algorithm, and σ 2 env represents the uncertainty parameter of the environment.

[0082] In this embodiment, the environmental uncertainty mainly comes from the randomness of external factors such as wind speed and air flow. Therefore, the environmental uncertainty parameters consist of wind speed uncertainty and air flow uncertainty. For wind speed uncertainty, it is assumed that the wind speed measurement error follows a normal distribution. Thus, the variance of the wind speed measurement values can be calculated through historical wind speed measurement data, and this variance is taken as the wind speed uncertainty. For air flow uncertainty, based on the spatial distribution model of air flow established using Gaussian process regression, the variance of its air flow prediction is calculated, and this variance is taken as the air flow uncertainty. Finally, the variance representing wind speed uncertainty and the variance representing air flow uncertainty are added together to obtain the environmental uncertainty parameter. Finally, according to the mean, variance, and uncertainty propagation parameter, the confidence interval is determined: CI = μ ± z × σ total , where z is the quantile of the standard normal distribution and μ is the mean calculated by the Monte Carlo algorithm.

[0083] In this embodiment, the total uncertainty of trajectory prediction is quantified through the confidence interval based on Monte Carlo and uncertainty propagation, providing a reliable error range for the UAV trajectory prediction, thereby improving the accuracy of trajectory prediction.

[0084] After obtaining the accurate predicted flight trajectory of the suspicious UAV, the threat level of the UAV can be classified based on the predicted flight trajectory, and different countermeasures can be taken according to different threat levels. The specific steps include:

[0085] Obtain the UAV model and usage frequency of the suspicious UAV, and determine the threat level of the suspicious UAV according to the predicted flight trajectory, the UAV model, and the usage frequency;

[0086] When the threat level is a low threat level, execute the target tracking strategy and continuously monitor the flight path of the suspicious UAV through the camera device;

[0087] When the threat level is a medium threat level, execute the signal interference strategy and interfere with the suspicious UAV by activating the signal jammer;

[0088] When the threat level is a high threat level, execute the UAV countermeasure strategy and interfere with and intercept the suspicious UAV by dispatching countermeasure UAVs.

[0089] In this embodiment, the threat level is judged from three dimensions: the model, usage frequency, and predicted flight trajectory of the suspicious drone. First, it is judged whether both the model of the suspicious drone and the frequency of use of the drone are in the trusted list. If any one of the conditions is not met, the flight trend of the drone in the target airspace is judged according to the predicted flight trajectory, including judging whether the drone is approaching a preset area in the target airspace through the speed and direction in the flight trajectory, and judging whether there is an intention to attack a protected target in the target airspace through the altitude change trend in the flight trajectory.

[0090] In this embodiment, preferably, the target airspace is divided into a warning area and a countermeasure area. When it is judged that the predicted flight trajectory of the drone is at the edge of the warning area, it is considered to be a low threat level. At this time, the target tracking strategy is executed, that is, the flight path of the drone is continuously monitored through a camera device; when the flight path of the drone enters the warning area and has a tendency to fly towards the countermeasure area, it is considered to be a medium threat level. At this time, the signal interference strategy is executed, that is, a jamming signal is sent to the drone through a ground signal jammer to drive away the drone; when the drone has approached the countermeasure area and the predicted flight path is within the countermeasure area, it is considered to be a high threat level. At this time, the drone countermeasure strategy is executed, that is, countermeasure drones are dispatched to interfere with and intercept the suspicious drone.

[0091] When the countermeasure drones are dispatched, different countermeasure devices will be mounted according to the characteristics of the suspicious drone, including radio countermeasure equipment with different frequencies, directional antennas, omnidirectional antennas, strong light irradiation lamps, capture nets, etc. The countermeasure drones cooperate with the ground signal jammer to jointly counter the suspicious drone. In this embodiment, the interference signal adopts intelligent power control, adaptively adjusts the transmission power according to the target distance, and uses beamforming technology to improve the directivity. It should be noted that the determination of the threat level in this embodiment is only a preferred method, and other threat level determination methods can also be used, which are not specifically limited here.

[0092] Furthermore, when there are multiple suspicious drones with different threat levels, corresponding countermeasure strategies can be taken for these drones at the same time. When there are multiple suspicious drones with a high threat level, a strategy of dispatching multiple countermeasure drones for collaborative countermeasure can be adopted. When multiple countermeasure drones perform collaborative countermeasure, their flight trajectories are designed with the constraint goal of the shortest total time to reach the positions of all suspicious drones, so as to ensure that all suspicious drones can be countered in time. Through the threat level classification and countermeasure strategy in this embodiment, dynamic countermeasure and precise strike can be achieved, effectively avoiding accidental injury.

[0093] In a preferred embodiment, in order to improve the recognition accuracy of the drone, when analyzing and recognizing the target image collected by the imaging device, the quality scores of the images captured by each camera are calculated, and the target image is selected for recognition according to the quality scores. In addition, each camera has the functions of automatic exposure control and intelligent tracking to ensure comprehensive monitoring of the target airspace.

[0094] In another preferred embodiment, in order to improve the effect of drone countermeasures, a rule-based decision engine is established to handle scenarios of conventional threats. In this decision engine, different threat scenarios can be refined, and targeted countermeasure strategies can be inferred. At the same time, a reinforcement learning model is adopted to optimize the countermeasure strategies, and an optimal countermeasure strategy under different threat scenarios is stored through a scenario library. In addition, a safety assessment is set up to ensure that the countermeasure strategies will not cause collateral damage.

[0095] A drone countermeasure method based on image depth recognition provided in this embodiment realizes full coverage of the airspace through a hierarchical camera array, realizes precise recognition and positioning of the drone through a target detection algorithm and multi-source information fusion technology, and automatically evaluates the threat level and takes corresponding countermeasure measures through an intelligent decision-making mechanism, realizing precise strikes on suspicious drones. The present invention realizes automatic detection, tracking and countermeasures of suspicious drones through multi-level visual acquisition and deep learning algorithms, not only effectively improves the accuracy of drone recognition and positioning, but also realizes precise strikes on suspicious drones, effectively ensuring the safety of the airspace. The present invention provides an effective solution for airspace security protection in important areas and has good practicability and popularization.

[0096] Please refer to Figure 2 , based on the same inventive concept, a drone countermeasure system based on image depth recognition proposed in the second embodiment of the present invention includes:

[0097] A target monitoring module 10, configured to perform real-time monitoring on a target airspace through a pre-deployed imaging device and obtain monitoring images;

[0098] A target recognition module 20, configured to recognize the monitoring images according to a preset target recognition model, determine whether there is a drone in the monitoring images, and if there is a drone, analyze the monitoring images according to a feature matching algorithm to determine whether the drone is a suspicious drone;

[0099] A trajectory prediction module 30, configured to, if it is a suspicious drone, calculate the three-dimensional coordinates of the suspicious drone by using triangulation, and predict the flight trajectory of the suspicious drone according to the three-dimensional coordinates to obtain a predicted flight trajectory;

[0100] The decision-making countermeasure module 40 is configured to perform threat assessment on the suspicious UAV according to the predicted flight trajectory, determine the threat level of the suspicious UAV, and trigger corresponding countermeasure strategies according to the threat level.

[0101] The technical features and technical effects of the UAV countermeasure system based on image depth recognition proposed in the embodiments of the present invention are the same as those of the method proposed in the embodiments of the present invention, and will not be elaborated herein. Each module in the above-mentioned UAV countermeasure system based on image depth recognition can be implemented in whole or in part by software, hardware, and their combinations. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above-mentioned modules.

[0102] In summary, the embodiments of the present invention propose a UAV countermeasure method and system based on image depth recognition. The method performs real-time monitoring on the target airspace through a pre-deployed camera device and obtains monitoring images; identifies the monitoring images according to a preset target recognition model to determine whether there is a UAV in the monitoring images. If there is a UAV, analyze the monitoring images according to the feature matching algorithm to determine whether the UAV is a suspicious UAV; if it is a suspicious UAV, calculate the three-dimensional coordinates of the suspicious UAV using the triangulation method, and predict the flight trajectory of the suspicious UAV according to the three-dimensional coordinates to obtain a predicted flight trajectory; perform threat assessment on the suspicious UAV according to the predicted flight trajectory, determine the threat level of the suspicious UAV, and trigger corresponding countermeasure strategies according to the threat level. The present invention realizes the automatic detection, tracking, and countermeasure of suspicious UAVs through multi-level visual acquisition and deep learning algorithms, not only effectively improving the accuracy of UAV recognition and positioning, but also achieving precise strikes on suspicious UAVs, effectively ensuring the safety of the airspace. The present invention provides an effective solution for the airspace security protection of important areas and has good practicability and popularization.

[0103] Each embodiment in this specification is described in a progressive manner. For the parts that are the same or similar in each embodiment, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combinations of these technical features do not conflict, they should be considered as the scope recorded in this specification.

[0104] The above-described embodiments merely represent several preferred embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and substitutions can be made, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the protection scope of the claims described above.

Claims

1. A drone countermeasure method based on image depth recognition, characterized in that: include: Use pre-deployed cameras to monitor the target airspace in real time and obtain monitoring images; The surveillance image is identified according to the preset target recognition model to determine whether there is a drone in the surveillance image. If there is a drone, the surveillance image is analyzed according to the feature matching algorithm to determine whether the drone is a suspicious drone; If it is a suspicious drone, the three-dimensional coordinates of the suspicious drone are calculated using triangulation, and the flight trajectory of the suspicious drone is predicted based on the three-dimensional coordinates to obtain a predicted flight trajectory; A threat assessment is performed on the suspicious drone based on the predicted flight trajectory to determine the threat level of the suspicious drone, and a corresponding countermeasure strategy is triggered based on the threat level.

2. The method for countering drones based on image depth recognition according to claim 1, characterized in that: The camera device includes three groups, each group of which includes a number of cameras, wherein the first group of camera devices is used for low and medium altitude coverage monitoring, the second group of camera devices is used for high altitude coverage monitoring, and the third group of camera devices is used for lateral coverage monitoring; The field of view overlap rate of adjacent cameras in each group of camera devices is not less than the overlap rate threshold, and at least two cameras are not within the direct sunlight range in each time period.

3. The method for countering drones based on image depth recognition according to claim 1, characterized in that: The step of monitoring the target airspace in real time by using a pre-deployed camera device and acquiring monitoring images includes: Real-time monitoring of target airspace through pre-deployed cameras; When any camera in the camera device detects an object in the air, other adjacent cameras that are not disturbed by sunlight are called to perform collaborative observation to obtain multi-channel target images, which include multiple target images captured by multiple cameras.

4. The method for countering drones based on image depth recognition according to claim 3 is characterized in that: The steps of identifying the surveillance image according to the preset target recognition model and determining whether there is a drone in the surveillance image include: Preprocessing the multiple target images and inputting them into a target recognition model to determine whether the aerial object is a drone; The target recognition model uses a residual network as the backbone network, including a residual network, a feature pyramid network, a region proposal network and a feature extraction classification network connected in sequence; A channel attention module is provided in each residual block of the residual network, and a convolutional block attention module is provided after each output layer of the feature pyramid network; The residual network is used to extract feature maps of the multi-channel target images at different stages, the feature pyramid network is used to fuse the feature maps to generate enhanced feature maps, the region proposal network is used to anchor the enhanced feature maps to generate candidate regions, and the feature extraction separation network is used to extract and classify features of the candidate regions to obtain predicted target types.

5. The method for countering drones based on image depth recognition according to claim 1, characterized in that: The step of analyzing the monitoring image according to the feature matching algorithm to determine whether the drone is a suspicious drone includes: Extract the outline of the drone from the surveillance image according to the edge detection algorithm, and calculate the geometric features of the drone based on the extracted outline; According to the extracted contours, the color space conversion and histogram calculation are performed on the drone image in the monitoring image to obtain the color characteristics of the drone; The monitoring image is analyzed according to the optical flow method, the motion vector of the UAV is calculated, and the motion characteristics of the UAV are obtained according to the running vector; The cosine similarity is used to calculate the similarity between the geometric features and color features and the corresponding features in the feature database. The dynamic time warping algorithm is used to calculate the similarity between the motion features and the motion features in the feature database. The calculated similarities are weighted and summed to obtain the matching degree of the drone. According to the matching degree, it is determined whether the drone is a suspicious drone.

6. The method for countering drones based on image depth recognition according to claim 3 is characterized in that: The step of calculating the three-dimensional coordinates of the suspicious drone using triangulation includes: Perform target detection on any target image among the multiple target images to determine the position of the suspicious drone in the image, and estimate the initial height of the suspicious drone based on the proportional relationship between the position and the size of the reference object; A feature point detection algorithm is used to extract target feature points from multiple target images, and a triangulation method is used to calculate the three-dimensional coordinates of the suspicious drone according to the initial height, the target feature points and calibration parameters of each camera; The three-dimensional coordinates are smoothed by using Kalman filtering to obtain smooth three-dimensional coordinates.

7. The method for countering drones based on image depth recognition according to claim 1, characterized in that: The step of predicting the flight trajectory of the suspicious drone according to the three-dimensional coordinates to obtain the predicted flight trajectory includes: Acquire a three-dimensional coordinate sequence and weather environment parameters within a preset time period, wherein the weather environment parameters include wind speed, airflow intensity and air pressure; The three-dimensional coordinate sequence and weather environment parameters are used as input data and input into a pre-built trajectory prediction model to obtain the predicted flight trajectory of the suspicious UAV. The trajectory prediction model is constructed based on a long short-term memory network, and the confidence interval of the trajectory prediction model is calculated based on Monte Carlo uncertainty propagation.

8. The method for countering drones based on image depth recognition according to claim 1, characterized in that: The steps of performing threat assessment on the suspicious drone according to the predicted flight trajectory, determining the threat level of the suspicious drone, and triggering a corresponding countermeasure strategy according to the threat level include: Obtaining the drone model and usage frequency of the suspicious drone, and determining the threat level of the suspicious drone based on the predicted flight trajectory, the drone model, and the usage frequency; When the threat level is a low threat level, a target tracking strategy is executed to continuously monitor the flight path of the suspicious drone through a camera device; When the threat level is a medium threat level, a signal jammer is activated to jam the suspicious drone. When the threat level is a high threat level, the drone countermeasure strategy is executed to interfere with and intercept suspicious drones by dispatching countermeasure drones.

9. The method for countering drones based on image depth recognition according to claim 8, characterized in that: The step of triggering a corresponding countermeasure strategy according to the threat level further includes: When there are multiple suspicious drones with high threat levels, multiple counter-drones are started to coordinately counter the multiple suspicious drones, wherein the coordinated operation trajectory of the multiple counter-drones is set with the shortest time to reach the positions of all suspicious drones as the constraint objective function.

10. A drone countermeasure system based on image depth recognition, characterized in that: The system is applied to the method according to any one of claims 1 to 9, and the system comprises: The target monitoring module is used to monitor the target airspace in real time through the pre-deployed camera device and obtain monitoring images; The target recognition module is used to recognize the monitoring image according to the preset target recognition model, determine whether there is a drone in the monitoring image, and if there is a drone, analyze the monitoring image according to the feature matching algorithm to determine whether the drone is a suspicious drone; The trajectory prediction module is used to calculate the three-dimensional coordinates of the suspicious drone by triangulation if it is a suspicious drone, and predict the flight trajectory of the suspicious drone based on the three-dimensional coordinates to obtain a predicted flight trajectory; The decision-making countermeasure module is used to perform threat assessment on the suspicious drone according to the predicted flight trajectory, determine the threat level of the suspicious drone, and trigger a corresponding countermeasure strategy according to the threat level.

Images