Intelligent unmanned aerial vehicle intrusion detection and tracking method and system based on photoelectric fusion perception

Abstract

The invention discloses an intelligent unmanned aerial vehicle intrusion detection and tracking method and system based on photoelectric fusion perception, and relates to the technical field of target detection and tracking, and the method comprises the steps: obtaining a two-dimensional visible light image, a two-dimensional infrared image and three-dimensional point cloud data of a current frame; fusing the two-dimensional visible light image and the two-dimensional infrared image into a multi-modal image; target detection is carried out on the multi-modal image and the three-dimensional point cloud data respectively, fusion target detection is carried out on the multi-modal image and the three-dimensional point cloud data, and coordinates, bounding boxes and category information of a target in an image space, a three-dimensional space and after fusion are obtained; fusing bounding boxes output by three groups of target detection to obtain a current frame target optimal detection result and predict a target motion track; and if the threat suspicious unmanned aerial vehicle target exists, outputting a control signal to continuously track the threat suspicious unmanned aerial vehicle target. The method is low in software and hardware cost, highly automatic in detection, free of long-term guarding of personnel and capable of meeting the actual intelligent anti-unmanned-aerial-vehicle security and protection requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection and tracking, and in particular to an intelligent unmanned aerial vehicle intrusion detection and tracking method and system based on photoelectric fusion perception. Background Art

[0002] Unmanned aerial vehicles, or "drones", are unmanned aircraft that are controlled by radio remote control equipment and self-contained program control devices, or are fully or intermittently autonomously operated by onboard computers. They have the advantages of small size, low cost, easy use, and low environmental requirements. With the development of drone technology and the continuous decline in costs, drones have gradually become popular in all walks of life and are widely used in aerial photography, agricultural plant protection, logistics and transportation, etc. However, this has led to frequent intrusions of drones into restricted areas, posing a threat to public safety. Drones can carry explosives, toxins or other dangerous substances and are used as criminal tools for malicious attacks and destruction. The cameras and sensors equipped with drones can violate the privacy of individuals and institutions and may be used for illegal surveillance, snooping or espionage. The intrusion of drones may interfere with the operation of aircraft, hinder normal aviation activities and pose a threat to the safety of pilots and passengers. In addition, improper operation, loss of control or crash of drones may also cause property damage, especially in densely populated areas, near buildings or around important facilities. Therefore, anti-drone intrusion detection and target tracking are important needs of today's society.

[0003] Target detection and tracking is a hot research and development topic at present, and its core technology is to extract and utilize target feature information. UAVs have many significant features. First, UAVs are similar to manned aircraft in that they have the same appearance features such as lift structure, power source, and landing gear. Secondly, the power source of UAVs, such as batteries, motors, and speed regulators, generates heat when working, which is more significant in the natural background. Furthermore, UAVs often need to communicate with the outside world and will actively send radio signals to the outside world. Finally, UAVs have a certain reflectivity to electromagnetic waves.

[0004] According to the above characteristics, the existing anti-UAV detection and tracking solutions mainly include conventional visual solutions, radio detection and interception solutions, and electromagnetic wave radar scanning detection solutions. All of the above methods have many imperfections. Among them, the conventional visual solution is relatively simple to implement, but the effect is poor under complex environmental conditions (rainstorm, smoke, multiple buildings or jungle background) or when the UAV takes stealth measures (infrared camouflage, environmental camouflage). As a passive detection solution, the radio detection and interception solution can only take effect when the UAV actively sends radio signals, which has certain limitations; at the same time, this solution is difficult to deal with radio countermeasures such as communication encryption, use of special frequency bands, and anti-detection interference. The electromagnetic wave radar scanning detection solution has high accuracy and wide range, but the software and hardware costs are high. In order to achieve a larger coverage range, it is usually equipped with a larger antenna, which is large in size and high in power consumption and can only be deployed in fixed sites. Common radars are difficult to intuitively distinguish target categories, such as distinguishing between birds and drones, and often require cooperation with other solutions or manual secondary confirmation. Summary of the invention

[0005] In view of the above problems and technical needs, the inventors have proposed an intelligent drone intrusion detection and tracking method and system based on optoelectronic fusion perception, which can achieve high-precision, automatic, intelligent, local real-time anti-drone detection and tracking, and can be applied to the real-time detection and tracking of drones in automated intelligent perimeter security in reality. The technical solution of the present invention is as follows:

[0006] In the first aspect, the present application provides an intelligent drone intrusion detection and tracking method based on optoelectronic fusion perception, comprising the following steps:

[0007] Obtain the two-dimensional visible light image, two-dimensional infrared image and three-dimensional point cloud data of the current frame;

[0008] Fusing a two-dimensional visible light image with a two-dimensional infrared image into a multimodal image;

[0009] Perform target detection on multimodal images and 3D point cloud data separately, and perform fused target detection on the two to obtain the coordinates, bounding box and category information of the target in image space, 3D space and after fusion;

[0010] The bounding boxes of the three sets of target detection outputs are fused to obtain the optimal detection result of the target in the current frame and predict the target motion trajectory;

[0011] If there is a threatening suspicious UAV target, a control signal is output to continuously track the threatening suspicious UAV target.

[0012] A further technical solution is to fuse the two-dimensional visible light image and the two-dimensional infrared image into a multimodal image, including:

[0013] The two-dimensional visible light image and the two-dimensional infrared image are decomposed into low-frequency components and high-frequency components respectively through wavelet transform;

[0014] The low frequency components of the two-dimensional visible light image and the two-dimensional infrared image are added through the channel, and the high frequency components of the two-dimensional visible light image and the two-dimensional infrared image are added through the channel;

[0015] The added high and low frequency components enter the multi-head self-attention MHSA layer for feature extraction and pass through the MLP layer to compress the channel;

[0016] The output feature map of each channel enters PCNN, and the pixels with large ignition times are selected as the pixels of the multimodal image, and then the multimodal image is obtained through inverse wavelet transform.

[0017] A further technical solution is to perform target detection on multimodal images, including:

[0018] The multimodal image is input into the YOLO backbone network, and after multiple downsampling, feature maps of multiple scales are extracted;

[0019] The multi-scale feature map input uses the feature fusion pyramid network FPN of the encoding layer-MHSA layer;

[0020] The output feature maps of each scale are input into the multi-scale feature fusion MFF module and output the fused feature map, which is then passed through the YOLO algorithm detection head for target detection, and finally the coordinates, bounding box and category information of the target in the image space are output.

[0021] Its further technical solution is that before the multi-scale feature map is input into the network, it is first rasterized into a P×P×C patch to obtain a two-dimensional vector sequence Patch Query, where P is the scale and C is the number of channels;

[0022] The network operations of the encoding layer include:

[0023] Q2=XW q ,K2=XW k ,V2=XW v ;

[0024]

[0025] Q out =MHSA(LN(X))+X;

[0026] X out =MLP(LN(Q out ))+Q out ;

[0027] Where X is the element in Patch Query, W q , Wk , W v is the linear transformation matrix obtained through training, d k is the dimension of K, MHSA() represents the multi-head self-attention operation, MLP() represents the compression operation, and LN() represents the normalization operation.

[0028] A further technical solution is to use trainable weight parameters in the MFF module to fuse the correct feature vectors of pixels with information conflicts in the output feature maps of each scale. The formula is:

[0029]

[0030] In the formula, the subscript ij represents a pixel in the feature map. It represents the feature vector at the pixel ij after the nth layer is fused and scaled to the lth layer. represents the fused feature vector output at the pixel point ij in the lth layer, α ij , β ij , γ ij They are the trainable weight parameters of different levels.

[0031] Its further technical solution is to perform target detection on 3D point cloud data, including:

[0032] The 3D point cloud data is input into the 3D feature extraction network after segmentation screening, point cloud filtering, and voxelization preprocessing operations;

[0033] On the one hand, the extracted voxel features are compressed in the Z-axis direction to output a two-dimensional BEV feature map; on the other hand, the empty voxels that do not contain point clouds in the extracted voxel features are eliminated, and the remaining voxels are compressed into a sequence and encoded with the voxel position information;

[0034] The output feature encoding matrix is ​​used as the query matrix Q3, and the two-dimensional BEV feature map is used as the key matrix K3 and the value matrix V3 to input into the decoding layer;

[0035] The decoded feature input is detected by the feed-forward neural network FFN to perform coordinate bounding box regression and target category prediction, and finally output the coordinates, bounding box and category information of the target in three-dimensional space.

[0036] Its further technical solution is to perform fusion target detection on multimodal images and 3D point cloud data, including:

[0037] Extracting 2D BEV feature maps from the target detection process of 3D point cloud data;

[0038] At the same time, from the target detection process of multimodal images, the fusion feature map output by the MFF module is extracted and encoded with the pixel position information;

[0039] The output feature encoding matrix is ​​used as the query matrix Q4, and the two-dimensional BEV feature map is used as the key matrix K3 and the value matrix V3 to input into the decoding layer;

[0040] The decoded feature input is detected by the feed-forward neural network FFN to perform coordinate bounding box regression and target category prediction, and finally output the target coordinates, bounding box and category information after two-dimensional and three-dimensional fusion.

[0041] A further technical solution is that the network operation of the decoding layer includes:

[0042]

[0043] Q out =LN(MHSA(Q,K,V)+Q);

[0044] Q final =LN(FFN(Q out )+Q out );

[0045] FFN(Q out )=ReLU(Q out W1+B1)W2+B2;

[0046] Where, d k is the dimension of K, LN() represents the normalization operation, W1 and W2 are the linear transformation matrices in the feedforward neural network, B1 and B2 are the bias matrices in the feedforward neural network, Q final Decoded features output by the decoding layer.

[0047] A further technical solution is to fuse the bounding boxes of the three sets of target detection outputs, including:

[0048] The bounding boxes of the three sets of target detection outputs are projected into a custom world coordinate system, the intersection over union (IoU) ratio is used as the threshold to determine the bounding boxes to be fused, and the confidence of the bounding boxes is used as the weight to perform pairwise weighted fusion on the bounding boxes to be fused until the optimal detection result output is obtained.

[0049] In the second aspect, the present application provides an intelligent drone intrusion detection and tracking system based on optoelectronic fusion perception, including:

[0050] The data acquisition device includes a wide-angle visible light camera, a zoom visible light camera, a zoom infrared camera and a laser radar; the zoom visible light camera, the zoom infrared camera and the laser radar are used to collect two-dimensional visible light images, two-dimensional infrared images and three-dimensional point cloud data of the target to be measured in the monitoring area, and the wide-angle visible light camera is used to provide a global view when detecting and tracking the target;

[0051] A computing module, used to implement the intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception as in the first aspect;

[0052] The motion gimbal includes two mechanical rotating structures controlled by a computing module, which are responsible for the pitch and yaw movements of the data acquisition device according to the control signals output by the computing module.

[0053] The beneficial technical effects of the present invention are:

[0054] This application proposes an intelligent drone intrusion detection and tracking method and system based on optoelectronic fusion perception. The system introduces a small-sized, low-power laser radar detection based on visible light and infrared fusion vision, connects the high-precision three-dimensional point cloud data detected by laser echo with the two-dimensional image, establishes an optoelectronic fusion perception digital detection space, and cooperates with deep learning algorithms and motion gimbals to achieve local high-precision real-time automatic drone detection and tracking. Compared with the existing scheme, this system uses mature visible light and infrared cameras and laser radars, and takes into account the excellent characteristics of visible light camera and infrared camera imaging schemes and electromagnetic wave radar scanning schemes at a lower cost, overcomes complex environments and has high detection accuracy, and the missed detection rate of threatening drone targets is less than 1%, and the tracking accuracy is as high as 98%. In addition, the system integrates a computing module, which can realize the edge deployment of modern neural networks, AI-enabled anti-drone detection and tracking, avoid large amounts of data transmission and dependence on host computing power, and can achieve flexible deployment and real-time automatic detection. It can be remote or unmanned, meeting the actual intelligent anti-drone security needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a schematic diagram of the structure of an intelligent UAV intrusion detection and tracking system based on optoelectronic fusion perception provided in this application, wherein (a) is a front view and (b) is a side view.

[0056] Figure 2 This is a schematic diagram of the camera and lidar field of view coverage provided by this application.

[0057] Figure 3 This is a schematic diagram of the non-repetitive scanning laser radar scanning principle provided by this application.

[0058] Figure 4 This is the workflow diagram of the intelligent drone intrusion detection and tracking system provided by this application.

[0059] Figure 5 It is a schematic diagram of the relative relationship of the system coordinate system provided in this application.

[0060] Figure 6 This is the intelligent drone intrusion detection and tracking network algorithm diagram based on optoelectronic fusion perception provided by this application.

[0061] Figure 7 This is a schematic diagram of the network structure of the encoding layer Transformer Encoder provided in this application.

[0062] Figure 8 This is a schematic diagram of the network structure of the decoding layer Transformer Decoder provided in this application. DETAILED DESCRIPTION

[0063] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0064] The present application provides an intelligent drone intrusion detection and tracking system based on optoelectronic fusion perception, which includes: a data acquisition device for acquiring two-dimensional images and three-dimensional point clouds, a motion gimbal for target tracking, and a computing module for gimbal control, data processing, and deep learning operations. The data acquisition device is installed on the motion gimbal, which has a good detection field of view and range of motion. The computing module is arranged in the base below the motion gimbal to facilitate local real-time data processing and rapid response. Figure 1 The structures shown in (a) and (b) are described in detail, and the principles of each component and the overall workflow of the system are introduced.

[0065] The data acquisition device includes a wide-angle visible light camera 1, a zoom visible light camera 2, a zoom infrared camera 3 and a laser radar 4. The high-resolution zoom visible light camera 2 is used to collect visible light images of the target to be measured in the monitoring area, the zoom infrared camera 3 is used to collect infrared heat source characteristic maps (i.e., two-dimensional infrared images) of the target to be measured in the monitoring area, and the laser radar 4 is used to collect high-precision three-dimensional point cloud data of the target to be measured in the monitoring area. The wide-angle visible light camera 1 is used to provide a global view when detecting and tracking targets. Figure 2 As shown, the image field of view area covered by the zoom visible light camera 2 and the zoom infrared camera 3 can refer to the small rectangular area a, the image field of view area covered by the wide-angle visible light camera 1 can refer to the large rectangular area b, and the point cloud field of view area covered by the lidar 4 can refer to the circular area c.

[0066] In order to adapt to complex environmental conditions or solve the problem that the UAV has poor stealth effect, this embodiment uses a 1550nm wavelength multi-line prism non-repetitive scanning laser radar as a supplementary detection method. The scanning principle is as follows: Figure 3As shown in the figure, when the light beam enters the radar (point O) against the optical axis Z and exits from prism 1, as the prism rotates, the starting point of the deflection vector θ1 remains unchanged, and the end moves in a circle with a radius of θ1. The light beam continues to enter prism 2, and the deflection vector θ2 continues to be superimposed on θ1. The total deflection vector can be regarded as the vector sum of θ1 and θ2. By adjusting the rotation speed of the two prisms, non-repetitive scanning within a certain period of time can be achieved. At the same time, by optimizing the values ​​of θ1 and θ2, full coverage and seamless scanning can be achieved within the field of view. Therefore, compared with the existing mechanical repetitive scanning laser radar, the prism-type non-repetitive scanning laser radar can significantly improve the detection effect of long-distance, small-volume, and high-speed drone targets. Compared with the traditional 905nm, the 1550nm wavelength laser can work at a higher luminous power while ensuring the safety of human eyes, thereby achieving a longer detection distance and strong anti-interference ability. Moreover, the light-emitting surface of the 1550nm fiber laser is only about 10 microns, and the far-field divergence angle is small, which has a better measurement effect on long-distance UAV targets.

[0067] The computing module 5 includes an embedded edge computing module (SOM), a gimbal controller and a power supply. The SOM integrates a modular end-to-end multi-modal drone target detection and tracking network algorithm, which can run modern deep learning networks in parallel and process data from multiple high-resolution sensors. It sends control signals to the gimbal controller based on the processing results to realize the detection and tracking software functions, thereby improving the system's detection real-time performance, result accuracy and data security. At the same time, it reduces external wiring and reduces the difficulty of system deployment.

[0068] The motion gimbal includes two mechanical rotating structures 6 and 7 controlled by the computing module 5. The mechanical rotating structure 6 is responsible for the 360-degree movement of the data acquisition device in the yaw direction according to the control signal output by the computing module 5, and the mechanical rotating structure 7 is responsible for the movement of the data acquisition device in the pitch direction. The device is used to continuously track threatening UAV targets. When the target is about to leave the center area of ​​the field of view, the gimbal can automatically adjust the angle according to the fusion data collected in real time, so that the tracked target remains in the center area of ​​the field of view for a long time.

[0069] The overall workflow of the above system is as follows: Figure 4As shown in the figure, after the system is started, the data acquisition device starts to collect two-dimensional visible light images, two-dimensional infrared images and three-dimensional point cloud data. The collected data is fused and processed by the computing module and target detection is performed. If there is a suspicious drone target in the current monitoring area, the computing module reports "Alert" type information to the host computer, including the target two-dimensional image and spatial coordinates, and further calculates and predicts the motion trajectory of the detected suspicious drone target to determine whether the suspicious drone target has threatening behaviors such as intrusion, harassment, and retention. If there is a threatening suspicious drone target, the computing module reports "Warning" type information to the host computer, including the target two-dimensional image, spatial coordinates, motion trajectory and threat confidence. The pan-tilt controller receives the control signal sent by the SOM of the computing module, controls the motion pan-tilt to continuously track the threatening drone target, and the zoom visible light camera and zoom infrared camera of the data acquisition device jointly zoom to focus the field of view on the threatening drone target. At the same time, the wide-angle visible light camera provides a temporary global field of view. The system will continue to track the threatening drone target until it leaves the monitoring area, disappears, or the computing module receives a termination command manually judged by the host computer.

[0070] Before the system is put into use, the camera and radar need to be jointly calibrated to build a custom world coordinate system (Xw, Yw, Zw) for detection and tracking, such as Figure 5 The coordinate system Ow-XwYwZw in the middle shows the three-dimensional spatial coordinates of the spatial feature points of a certain frame detection in this coordinate system. With each camera as the center, establish the camera coordinate system (Xc, Yc, Zc) respectively, as shown in Figure 5 As shown in the coordinate system Oc-XcYcZc, the transformation relationship between the two-dimensional coordinates of the image feature points in the two-dimensional image coordinate system (u, v), the three-dimensional coordinates of the spatial feature points in the camera coordinate system (Xc, Yc, Zc) and the three-dimensional coordinates of the world coordinate system (Xw, Yw, Zw) is as follows:

[0071]

[0072] Where: K is the camera intrinsic parameter matrix; R is the camera extrinsic rotation matrix; t is the camera extrinsic translation matrix.

[0073] The initial data collected by the laser radar is (L, δ, ε), where L is the scanning distance of the beam; δ is the pitch scanning angle of the beam; ε is the horizontal scanning angle of the beam. With the radar as the center, establish the radar coordinate system (X l ,Y l ,Z l ),like Figure 5 Coordinate system O l -X l Y l Z lAs shown, the spatial feature points are in the radar coordinate system (X l ,Y l ,Z l ) are as follows:

[0074]

[0075] Where: (s,β,H o ,V o ,D o ) is the radar internal parameter, s is the scale factor of the beam scanning distance, β represents the deviation of the horizontal scanning angle of the beam, and H o Indicates the horizontal offset of the beam origin, V o Indicates the vertical offset of the beam origin, D o is the offset between the beam origin and the rotation axis. The two-dimensional coordinates of the image feature point in the two-dimensional image coordinate system (u, v), the spatial feature point in the radar coordinate system (X l ,Y l ,Z l ) and the three-dimensional coordinates of the world coordinate system (Xw, Yw, Zw) are as follows:

[0076]

[0077] Where: is the rotation matrix from the lidar to the camera, is the translation matrix from lidar to camera.

[0078] The external parameters from the camera to the world coordinate system (R, t) and the external parameters from the laser radar to the camera It can be obtained by joint calibration calculation based on non-chessboard two-dimensional targets. The non-chessboard two-dimensional target calibration plate uses ArUco tags to determine the relationship between the calibration plate in the radar coordinate system and the camera coordinate system, extracts the coordinates of the calibration plate corners through a random sampling consistency algorithm and establishes a corresponding relationship equation, and finally uses the Kabsh algorithm to solve the registration parameters.

[0079] This application also provides an intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception. The method is a modular end-to-end multi-modal UAV target detection and tracking network algorithm integrated in SOM. The network structure is as follows: Figure 6 As shown, the specific steps include:

[0080] Step 1: Get the current frame of 2D visible light image, 2D infrared image and 3D point cloud data. When the data acquisition device completes the acquisition of a frame of 2D visible light image, 2D infrared image and 3D point cloud data, the computing module starts to perform network computing on each frame of data to be fused transmitted through the internal data cable.

[0081] Step 2: Input the 2D visible light image and the 2D infrared image into WA-PCNN (Wavelet Attention-Pulse Coupled Neural Network) to fuse them into a multimodal image. The visible light image and the infrared image come from different modalities and provide scene information from different aspects. The visible light image captures color texture, and the infrared image captures thermal radiation. The combination of their information can present the inherent characteristics of almost all objects. Therefore, for the target detection task, this fused multimodal image has more information value than independent multiple single-modal images. The network processing flow is as follows: First, the 2D visible light image and the 2D infrared image are decomposed into low-frequency components (LL) and high-frequency components (LH, HL, HH) by wavelet transform. Then, the low-frequency components of the 2D visible light image and the 2D infrared image are concatenated through channels, and the high-frequency components of the 2D visible light image and the 2D infrared image are concatenated through channels. The high and low frequency components after addition enter the MHSA (multi-head self-attention) layer for feature extraction and compress the channels through the MLP layer. The formula for this part of the process is as follows:

[0082] X Low =Concat(LL vis ,LL ir )

[0083] X High =Concat(LH vis ,HL vis ,HH vis ,LH ir ,HL ir ,HH ir )

[0084] Q1=XW q ,K1=XW k ,V1=XW v ,R=r h +r w

[0085]

[0086] Out = MLP (MHSA (XW q ,XW k ,XW v ,R))

[0087] Where X is the wavelet transform component image including two channels X Low With X High , the subscript ir corresponds to the two-dimensional infrared image data, the subscript vis corresponds to the two-dimensional visible light image data, W q , Wk , W v is the trained linear transformation matrix (1×1), which is used to decompose X pixel-wise into three different vectors: query matrix Q1, key matrix K1 and value matrix V1. R is the pixel position encoding matrix, r h is the position information of the pixel at the image height, r w is the position information of the pixel in the image width, and Out is the output of this part of the network. Finally, the output feature map of each channel enters the PCNN. The network uses the pixel value of the image as the excitation value of the neuron. The number of neurons in the network is consistent with the number of image pixels. Each neuron has two output states: ignition and non-ignition. The sum of the ignition times of each neuron will constitute an ignition map. The ignition times are used as the activity level. Pixels with large ignition times are selected from different channel feature maps as pixels of the multimodal image. The pixels of the two channels are then inversely transformed to obtain the multimodal image.

[0088] Step 3: Detect the target in the multimodal image through the YOLO Former target detection network to obtain the coordinates, bounding box and category information of the target in the image space. Modern image target detection algorithms based on deep learning neural networks can have both accuracy and real-time performance. Single-stage detection algorithms represented by YOLOv8 can directly predict the target bounding box and its category from the image with only one convolutional neural network. The YOLOv8 algorithm greatly compresses the amount of calculation while ensuring the detection accuracy, and can be lightweight and deployed on a variety of computing platforms including mobile phones. The network processing flow is as follows: First, the multimodal image output by WA-PCNN is input into the YOLO backbone network. After multiple downsampling, feature maps of three scales are extracted. Since the scale of the drone target varies with factors such as the type of aircraft and distance, the multi-scale feature map can meet the detection requirements of multi-scale targets. Subsequently, the three-scale feature maps are input into the feature fusion pyramid network FPN using the Transformer Encoder (encoding layer)-MHSA layer. Multi-scale image data needs to be patterned before input. To reduce the amount of calculation, this operation rasterizes the image into a P×P×C (P is the scale, C is the number of channels) patch, thereby converting the three-dimensional data into a two-dimensional vector sequence Patch Query. Figure 7 As shown, the network operations of the encoding layer include:

[0089] Q2=XW q ,K2=XW k ,V2=XW v

[0090]

[0091] Q out=MHSA(LN(X))+X

[0092] X out =MLP(LN(Q out ))+Q out

[0093] Where X is the element in Patch Query, d k is the dimension of K2, divided by For normalization; MHSA() represents multi-head self-attention operation, MLP() represents compression operation, and LN() represents normalization operation. The feature maps of the three scales pass through the FPN network and correspond to the three outputs X out , three types of X out Conflicting information is inevitable in the output feature map. Therefore, this application designs a multi-scale feature fusion MFF module, which uses trainable weight parameters to fuse the correct feature vectors of pixels with information conflicts in the output feature map of each scale. The formula is as follows:

[0094]

[0095] In the formula, the subscript ij represents a pixel in the feature map. It represents the feature vector at the pixel ij after the nth layer is fused and scaled to the lth layer. represents the fused feature vector output at the pixel point ij in the lth layer, α ij , β ij , γ ij are trainable weight parameters at different levels, and n=3 in this embodiment. The output feature maps of each scale are input into the above-mentioned MFF module and output the fused feature map, which is then passed through the YOLO algorithm detection head for target detection, and finally the coordinates, bounding box and category information of the target in the image space are output. In this process, the characteristic information extracted by the YOLO Former network from the multimodal image is used for fusion target detection with the three-dimensional point cloud features, see step 5 for details.

[0096] Step 4: Perform target detection on the three-dimensional point cloud data to obtain the coordinates, bounding box and category information of the target in the three-dimensional space. The three-dimensional point cloud data is input into the VoxelFormer target detection network after segmentation screening, point cloud filtering, and voxelization preprocessing operations. The network processing flow is as follows: First, the voxelized three-dimensional point cloud data is input into the 3D feature extraction network. The extracted voxel features are compressed in the Z-axis direction on the one hand, and the two-dimensional BEV (Bird's Eye View) feature map is output; on the other hand, the empty voxels that do not contain point clouds in the extracted voxel features are eliminated, and the remaining voxels are compressed into a sequence query and encoded with the voxel position information, i.e., Q pos =Q+PQ , where P Q is the position encoding matrix. Then, the output feature encoding matrix Q pos As the query matrix Q3, the two-dimensional BEV feature map is input into the Transformer Decoder (decoding layer) as the key matrix K3 and the value matrix V3. Figure 8 As shown, the network operations of the encoding layer include:

[0097]

[0098] Q out =LN(MHSA(Q3,K3,V3)+Q3);

[0099] Q final =LN(FFN(Q out )+Q out );

[0100] FFN(Q out )=ReLU(Q out W1+B1)W2+B2;

[0101] Where W1 and W2 are linear transformation matrices in the feedforward neural network FFN, B1 and B2 are bias matrices in FFN, ReLU() represents the activation function, and Q final The decoded features are output by the decoding layer. The decoded features are input into the detection FFN for coordinate bounding box regression and target category prediction, and finally the coordinates, bounding box and category information of the target in three-dimensional space are output.

[0102] Step 5: Use the Fusion Former target detection network to fuse the multimodal image and three-dimensional point cloud data for target detection, and obtain the coordinates, bounding box and category information of the fused target. The network processing flow is as follows: Extract the two-dimensional BEV feature map from the target detection process of the three-dimensional point cloud data. At the same time, extract the fused feature map output by the MFF module from the target detection process of the multimodal image and encode it with the pixel position information. The position encoding process is consistent with the MHSA process in WA-PCNN. Then, the output feature encoding matrix is ​​used as the query matrix Q4, and the two-dimensional BEV feature map is used as the key matrix K3 and the value matrix V3 to input into the Transformer Decoder. Combined with Figure 8As shown in the figure, the coding layer network operation of (Q4, K3, V3) is consistent with Voxel Former, so it will not be repeated here. Finally, the decoded features are input into the detection feed-forward neural network FFN for coordinate bounding box regression and target category prediction, and finally the target coordinates, bounding box and category information after 2D and 3D fusion are output. It is worth noting that the YOLO Former, Fusion Former and Voxel Former target detection networks run in parallel.

[0103] Step 6: Fuse the bounding boxes output by the YOLO Former, Fusion Former, and Voxel Former target detection networks to obtain the optimal detection result for the current frame. The main operation is to first project the bounding boxes output by the three sets of target detection into a custom world coordinate system (Xw, Yw, Zw). For coordinate system conversion, refer to the camera and radar joint calibration content. Then, the bounding box to be fused is determined using the intersection over union (IoU) ratio as the threshold, and the bounding boxes to be fused are weighted fused pairwise using the confidence of the bounding box as the weight until the optimal detection result is output.

[0104] Step 7: Predict the target trajectory. If there is a threatening suspicious drone target, output a control signal to continuously track the threatening suspicious drone target. The tracking part uses the Kalman filter to update the trajectory. That is, according to the detection information of the past frames, the Kalman filter can update the target's bounding box and speed. The target category is determined by the majority in the past 3 frames, and the target confidence is determined by the maximum value in the past 3 frames.

[0105] In this embodiment, the drone target detection and tracking tasks are modularized, wherein the WA-PCNN module using wavelet transform and multi-head attention is used for visible light and infrared image fusion, the YOLO Former module is used for two-dimensional image target detection, the Voxel Former module is used for three-dimensional voxel target detection, the Fusion Former module is used for image feature-guided point cloud BEV feature map target detection, and the Kalman filter is used for post-detection multi-frame tracking. This set of algorithms efficiently utilizes the multimodal fusion information of visible light, infrared, and laser three-dimensional point clouds. After hardware acceleration of the computing module, the processing time of a single frame of the algorithm is within 40ms. At the same time, the missed detection rate of threatening drone targets is less than 1%, and the tracking accuracy is as high as 98%, meeting the high real-time and high accuracy requirements of security detection.

[0106] The detection process of this system is closed-loop, and the calculation, processing and detection results of the whole process can be sent to the interactive interface of the host computer by wireless or wired means for real-time display and storage. This method mainly relies on pre-calibrated cameras and laser radars, and the high-precision two-dimensional images and three-dimensional point cloud data they perceive are the basis for the algorithm and function realization of this system. For cameras of other spectra or other types of laser radars, as long as they are accurately calibrated and can accurately perceive images or point cloud data in two-dimensional or three-dimensional space, target detection and tracking can be performed through the computing module. Therefore, in theory, this system can detect and track a class of targets with certain appearance structure characteristics and motion trajectory characteristics in the space perceived by the camera and laser radar. Its characteristics are trained by deep learning neural networks and deployed on the computing module, and this type of target can be detected and tracked. Therefore, this method is theoretically suitable for most modern security tasks of monitoring moving targets invading restricted areas. At the same time, this system is small in size, low in cost, does not require additional computer connection, has high installation location flexibility, only needs to pre-set some parameters, and only needs to cover the monitoring area with the camera and laser radar field of view on site, power on and simple self-check calibration before it can be put into use.

[0107] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0108] The above is only a preferred embodiment of the present application, and the present invention is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included in the protection scope of the present invention.

Claims

1. An intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception, characterized in that: The method comprises: Obtain the two-dimensional visible light image, two-dimensional infrared image and three-dimensional point cloud data of the current frame; fusing the two-dimensional visible light image and the two-dimensional infrared image into a multimodal image; Performing target detection on the multimodal image and the three-dimensional point cloud data respectively, and performing fusion target detection on the two, to obtain coordinates, bounding boxes and category information of the target in the image space, three-dimensional space and after fusion; The bounding boxes of the three sets of target detection outputs are fused to obtain the optimal detection result of the target in the current frame and predict the target motion trajectory; If there is a threatening suspicious UAV target, a control signal is output to continuously track the threatening suspicious UAV target.

2. The intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception according to claim 1 is characterized in that: The two-dimensional visible light image and the two-dimensional infrared image are fused into a multimodal image, comprising: The two-dimensional visible light image and the two-dimensional infrared image are respectively decomposed into a low-frequency component and a high-frequency component by wavelet transformation; The low-frequency components of the two-dimensional visible light image and the two-dimensional infrared image are added through a channel, and the high-frequency components of the two-dimensional visible light image and the two-dimensional infrared image are added through a channel; The added high and low frequency components enter the multi-head self-attention MHSA layer for feature extraction and pass through the MLP layer to compress the channel; The output feature map of each channel enters PCNN, and the pixels with large ignition times are selected as the pixels of the multimodal image, and then the multimodal image is obtained through inverse wavelet transform.

3. The intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception according to claim 1 is characterized in that: Performing target detection on the multimodal image includes: The multimodal image is input into the YOLO backbone network, and feature maps of multiple scales are extracted after multiple downsampling; The multi-scale feature map input uses the feature fusion pyramid network FPN of the encoding layer-MHSA layer; The output feature maps of each scale are input into the multi-scale feature fusion MFF module and the fused feature map is output. The YOLO algorithm detection head is then used for target detection, and finally the coordinates, bounding box and category information of the target in the image space are output.

4. The intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception according to claim 3 is characterized in that: Before the multi-scale feature map is input into the network, it is first rasterized into a P×P×C patch to obtain a two-dimensional vector sequence Patch Query, where P is the scale and C is the number of channels; The network operation of the coding layer include: <h2 style=";text-align:left;direction:ltr">Q2=XW<h2 style=";text-align:left;direction:ltr"> q <h2 style=";text-align:left;direction:ltr"> K2=XW<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> V2=XW<h2 style=";text-align:left;direction:ltr"> v <h2 style=";text-align:left;direction:ltr"> ; Q out =MHSA(LN(X))+X; X out =MLP(LN(Q out ))+Q out ; Where X is the element in the Patch Query, W q , W k , W v is the linear transformation matrix obtained through training, d k is the dimension of K, MHSA() represents the multi-head self-attention operation, MLP() represents the compression operation, and LN() represents the normalization operation.

5. The intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception according to claim 3 is characterized in that: In the MFF module, trainable weight parameters are used to fuse the correct feature vectors of pixels with information conflicts in the output feature maps of each scale. The formula is: In the formula, the subscript ij represents a pixel in the feature map. It represents the feature vector at the pixel ij after the nth layer is fused and scaled to the lth layer. represents the fused feature vector output at the pixel point ij in the lth layer, α ij , β ij , γ ij They are the trainable weight parameters of different levels.

6. The intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception according to claim 1 is characterized in that: Performing target detection on the three-dimensional point cloud data includes: The three-dimensional point cloud data is sequentially subjected to segmentation screening, point cloud filtering, and voxelization preprocessing operations and then input into a 3D feature extraction network; On the one hand, the extracted voxel features are compressed in the Z-axis direction to output a two-dimensional BEV feature map; on the other hand, the empty voxels that do not contain point clouds in the extracted voxel features are eliminated, and the remaining voxels are compressed into a sequence and encoded with the voxel position information; The output feature encoding matrix is ​​used as the query matrix Q3, and the two-dimensional BEV feature map is used as the key matrix K3 and the value matrix V3 to input into the decoding layer; The decoded feature input is detected by the feed-forward neural network FFN to perform coordinate bounding box regression and target category prediction, and finally output the coordinates, bounding box and category information of the target in three-dimensional space.

7. The intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception according to claim 1 is characterized in that: Performing fusion target detection on the multimodal image and the three-dimensional point cloud data, including: Extracting a two-dimensional BEV feature map from a target detection process on the three-dimensional point cloud data; At the same time, from the target detection process of the multimodal image, a fusion feature map output by the MFF module is extracted and encoded with pixel position information; The output feature encoding matrix is ​​used as the query matrix Q4, and the two-dimensional BEV feature map is used as the key matrix K3 and the value matrix V3 to input into the decoding layer; The decoded feature input is detected by the feed-forward neural network FFN to perform coordinate bounding box regression and target category prediction, and finally output the target coordinates, bounding box and category information after two-dimensional and three-dimensional fusion.

8. The intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception according to claim 6 or 7 is characterized in that: The network operations of the decoding layer include: Q out =LN(MHSA(Q,K,V)+Q); Q final =LN(FFN(Q out )+Q out ); <h2 style=";text-align:left;direction:ltr">FFN(Q<h2 style=";text-align:left;direction:ltr"> out <h2 style=";text-align:left;direction:ltr"> ReLU(Q)<h2 style=";text-align:left;direction:ltr"> out <h2 style=";text-align:left;direction:ltr"> W1+B1)W2+B2; Where, d k is the dimension of K, LN() represents the normalization operation, W1 and W2 are the linear transformation matrices in the feedforward neural network, B1 and B2 are the bias matrices in the feedforward neural network, and Q final Decoded features output by the decoding layer.

9. The intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception according to claim 1 is characterized in that: The bounding boxes of the three sets of target detection outputs are fused, including: The bounding boxes of the three sets of target detection outputs are projected into a custom world coordinate system, the bounding boxes to be fused are determined using the intersection over union (IoU) ratio as the threshold, and the bounding boxes to be fused are weighted fused pairwise using the confidence of the bounding boxes as the weight until the optimal detection result output is obtained.

10. An intelligent UAV intrusion detection and tracking system based on optoelectronic fusion perception, characterized in that: The system comprises: The data acquisition device includes a wide-angle visible light camera, a zoom visible light camera, a zoom infrared camera and a laser radar; the zoom visible light camera, the zoom infrared camera and the laser radar are used to collect two-dimensional visible light images, two-dimensional infrared images and three-dimensional point cloud data of the target to be measured in the monitoring area, and the wide-angle visible light camera is used to provide a global field of view when detecting and tracking the target; A computing module, used to implement the intelligent UAV intrusion detection and tracking method based on optoelectronic fusion perception as described in any one of claims 1 to 9; The motion platform comprises two mechanical rotating structures controlled by the computing module, which are respectively responsible for the pitch and yaw movements of the data acquisition device according to the control signals output by the computing module.

Images