Anti-unmanned aerial vehicle system and method based on air-to-air visual recognition

Abstract

The invention discloses an anti-unmanned aerial vehicle system and method based on air-to-air visual identification, and belongs to the technical field of unmanned aerial vehicle countering. The holder unit is used for driving the detection unit to rotate; the motion unit is used for patrolling in the air and following a target; the capturing net unit is used for capturing and recovering a target; and the main control unit is used for sending a detection control instruction to the detection unit, receiving the monitoring information transmitted by the detection unit, receiving the azimuth information of the holder unit, sending a holder control instruction to the holder unit, receiving the azimuth information of the motion unit, sending an unmanned aerial vehicle motor control instruction to the motion unit and sending a net capturing instruction to the net capturing unit. According to the anti-unmanned aerial vehicle system and method based on air-to-air visual recognition, full-automatic unmanned aerial vehicle countering measures of three stages of recognition, tracking and capturing are achieved by means of a small number of common simple devices, and the real-time requirement under a certain condition can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of drone countermeasures, and in particular to an anti-drone system and method based on air-to-air visual recognition. Background Art

[0002] The rapid development of deep learning in recent years has greatly promoted downstream tasks such as target recognition. As a low-altitude aircraft that is becoming widely popular, drones have brought convenience to people while also causing many dangerous accidents. In addition, the gap in existing technology in the field of drone intelligent countermeasures urgently requires an efficient and fast anti-drone visual recognition system and effective countermeasures.

[0003] Regarding drone identification technology, we have noticed that most of the existing results directly borrow model methods from the CV field, such as the YoloV5 model, and use this as the core to build the corresponding detection system. However, given the high flight speed of drones (such as FPV drones), the real-time performance of the model algorithm and the effectiveness of the countermeasure technology have become bottlenecks. Existing visual recognition technology is usually unable to capture high-speed flying drones in real time at high frame rates, and existing countermeasures (such as electromagnetic interference, shotgun strikes, etc.) cannot play a very effective role. Summary of the invention

[0004] The purpose of the present invention is to provide an anti-UAV system and method based on air-to-air visual recognition to solve the problems existing in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides an anti-UAV system based on air-to-air visual recognition, comprising: The detection unit is used for visual detection of the airspace around the drone, real-time transmission of video information, and acquisition of monitoring information; The pan / tilt unit is used to drive the detection unit to rotate; Motion unit, used for air patrol and target following; A net unit, used to capture and recover the target; The main control unit is used to send detection control instructions to the detection unit, receive monitoring information transmitted by the detection unit, receive the orientation information of the gimbal unit, send gimbal control instructions to the gimbal unit, receive the orientation information of the motion unit, send drone motor control instructions to the motion unit, and send net capturing instructions to the net capturing unit.

[0006] Preferably, the detection unit includes one or more of the camera visual detection equipment, laser visual detection equipment, and infrared visual detection equipment carried by the gimbal, and also includes an image transmission module and a target recognition module. The camera visual detection module uses a monocular camera to two-dimensionalize the three-dimensional world and obtain video information of the airspace around the drone. It is one of the most commonly used visual detection equipment. It collects the image of the object through an optical sensor and uses image processing algorithms to extract feature information, such as edges, textures, colors, etc., to achieve object detection, recognition and analysis. The laser visual detection equipment is equipped with a laser light source on an analog camera. The laser light uses spot uniform enhancement technology and spot autofocus technology, and the picture width is strong and more uniform. The infrared visual detection equipment uses infrared light to scan the object, and obtains the thermal characteristics and infrared images of the object by measuring infrared radiation or reflection. It can achieve effective monitoring of the target area in low-light scenes such as at night.

[0007] Preferably, the image transmission module transmits the video stream captured by the camera vision detection device, the laser vision detection device or the infrared vision detection device to the analog image transmission via the UART interface. The analog image transmission encodes the video stream using the MAVLink protocol and sends it to the ground station via a wireless communication link. The target recognition module of the ground station obtains the video stream of the drone vision detection module and recognizes the target aircraft through the target recognition algorithm.

[0008] Preferably, the target recognition algorithm includes a YOLOV8-SkipNet model or a SwinT model.

[0009] Preferably, the YOLOV8-SkipNet model specifically includes: In the model training phase, we use pictures close to the actual target recognition scene to form a UAV dataset, which includes pictures of drones with different backgrounds, shapes, and postures. We divide the UAV dataset into two validation sets, use one UAV validation set to supervise the model training, and continuously use the other UAV validation set to evaluate the model until the model performance meets the requirements. In the actual reasoning stage, the target recognition module obtains the video stream of the drone visual detection module and uses frame skipping technology to A frame capture is provided to the recognition model, and then the UAV is annotated in real time through the pre-trained model to achieve target recognition.

[0010] Preferably, the SwinT model consists of several stages, each stage contains several Swin TransformerBlocks, and the input of the SwinT model is a block image, which is processed layer by layer and finally outputs features for classification or other tasks.

[0011] Preferably, the gimbal unit is a high-precision three-axis stabilized gimbal, including heading, pitch, and roll, which can maintain the stability of the camera in the horizontal and vertical directions. The high-precision three-axis stabilized gimbal is mainly composed of a base, a rotation axis, a gimbal bracket, a motor and other components. The gimbal angle jitter is ±0.01°, the controllable pitch rotation range is -90°~ +25°, the controllable horizontal rotation range is -270°~ +270°, and the roll rotation range is -45° ~+45°. The motor controller on the gimbal receives instructions from the ground control unit, completes decoding and controls the gimbal motor to complete the corresponding action; the motion unit is a UAV power device, mainly composed of a base, a controller, a motor, blades and other components. The power motor receives the command from the flight control board on the UAV, controls the motor to complete the corresponding action, and realizes the UAV to follow and track the target; the net-catching unit is composed of a net-catching launching device and a capturing net. The ground control unit sends a command to change the high (low) level of a certain level output interface of the flight control board to a low (high) level, starts the net-catching launching device, launches and opens the capturing net; the main control unit is composed of the ground control unit and the flight control board on the UAV. The ground control unit is composed of a computer with Ubuntu operating system and ROS (such as ROS Melodic). The flight control board is an open source flight control board that supports open source flight control firmware such as px4 and (or) Ardupilot and has surplus level output interfaces except the motor control interface.

[0012] ROS (Robot Operating System) is a framework for robot software development, providing a set of tools and libraries to help developers create complex and reliable robot behaviors. When deployed on a drone ground station, ROS exchanges data with the drone flight control board through the MAVLink communication protocol. MAVLink is a lightweight messaging protocol for real-time communication between drones and ground stations. Through ROS nodes, the ground station can send commands and receive status information of the drone to control and monitor the drone.

[0013] An anti-UAV method based on air-to-air visual recognition comprises the following steps: S1. The ground control unit generates motion control instructions based on the received target identification information, performs distance measurement and calculation, and generates net capture control instructions; S2, send control instructions to the drone flight control board; S3, the flight control board controls the motor of the gimbal unit, the motor of the UAV power unit and the net launching device.

[0014] Preferably, the motion control instruction in S1 is generated by a motion control algorithm, specifically: the motion control algorithm predicts the moving direction and speed of the target object, which is used to adjust the position and posture of the drone in advance; according to the position of the target and the predicted path, its flight path is planned, gradually approaching the target until the net position is reached; a distance measurement algorithm is used for distance measurement, specifically: based on the principle of similar triangles, by taking an image of the target object, using image processing technology to measure the pixel width or height of the object, combining the focal length of the camera and the distance relationship between the object and the camera, the distance between the object and the camera is calculated; for a target with a pixel width of P identified in the image, assuming that its actual width is W and the distance from the target is D, the focal length of the camera is , if the focal length of the camera is known, the target distance is obtained This method does not require a complex stereo vision system or multiple cameras, nor does it require the additional installation of distance sensors, so it is low-cost and easy to implement.

[0015] Preferably, the transmission of the control instructions in S2 is packaged through the MAVLink communication protocol, the ground control unit creates a ros topic and sends it to the UAV flight control board, and decodes it to obtain information, and controls the motor and the net unit through electrical signals.

[0016] After receiving the target identification information, the main control unit generates motion control instructions through the motion control algorithm, controls the UAV to follow the target, and controls the gimbal unit to rotate so that the detection unit is always aimed at the target; at the same time, the distance to the target is detected through the ranging algorithm, and the net capture instruction is sent after reaching the effective range of the net capture unit to capture the target UAV.

[0017] Therefore, the present invention adopts the above-mentioned anti-UAV system and method based on air-to-air visual recognition, which has the following beneficial effects: (1) The main control unit is mainly concentrated on the ground platform. There are no redundant control components on the UAV platform except the flight control board required for ordinary UAV flight, which reduces the weight of the UAV and greatly improves its maneuverability. This makes the anti-UAV system suitable for both ordinary UAVs and especially for cross-country drones with the characteristics of "light, small and fast"; (2) The functions of the ranging module and the detection unit in the net-catching unit are realized by the same device (both are video images taken by the camera and transmitted to the ground station, and then solved by the monocular vision algorithm or the target recognition algorithm), avoiding the introduction of additional ranging sensors and other components, thus improving the level of system integration; (3) When performing tracking tasks, a motion ranging algorithm is introduced to predict the possible moving direction and speed of the target object, so as to adjust the position and attitude of the drone in advance and plan the flight path to approach the target. The path planning and attitude control algorithm using motion estimation increases the accuracy of motion unit tracking and the robustness of tracking when the video stream of the pursued drone is temporarily lost; (4) By using flight control boards, capture nets and other devices that are readily available on the market, without introducing complex mechanisms or devices, the drone can autonomously and stably complete take-off, tracking and capture tasks at a relatively low cost and with simple manual operations.

[0018] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is an overall schematic diagram of an anti-UAV system based on air-to-air visual recognition of the present invention; Figure 2 This is a schematic diagram of the structure of the detection unit of the present invention; Figure 3 This is a schematic diagram of the control state of the main control unit of the present invention; Figure 4 This is a schematic diagram of the working status of the YOLOV8-SkipNet model of the present invention. DETAILED DESCRIPTION

[0020] Example In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0021] like Figure 1-Figure 4 As shown, an anti-UAV system based on air-to-air visual recognition includes: The detection unit is used for visual detection of the airspace around the drone, real-time transmission of video information, and acquisition of monitoring information; The pan / tilt unit is used to drive the detection unit to rotate; Motion unit, used for air patrol and target following; A net unit, used to capture and recover the target; The main control unit is used to send detection control instructions to the detection unit, receive monitoring information transmitted by the detection unit, receive the orientation information of the gimbal unit, send gimbal control instructions to the gimbal unit, receive the orientation information of the motion unit, send drone motor control instructions to the motion unit, and send net capturing instructions to the net capturing unit.

[0022] The detection unit includes a monocular camera, an image transmission module and a target recognition module. The image transmission module transmits the video stream captured by the monocular camera to the analog image transmission through the UART interface. The analog image transmission encodes the video stream using the MAVLink protocol and sends it to the ground station through a wireless communication link. The target recognition module of the ground station obtains the video stream of the UAV visual detection module and recognizes the target aircraft through the target recognition algorithm.

[0023] Target recognition algorithm There are two sets of target recognition algorithms, which are based on the YOLOV8 model and SwinT (SwinTransformer) model respectively. In addition, combined with the "frame skipping" technology used by reinforcement learning agents to process environmental states, we have given improved versions of the two models, which greatly reduces the amount of calculation and shortens the model inference time, which is of great significance for downstream tasks such as drone target recognition that have strong real-time requirements.

[0024] 1. YOLOV8 ①Structure: The network structure of YOLOv8 is mainly composed of four parts: input layer, backbone network (Backbone), feature pyramid network (FPN), head network (Head). Each part of the structure undertakes different functions, such as image segmentation, feature extraction, etc.

[0025] ②Advantages: Efficient feature extraction and fusion: By using CSPNet and PANet, the efficiency of feature extraction and feature fusion is improved.

[0026] More accurate positioning capability: The model’s ability to locate target objects is enhanced through an improved IoU loss function and an adaptive anchor box mechanism.

[0027] Enhanced data augmentation techniques: Techniques such as Mosaic data augmentation improve the robustness and generalization ability of the model.

[0028] 2. Frame Skipping Technology Frame skipping technology refers to the process of selectively skipping some frames without rendering or transmitting during video playback or transmission to reduce computing or bandwidth requirements. It is usually used when system resources are limited, such as running high frame rate videos on low-end hardware, or transmitting video streams when network bandwidth is insufficient.

[0029] A well-known application is in the reinforcement learning DQN network, where researchers used frame skipping technology to greatly reduce the amount of network computation, making decision-making faster.

[0030] 3. YOLOv8-SkipNet This embodiment proposes a technology that aims to reduce the amount of calculation and ensure real-time recognition. The implementation method is as follows: The real-time video stream transmitted by the camera enters the model.

[0031] According to the frame skipping hyperparameter setting k of the YOLOV8-SkipNet network (i.e., how many frames are predicted and calculated every), the corresponding frame is intercepted.

[0032] Prediction markers.

[0033] An intuitive understanding is that, assuming k=4, the model captures a picture every 4 frames for prediction, the amount of calculation will be reduced to 1 / 4 of the original, and it is also beneficial to the real-time performance of target detection.

[0034] 4. SwinT Model As an application of the Transformer model in vision, the powerful capabilities of SwinT have been recognized by many researchers. The overall structure of Swin Transformer consists of multiple stages, each of which includes several Swin Transformer Blocks. As the stages progress, the resolution of the feature map gradually decreases, while the number of channels gradually increases. The input of the model is a patch image, which is processed layer by layer and finally outputs features for classification or other tasks.

[0035] We noticed that SwinT has certain advantages over YOLOV8: ① Better global context capture capability: Swin Transformer uses a self-attention mechanism and is good at capturing global features. This gives it an advantage in dealing with complex scenes (such as small target detection of drones in natural environments) and can better understand the relationship between the target and the background.

[0036] ② Ability to handle deformation and scale changes: Swin Transformer can better cope with the deformation and scale problems of the target by processing image blocks in layers, especially for targets such as drones, which have large size variations and various shapes.

[0037] ③Ability to adapt to complex environments: Drones usually fly in various complex environments, such as cities, forests, mountains, etc. The self-attention mechanism and global feature extraction capability of Swin Transformer enable it to perform better when dealing with complex backgrounds, occlusions, and multi-target scenes.

[0038] However, it also has fatal shortcomings: large amount of calculation and slow inference speed, which cannot meet the real-time requirements of drone target detection. However, after combining with frame skipping technology, the model inference speed has been improved to a certain extent, and it is still worth deploying in some tasks that do not have very strict real-time requirements.

[0039] The gimbal unit is a high-precision three-axis stabilized gimbal, including heading, pitch, and roll, which can maintain the stability of the camera in the horizontal and vertical directions. The high-precision three-axis stabilized gimbal is mainly composed of a base, a rotation axis, a gimbal bracket, a motor and other components. The gimbal angle jitter is ±0.01°, the controllable pitch rotation range is -90°~ +25°, the controllable horizontal rotation range is -270°~ +270°, and the roll rotation range is -45° ~+45°. The motor controller on the gimbal receives instructions from the ground control unit, completes decoding and controls the gimbal motor to complete the corresponding action; the motion unit is a drone power device, mainly composed of a base, a controller, a motor, blades and other components. The power motor receives the command from the flight control board on the UAV, controls the motor to complete the corresponding action, and realizes the UAV to follow and track the target; the net-catching unit is composed of a net-catching launching device and a capturing net. The ground control unit sends a command to change the high (low) level of a certain level output interface of the flight control board to a low (high) level, starts the net-catching launching device, launches and opens the capturing net; the main control unit is composed of the ground control unit and the flight control board on the UAV. The ground control unit is composed of a computer with Ubuntu operating system and ROS (such as ROS Melodic). The flight control board is an open source flight control board that supports open source flight control firmware such as px4 and (or) Ardupilot and has surplus level output interfaces except the motor control interface.

[0040] ROS (Robot Operating System) is a framework for robot software development, providing a set of tools and libraries to help developers create complex and reliable robot behaviors. When deployed on a drone ground station, ROS exchanges data with the drone flight control board through the MAVLink communication protocol. MAVLink is a lightweight messaging protocol for real-time communication between drones and ground stations. Through ROS nodes, the ground station can send commands and receive status information of the drone to control and monitor the drone.

[0041] An anti-UAV method based on air-to-air visual recognition comprises the following steps: S1. The ground control unit generates motion control instructions based on the received target identification information, performs distance measurement and calculation, and generates net capture control instructions; S2, send control instructions to the drone flight control board; S3, the flight control board controls the motor of the gimbal unit, the motor of the UAV power unit and the net launching device.

[0042] The motion control instructions in S1 are generated by the motion control algorithm, specifically: the motion control algorithm predicts the moving direction and speed of the target object, which is used to adjust the position and posture of the drone in advance; according to the position of the target and the predicted path, its flight path is planned, gradually approaching the target until it reaches the capture net position; the distance measurement algorithm is used for distance measurement, specifically: based on the principle of similar triangles, by taking an image of the target object, using image processing technology to measure the pixel width or height of the object, combining the focal length of the camera and the distance relationship between the object and the camera, the distance between the object and the camera is calculated; for the target with a pixel width of P identified in the image, assuming its actual width is W and the distance from the target is D, the focal length of the camera is , if the focal length of the camera is known, the target distance is obtained This method does not require a complex stereo vision system or multiple cameras, nor does it require the additional installation of distance sensors, so it is low-cost and easy to implement.

[0043] The transmission of control instructions in S2 is packaged through the MAVLink communication protocol. The ground control unit creates a ros topic and sends it to the UAV flight control board, which decodes and obtains information, and controls the motor and net unit through electrical signals.

[0044] After receiving the target identification information, the main control unit generates motion control instructions through the motion control algorithm, controls the UAV to follow the target, and controls the gimbal unit to rotate so that the detection unit is always aimed at the target; at the same time, the distance to the target is detected through the ranging algorithm, and the net capture instruction is sent after reaching the effective range of the net capture unit to capture the target UAV.

[0045] Therefore, the present invention adopts the above-mentioned anti-UAV system and method based on air-to-air visual recognition, and combines the integrated anti-UAV system with an efficient and real-time UAV target detection algorithm. It can not only achieve the three-stage fully automatic UAV countermeasures of "identification, tracking, and capture" without the additional introduction of an onboard computer or an onboard single-chip microcomputer, but also rely on a small number of simple devices commonly found on the market. It can also alleviate the large amount of calculation while meeting the real-time requirements under certain conditions.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An anti-UAV system based on air-to-air visual recognition, characterized in that: include: The detection unit is used for visual detection of the airspace around the drone, real-time transmission of video information, and acquisition of monitoring information; The pan / tilt unit is used to drive the detection unit to rotate; Motion unit, used for air patrol and target following; A net unit, used to capture and recover the target; The main control unit is used to send detection control instructions to the detection unit, receive monitoring information transmitted by the detection unit, receive the orientation information of the gimbal unit, send gimbal control instructions to the gimbal unit, receive the orientation information of the motion unit, send drone motor control instructions to the motion unit, and send net capturing instructions to the net capturing unit.

2. The anti-UAV system based on air-to-air visual recognition according to claim 1 is characterized in that: The detection unit includes one or more of a camera vision detection device, a laser vision detection device, and an infrared vision detection device mounted on the gimbal, and also includes an image transmission module and a target recognition module.

3. The anti-UAV system based on air-to-air visual recognition according to claim 2 is characterized in that: The image transmission module transmits the video stream captured by the camera vision detection device, laser vision detection device or infrared vision detection device to the analog image transmission through the UART interface. The analog image transmission encodes the video stream using the MAVLink protocol and sends it to the ground station through a wireless communication link. The target recognition module of the ground station obtains the video stream of the drone vision detection module and recognizes the target aircraft through the target recognition algorithm.

4. The anti-UAV system based on air-to-air visual recognition according to claim 3 is characterized in that: The target recognition algorithms include the YOLOV8-SkipNet model or the SwinT model.

5. The anti-UAV system based on air-to-air visual recognition according to claim 1 is characterized in that: The YOLOV8-SkipNet model specifically includes: In the model training phase, we use pictures similar to the actual target recognition scene to form a UAV dataset, which includes pictures of drones with different backgrounds, shapes, and postures. We divide the UAV dataset into two validation sets, use one UAV validation set to supervise the model training, and continuously use the other UAV validation set to evaluate the model until the model performance meets the requirements. In the actual reasoning stage, the target recognition module obtains the video stream of the drone visual detection module and uses frame skipping technology to A frame capture is provided to the recognition model, and then the UAV is annotated in real time through the pre-trained model to achieve target recognition.

6. The anti-UAV system based on air-to-air visual recognition according to claim 4 is characterized in that: The SwinT model consists of several stages, each of which contains several Swin Transformer Blocks. The input of the SwinT model is a block image, which is processed layer by layer and finally outputs features for classification or other tasks.

7. The anti-UAV system based on air-to-air visual recognition according to claim 1 is characterized in that: The gimbal unit is a high-precision three-axis stabilized gimbal, including heading, pitch, and roll; the motion unit is the UAV power unit; the net unit consists of a net launching device and a capture net; the main control unit consists of a ground control unit and a flight control board on the UAV. 8.An anti-UAV method based on air-to-air visual recognition, characterized in that: The following steps are involved: S1. The ground control unit generates motion control instructions based on the received target identification information, performs distance measurement and calculation, and generates net capture control instructions; S2, send control instructions to the drone flight control board; S3, the flight control board controls the motor of the gimbal unit, the motor of the UAV power unit and the net launching device.

9. The anti-UAV method based on air-to-air visual recognition according to claim 8 is characterized in that: The motion control instructions in S1 are generated by the motion control algorithm, specifically: the motion control algorithm predicts the moving direction and speed of the target object, which is used to adjust the position and posture of the drone in advance; according to the position of the target and the predicted path, its flight path is planned, gradually approaching the target until it reaches the capture net position; the distance measurement algorithm is used for distance measurement, specifically: based on the principle of similar triangles, by taking an image of the target object, using image processing technology to measure the pixel width or height of the object, combining the focal length of the camera and the distance relationship between the object and the camera, the distance between the object and the camera is calculated; for the target with a pixel width of P identified in the image, assuming its actual width is W and the distance from the target is D, the focal length of the camera is , if the focal length of the camera is known, the target distance is obtained .

10. The anti-UAV method based on air-to-air visual recognition according to claim 8, characterized in that: The transmission of control instructions in S2 is packaged through the MAVLink communication protocol. The ground control unit creates a ros topic and sends it to the UAV flight control board, which decodes and obtains information, and controls the motor and the net unit through electrical signals. After receiving the target identification information, the main control unit generates motion control instructions through the motion control algorithm, controls the UAV to follow the target, and controls the gimbal unit to rotate so that the detection unit is always aimed at the target; at the same time, the distance to the target is detected through the ranging algorithm, and the net capture instruction is sent after reaching the effective range of the net capture unit to capture the target UAV.

Images