A method and electronic device for detecting unmanned aerial vehicle targets

Abstract

The present invention discloses a method and electronic device for detecting a drone target, wherein the method comprises: obtaining an image sequence to be detected according to video data to be detected; extracting dynamic region images and static space images corresponding to image frames in the image sequence; fusing dynamic region features and static space features, and performing multi-template matching to determine whether a drone target exists in the image frame. The present invention can improve the detection accuracy of low-speed, slow, and small drones.

Description

Technical Field

[0001] The invention belongs to the technical field of target detection, and in particular relates to a method and electronic equipment for detecting unmanned aerial vehicle targets. Background Art

[0002] With the rapid development of artificial intelligence technology and drone technology, small drones are being used more and more widely, especially in the open development of low-altitude economy, where small drones are widely used in low-altitude economic fields such as logistics, information exploration, and smart transportation. However, while "low, slow, and small" drones represented by consumer-grade drones play an important role in daily life and industrial production, they also bring new safety risks.

[0003] "Low, slow and small" is the abbreviation of "low altitude, slow speed and small target". At present, there is no clear and unified definition standard in the academic community. Generally, rotary-wing and fixed-wing drones with a maximum take-off weight of less than 25kg and a flight speed of less than 180km / h can be regarded as "low, slow and small" drones. To carry out safety management of drones, the first thing to solve is the problem of drone detection and early warning.

[0004] However, due to factors such as the small size of UAV targets, weak radiation, complex background and long shooting distance, there are still many difficulties and challenges in the effective detection of UAV targets. Summary of the invention

[0005] The present application provides a method and electronic equipment for detecting unmanned aerial vehicle targets, which can improve the detection accuracy of low-speed, slow and small unmanned aerial vehicles.

[0006] In a first aspect, the present application provides a method for detecting a drone target, the method comprising:

[0007] Obtaining an image sequence according to the video data to be detected;

[0008] Extracting dynamic area images corresponding to image frames in an image sequence;

[0009] Extracting an image corresponding to the position of the dynamic region image on the image frame as a static space image corresponding to the image frame;

[0010] Based on the dynamic area image and the static space image, template matching is performed to determine whether there is a drone target in the image frame.

[0011] In a possible implementation manner of the first aspect, extracting the dynamic area image corresponding to the image frame in the image sequence includes: extracting the dynamic area image corresponding to the image frame in the image sequence based on a frame difference method.

[0012] In a possible implementation manner of the first aspect, extracting a dynamic area image corresponding to an image frame in an image sequence based on a frame difference method includes:

[0013] Calculate the first Frame image and The difference between the frame images is used to obtain the first image ;

[0014] For the first Frame image, for the first image Perform binarization to obtain the second image ;

[0015] For the second image Perform corrosion and expansion processing to obtain the third image ;

[0016] Get the third image to obtain a fourth image marked with each contour;

[0017] A dynamic area image corresponding to the image frame is obtained based on the fourth image.

[0018] It should be understood that is the sequence number of the image frame in the image sequence, The frame image can be any frame image in the image sequence.

[0019] In a possible implementation manner of the first aspect, a first The dynamic area image corresponding to the frame image includes:

[0020] Perform Gaussian filtering on the fourth image to obtain a fifth image, and obtain a The dynamic area image corresponding to the frame image.

[0021] In a possible implementation manner of the first aspect, the first The dynamic area image corresponding to the frame image includes:

[0022] Based on each contour marked in the fifth image, determining in turn whether the size thereof is within a preset range;

[0023] If the size of the contour is within the preset range, it will be retained; if the size of the contour is not within the preset range, it will be removed;

[0024] The image inside the contour retained on the fifth image is used as the The dynamic area image corresponding to the frame image.

[0025] In a possible implementation manner of the first aspect, extracting a static spatial image corresponding to an image frame in an image sequence includes:

[0026] For the first Frame image, extract the The area on the frame image corresponding to the contour retained on the fifth image is used as the The static spatial image corresponding to the frame image.

[0027] In a possible implementation manner of the first aspect, performing template matching based on the dynamic area image and the static space image to determine whether a drone target exists in the image frame includes:

[0028] Matching the dynamic area image corresponding to the image frame with the dynamic area template image to obtain a first matching value;

[0029] Matching the static space image corresponding to the image frame with the static space template image to obtain a second matching value;

[0030] Performing weighted calculation on the first matching value and the second matching value to obtain a third matching value;

[0031] If the first matching value is greater than the first threshold, the second matching value is greater than the second threshold, and the third matching value is greater than the third threshold, it is determined that there is a drone target in the image frame.

[0032] In a possible implementation manner of the first aspect, performing template matching based on the dynamic area image and the static space image to determine whether a drone target exists in the image frame includes:

[0033] The dynamic area image corresponding to the image frame is matched with the dynamic area template image to obtain a first matching value; if the first matching value is less than or equal to the first threshold, it is determined that there is no drone target in the image frame; if the first matching value is greater than the first threshold, the static space image corresponding to the image frame is matched with the static space template image to obtain a second matching value; if the second matching value is less than or equal to the second threshold, it is determined that there is no drone target in the image frame; if the second matching value is greater than the second threshold, the first matching value and the second matching value are weightedly calculated to obtain a third matching value; if the third matching value is less than or equal to the third threshold, it is determined that there is no drone target in the image frame; if the third matching value is greater than the third threshold, it is determined that there is a drone target in the image frame.

[0034] In a possible implementation manner of the first aspect, performing template matching based on the dynamic area image and the static space image to determine whether a drone target exists in the image frame includes:

[0035] The static space image corresponding to the image frame is matched with the static space template image to obtain a second matching value; if the second matching value is less than or equal to the second threshold, it is determined that there is no drone target in the image frame; if the second matching value is greater than the second threshold, the dynamic area image corresponding to the image frame is matched with the dynamic area template image to obtain a first matching value; if the first matching value is less than or equal to the first threshold, it is determined that there is no drone target in the image frame; if the first matching value is greater than the first threshold, the first matching value and the second matching value are weightedly calculated to obtain a third matching value; if the third matching value is less than or equal to the third threshold, it is determined that there is no drone target in the image frame; if the third matching value is greater than the third threshold, it is determined that there is a drone target in the image frame.

[0036] In a possible implementation manner of the first aspect, performing template matching based on the dynamic area image and the static space image to determine whether a drone target exists in the image frame includes:

[0037] Extract the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area image corresponding to the image frame; match the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area image corresponding to the image frame with the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area template image respectively to obtain the first dynamic matching value, the second dynamic matching value and the third dynamic matching value; perform weighted calculation on the first dynamic matching value, the second dynamic matching value and the third dynamic matching value to obtain the first matching value; wherein the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area image are respectively the dynamic area image itself, the histogram of oriented optical flow (HOF) feature of the dynamic area image and the fully convolutional network (FCN) feature of the dynamic area image; accordingly, the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area template image are respectively the dynamic area template image itself, the histogram of oriented optical flow (HOF) feature of the dynamic area template image and the fully convolutional network feature of the dynamic area template image;

[0038] Extract the first static feature, the second static feature and the third static feature of the static space image corresponding to the image frame; match the first static feature, the second static feature and the third static feature of the static space image corresponding to the image frame with the first static feature, the second static feature and the third static feature of the static space template image respectively to obtain a first static matching value, a second static matching value and a third static matching value; perform weighted calculation on the first static matching value, the second static matching value and the third static matching value to obtain a second matching value; wherein the first static feature, the second static feature and the third static feature of the static space image are respectively the static space image itself, the local binary pattern (LBP) feature of the image and the full convolutional network feature of the image; accordingly, the first static feature, the second static feature and the third static feature of the static space template image are respectively the static space template image itself, the local binary pattern feature of the static space template image and the full convolutional network feature of the static space template image;

[0039] The first matching value and the second matching value are weightedly calculated to obtain a third matching value; if the first matching value is greater than the fourth threshold, the second matching value is greater than the fifth threshold, and the third matching value is greater than the sixth threshold, it is determined that there is a drone target in the image frame.

[0040] In a second aspect, the present application provides a drone target detection system, comprising:

[0041] An image sequence acquisition module is used to obtain an image sequence according to the video data to be detected;

[0042] A dynamic area image extraction module is used to extract the dynamic area image corresponding to the image frame in the image sequence;

[0043] A static space image extraction module, used for extracting a static space image corresponding to an image frame in an image sequence;

[0044] The judgment module is used to perform template matching based on the dynamic area image and the static space image to determine whether there is a drone target in the image frame.

[0045] It should be understood that the above-mentioned drone target detection system has the function of implementing the method in the first aspect and any possible implementation thereof. Its functional modules can be implemented by hardware, or by hardware executing corresponding software.

[0046] In a third aspect, the present application provides an electronic device, including: a memory and a processor;

[0047] Memory for storing computer programs;

[0048] A processor is used to call a computer program to execute the method in the first aspect and any possible implementation thereof.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program runs on an electronic device, the electronic device implements the method in the first aspect and any possible implementation thereof.

[0050] In a fifth aspect, the present application provides a computer program product, including a computer program. When the computer program runs on an electronic device, the electronic device executes the method described in the first aspect and any possible design thereof.

[0051] Beneficial effect: The technical solution provided by the present application, by acquiring the dynamic area image and static space image corresponding to the image frame in the video data to be detected, adopts a dynamic + static multi-template matching method for target detection, which can improve the detection capability of low-speed and small UAV targets with extremely small pixels in dynamic and large-angle scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of an embodiment of the present application;

[0053] Figure 2 This is a template matching flow chart of an embodiment of the present application;

[0054] Figure 3 A flowchart of building and formatting a multi-feature library according to an embodiment of the present application;

[0055] Figure 4 This is a drone target detection result in the visible light mode of an embodiment of the present application;

[0056] Figure 5 This is a drone target detection result in infrared mode according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to better understand the technical solution of the present invention, the relevant technology is first briefly introduced.

[0058] 1. Image target detection

[0059] In recent years, image target detection methods are mainly based on deep learning algorithms, most of which are based on multi-scale feature fusion, attention structure enhancement and other methods to improve the performance of small target detection. However, conventional improvement methods still have certain problems in dealing with extremely small target detection. By annotating the target drone and training it with the most popular YOLOV8 series algorithm for 100 epochs, it was found that: first, the model is difficult to converge, and second, the trained model can hardly detect the target. Therefore, it is basically difficult to rely on single-frame image detection methods on low-speed small target data sets with extremely small pixels. Because the target with extremely small pixels is gradually lost in the downsampling process of convolution and pooling in deep learning, it is not transmitted to the deep layer with the network. Assume that the original image has 1920×1280 pixels and the drone target has 30×20 pixels. Taking YOLOV8 as an example, the original image is usually compressed to 640×640 before inputting into the network, so the effective pixels of the input target are only 10×6. The YOLOV8 backbone network uses magnifications of 1 / 8, 1 / 16, and 1 / 32 for the feature map transmitted backward, so the target resolution is less than 1×1. It can be seen that due to the small proportion of extremely small target pixels, the existing deep learning feature extraction structure based on downsampling is easy to drown out the target information. Therefore, it is basically not feasible to rely solely on deep learning methods to detect low, slow, and small targets in a large field of view.

[0060] 2. Video target detection

[0061] Video target detection uses the temporal and spatial context information of the target and makes judgments based on its motion change characteristics. Motion target detection includes two methods: association verification and direct dynamic area determination. Among them, association verification is to perform association judgment based on single-frame detection with the assistance of multi-frame information to reduce false alarm rate and missed detection rate. Direct determination is to obtain moving targets through calculations such as frame difference, background difference, and optical flow method. Since the multi-frame association verification method needs to be based on single-frame image detection, there are still great challenges in detecting extremely small targets in a large field of view. The direct dynamic area determination method directly obtains the target motion area based on the mutual calculation of multi-frame images. The calculation is simple, but it is prone to false alarms when there are interference from other targets. Therefore, both methods have certain limitations.

[0062] From the case of human eyes looking for floaters, we can conclude that the human eye's discovery of extremely small targets mostly begins with being attracted by their motion characteristics, and then makes secondary identifications based on this. Similarly, similar patterns can be found by observing a large number of low-speed and slow drone videos. In fact, there are certain similarities between floaters and low-speed and slow drones: first, the target size is very small and the features are not significant, and the pixel ratio in the entire image is very small; second, the background is complex, and the target is easily submerged in the background texture.

[0063] By observing the original images of low-speed, slow and small targets in visible light mode (RGB mode) and infrared mode, the images after dynamic background difference, and the binarized images after dynamic background difference, it can be seen that it is very difficult to detect the target drone by relying solely on a single-frame image. However, after obtaining dynamic information through a simple difference between the previous and next frames, the area of ​​the suspected target drone can be preliminarily determined.

[0064] 3. Low-speed and small drone visual data set

[0065] Compared with the Common Objects in Context (COCO) dataset, the low-speed, slow, and small drone target data has its own particularity, which is manifested in the following aspects: ① The platform shooting distance is far, the target size is small, and the pixel ratio of the target in the original image is extremely low; ② The data background is complex, such as the influence of various complex environmental noises in the background of mountains and buildings, and the target is easily submerged in the complex background. However, it is worth noting that compared with multiple target category detection datasets in natural scenes, the low-speed, slow, and small target dataset is mainly for drone category detection, focusing on the judgment of the presence or absence of drones.

[0066] At present, public datasets of low-speed, slow, and small UAV targets are gradually emerging. Some public UAV target datasets include the Det-Fly dataset for air-to-air scenarios proposed by Westlake University, and the Anti-UAV multimodal UAV dataset jointly proposed by the University of Science and Technology of China and the Northern Electronic Equipment Research Institute. This type of dataset includes different modes and different backgrounds, and basically meets the needs of low-speed, slow, and small UAV target detection and tracking.

[0067] Based on the existing data sets of Det-Fly and Anti-UAV, this embodiment of the application builds a ground-fixed optoelectronic pod acquisition system to collect some data, which together constitute the experimental data set of this embodiment of the application. There are 102 segments of experimental data with a total capacity of 12.6G. The acquisition methods include ground-fixed optoelectronic acquisition and aerial drone optoelectronic acquisition. The experimental data includes different complex backgrounds such as sky, mountains, buildings, and two modes of visible light and infrared. The data is saved in video form. The resolution of light data can be 1920×1080, and the resolution of infrared data can be 640×512.

[0068] Academically, the relative area ratio of the target instance in the image is usually used to measure whether it is a small target. The ratio of the small target box area to the image area is usually less than 0.12%. Considering that the embodiment of this application focuses on the problem of large-field-of-view and small-size drone target detection, the targets with a target pixel ratio greater than 0.12% in the original data set are discarded, and the data with a pixel ratio less than 0.12% is divided into smaller targets, small targets, and extremely small targets. As shown in Table 1, it is the target size classification standard.

[0069] Table 1 Object size classification standard (original image 1920×1080) ,

[0070] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] The embodiments of the present application can be applied to solve the problem of low, slow and small UAV target detection. Low, slow and small UAV target detection is a very complex task. Common "low, slow and small" UAV detection methods include radar detection, photoelectric detection and radio detection. At present, there is no ideal means to solve all detection problems. Taking photoelectric detection as an example, choosing a large field of view means that the target pixel size in a 1920×1080 high-definition image is often only a few dozen pixels, which is a great challenge for detection methods based on deep learning; and choosing a small field of view means that the observation area is very small, which is basically not suitable for initial target discovery. It is a common idea to use radar to guide photoelectricity to stably track the target under a small field of view. However, in this way, each photoelectric pod can only deal with a few UAV targets. When there are multiple UAV targets or interference objects such as flying birds and low-altitude floating objects at the same time, it is easy to have low efficiency and handling errors. Therefore, it is very important to detect and identify low, slow and small targets under large viewing angle conditions. Secondly, in view of the small size and characteristics of low-speed and slow small targets, a multi-template matching strategy is proposed. By comprehensively judging the dynamic and static space template images, the target detection results are output; in order to improve the detection capability of low-speed and slow small drone targets with extremely small pixels, this application proposes a detection method that integrates inter-frame information and template matching. First, a dynamic area image extraction module is designed to filter out background information interference and guide the algorithm to focus on dynamically changing small target areas; secondly, a multi-template matching strategy is used to judge the similarity of the selected dynamic areas to complete drone target detection.

[0072] Based on this, the embodiment of the present application provides a method for detecting drone targets, which obtains an image sequence according to the video data to be detected; extracts the dynamic area image and the static space image corresponding to the image frame in the image sequence; and performs template matching based on the dynamic area image and the static space image to determine whether there is a drone target in the image frame. Since the pixel proportion of small-sized drone targets under large field of view is extremely small, it is very difficult to detect targets only by relying on a single frame image. The method provided in this embodiment obtains the target position by extracting the dynamic area image. At the same time, in order to prevent false alarms when there are other targets interfering, the embodiment of the present application obtains static space image matching based on the motion area, and fuses the dynamic area image with the static space image for target detection, which can improve the accuracy of target detection; in addition, the present embodiment proposes a multi-template matching strategy, which realizes the detection of the presence or absence of the target through the comprehensive judgment of dynamic matching and static space matching. Through the above scheme, the embodiment of the present application improves the accuracy of detection and recognition of low, slow and small targets under large viewing angle conditions.

[0073] In order to better understand the embodiments of the present application, the drone target detection method provided by the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0074] The present application provides a method for detecting a drone target, which may include:

[0075] S1. Obtain an image sequence according to the video data to be detected;

[0076] S2, extracting the dynamic area image corresponding to the image frame in the image sequence;

[0077] In some embodiments, the dynamic area image corresponding to the image frame in the image sequence may be extracted based on the frame difference method.

[0078] Classic image dynamic area image extraction methods include frame difference, background difference, optical flow method, etc. The frame difference method uses the principle that the background changes between two adjacent frames are small and the moving target changes are large, and extracts the target position through frame difference. The extraction results of different image dynamic area image extraction methods are compared, among which the frame difference is the simplest front and back frame difference method, the background difference adopts the Gaussian background modeling method, and the optical flow method is selected as the FlowNetV2 optical flow calculation method. It can be seen from the results that the frame difference and background difference methods can retain the information of the target drone, while the FlowNetV2 optical flow method is difficult to retain small-size information. Based on this, the embodiment of the present application adopts the frame difference method to perform preliminary dynamic area image acquisition.

[0079] In some embodiments, extracting a dynamic region image corresponding to an image frame in an image sequence based on a frame difference method may include:

[0080] S21, for the first frame image, calculate the first Frame Image and Frame Image The difference between ,Right now ;

[0081] In some other embodiments, the first image You can also take Frame Image and Frame Image The difference is ;

[0082] S22: First image Perform binarization to obtain a binarized image, which can be recorded as the second image ;

[0083] Among them, binarization can enhance the moving target to improve the detection efficiency and accuracy. The binarization threshold can be recorded as , Take an empirical value. For example, in some embodiments, through preliminary experiments, the binarization threshold The value 20 is selected to be effective in scenes with similar target and background textures. More false alarm targets due to a smaller threshold value are filtered out in subsequent similarity matching.

[0084] S23, for the second image Perform corrosion and expansion processing to obtain an image after corrosion and expansion processing, which can be recorded as the third image ;

[0085] Among them, the parameters of corrosion and expansion processing can take the same value, denoted as ;

[0086] S24, obtaining the third image to obtain a fourth image marked with each contour;

[0087] Among them, the contour list can be recorded as ;

[0088] S25, obtaining a first The dynamic area image corresponding to each frame.

[0089] In some embodiments, the first The dynamic area image corresponding to the frame image may include:

[0090] Perform Gaussian filtering on the fourth image to obtain a fifth image, and obtain a The dynamic area image corresponding to the frame image.

[0091] Among them, performing Gaussian filtering preprocessing on the fourth image can filter out scattered points / noise points in the image after difference, and effectively eliminate the noise influence caused by the movement of the detection platform.

[0092] In some embodiments, the first The dynamic area image corresponding to the frame image may include:

[0093] Based on each contour marked in the fifth image, determining in turn whether the size thereof is within a preset range;

[0094] If the size of the contour is within the preset range, it will be retained; if the size of the contour is not within the preset range, it will be removed;

[0095] The image inside the contour retained on the fifth image is used as the The dynamic area image corresponding to the frame image.

[0096] Since contours that are too large or too small do not belong to drone targets, a preset range can be set according to the size of the drone target. By judging the size of each contour marked in the fifth image, contours that are too large or too small and whose sizes are not within the preset range are eliminated, and contours that do not belong to drone targets are eliminated, thereby reducing the amount of data in the dynamic area image, thereby reducing the amount of data required for subsequent template matching, and improving the efficiency of detection.

[0097] The dynamic area image actually has a certain similarity with the target's outline appearance, but because both the experimental platform and the target are in motion, and the background information (including mountains, buildings, etc.) is relatively complex, the accuracy of relying solely on the dynamic area image is relatively low. The original intention of designing the dynamic area image extraction is to retain the target drone information, while excluding as much background area as possible, filtering out background information interference, and guiding the algorithm to focus on the dynamically changing small target area to provide support for subsequent judgments.

[0098] S3, extracting an image corresponding to the position of the dynamic area image on the image frame as a static space image corresponding to the image frame;

[0099] In some embodiments, for the first Frame image, you can extract the The area on the frame image at a position corresponding to the contour retained on the fifth image is used as the static space image corresponding to the image frame.

[0100] It should be understood that there may be multiple contours retained on the fifth image in the above steps, and accordingly, there may also be multiple areas on the image frame corresponding to the contours retained on the fifth image. Therefore, the dynamic area image corresponding to the image frame may include multiple area images, and the static space image corresponding to the image frame may also include multiple area images. In the subsequent template matching process, each area image needs to be used as an image to be matched to perform similarity calculation with the dynamic area template image or the static space template image.

[0101] S4. Perform template matching based on the dynamic area image and the static space image to determine whether there is a drone target in the image frame.

[0102] It should be understood that in some embodiments, before step S1, it is necessary to first obtain image sequence samples based on the video sample library, annotate the drone target in the samples, create a dynamic area template image and a static space template image library of the drone target, and further perform feature extraction based on the dynamic area template image and the static space template image of the drone target to construct a feature library of the dynamic area template image and the static space template image of the drone target. The feature library can include the original texture information of the target, manual features, and deep convolutional neural network features to construct a multi-level feature library. Figure 3 As shown, the feature library record of a single target sample may include {unique ID number, target category, first static feature, second static feature, third static feature, first dynamic feature, second dynamic feature, third dynamic feature}. Among them, LBP feature extraction and HOF feature extraction can be calculated using the function library provided by OpenCV; FCN feature extraction can use ResNet full convolution network; in order to facilitate subsequent calculations, considering the small size of the input template image and the image to be matched, the convolution layer and output dimension of the full convolution network can be adjusted, and finally normalized to a one-dimensional array with an output dimension of 1×100. Of course, it can also be set to other dimensions according to needs.

[0103] When constructing the feature library, the above operations are performed on the static space template image and dynamic area template image of each UAV target to solve the corresponding feature data and store them in a structured manner in sequence.

[0104] The method for obtaining the static space template image and the dynamic area template image of each UAV target specifically includes the following steps: first, a video sample is selected, and the UAV target in the video sample image frame is annotated by manual annotation (the annotation tool can select LabelImg annotation software), and the annotation information of the UAV target is obtained as [id, x, y, w, h], where id is the frame number, (x, y) is the image coordinate of the upper left corner of the UAV target, and (w, h) is the image width and height of the UAV target; next, after the above operations are completed, the video is read to obtain each frame image, and the dynamic image corresponding to the current frame image is calculated according to the previous frame image data and the current frame image data (the frame difference method can be used, and it should be noted that the dynamic area obtaining method used in actual detection should be consistent); then, according to the current frame id, the image position [x, y, w, h] of the UAV target is obtained from the target annotation information, and the static space image and dynamic area image of the UAV target are obtained from the current frame image and the dynamic image corresponding to the current frame image according to the image position, as the static space template image and dynamic area template image of the UAV target.

[0105] The static space template image and the dynamic area template image of the UAV target are obtained, and then the static space features and dynamic area features of the UAV target are extracted for the static space template image and the dynamic area template image. For example, the static space template image itself can be used as the first static feature of the static space template image, the LBP feature of the static space template image can be used as the second static feature of the static space template image, and the full convolution network feature of the static space template image can be used as the third static feature of the static space template image; the dynamic area template image itself can be used as the first dynamic feature of the dynamic area template image, the HOF feature of the dynamic area template image can be used as the second dynamic feature of the dynamic area template image, and the full convolution network feature of the dynamic area template image can be used as the third dynamic feature of the dynamic area template image.

[0106] Template matching is actually to obtain the similarity of two images. On the basis of dynamically filtering out part of the background, the target area is further screened out based on template matching, which mainly involves what kind of feature matching (image feature extraction) and matching similarity calculation rules. Since the sample size is very small (the minimum is about 8×6), the embodiment of the present application does not involve too much image feature extraction. The embodiment of the present application uses two templates, namely static space template image and dynamic area template image. For different types of drones, different static space template images and dynamic area template images are selected respectively. For example, for three types of drones, including DJI Elf, DJI Mavic 3 and Daotong, three different static space template images and dynamic area template images are selected respectively. It should be noted that considering that the drone template is affected by the target shooting angle, flight speed, etc., the construction of the template library is very important and a long-term task, so building a complete template library is a long-term iterative process. The template constructed from the video sample is used for the detection of other videos through the experimental link to ensure that the constructed template has a certain scene adaptability.

[0107] In some embodiments, the resolutions of all images to be matched and template images are normalized to 30×20.

[0108] In some embodiments, the similarity of the images is calculated using a zero-mean normalized cross-correlation function, and the calculation formula is as follows:

[0109] (1)

[0110] in, and They are the image to be matched and the template image. is the coordinate on the template image The gray value of the pixel at position, is the coordinate on the image to be matched The grayscale value of the pixel at position. and They are And the average gray value of all pixels in the image to be matched.

[0111] Zero-mean normalized cross-correlation first subtracts the average grayscale value of all pixels on the image from the grayscale value of each pixel on the image to weaken the influence of image brightness, and then calculates the correlation of the two image matrices on this basis. Correlation is a mathematical distance that can be used to measure the similarity between two vectors (matrices). The product of the pixels on the image to be matched and the template image is the correlation coefficient of the two matrices. The larger the matching result, the greater the similarity of the two images.

[0112] In the above embodiment, step S4 can also be implemented through a deep learning recognition network, that is, the image area to be matched (dynamic and static at the same time) is used as the input of the deep learning recognition network, and the output is set to 0 and 1, 0 indicates that it is a drone, and 1 indicates that it is not a drone.

[0113] In some embodiments, step S4 may include:

[0114] Matching the dynamic area image corresponding to the image frame with the dynamic area template image to obtain a first matching value;

[0115] Matching the static space image corresponding to the image frame with the static space template image to obtain a second matching value;

[0116] Performing weighted calculation on the first matching value and the second matching value to obtain a third matching value;

[0117] If the first matching value is greater than the first threshold, the second matching value is greater than the second threshold, and the third matching value is greater than the third threshold, it is determined that there is a drone target in the image frame.

[0118] In other embodiments, Figure 2 As shown, step S4 may include:

[0119] The dynamic area image corresponding to the image frame is matched with the dynamic area template image to obtain a first matching value; if the first matching value is less than or equal to the first threshold, it is determined that there is no drone target in the image frame; if the first matching value is greater than the first threshold, the static space image corresponding to the image frame is matched with the static space template image to obtain a second matching value; if the second matching value is less than or equal to the second threshold, it is determined that there is no drone target in the image frame; if the second matching value is greater than the second threshold, the first matching value and the second matching value are weightedly calculated to obtain a third matching value; if the third matching value is less than or equal to the third threshold, it is determined that there is no drone target in the image frame; if the third matching value is greater than the third threshold, it is determined that there is a drone target in the image frame.

[0120] In some embodiments, the first threshold value may be equal to the second threshold value. In weighted calculation, the weights of the first matching value and the second matching value are both 0.5. That is, the dynamic area template image matching and the static space template image matching results must simultaneously meet the basic threshold value, and the average value of the two must be greater than a certain threshold value, i.e., the third threshold value, before it is determined that there is a drone target in the image.

[0121] The first threshold, the second threshold and the third threshold are determined based on experience. In some embodiments, the first threshold and the second threshold are determined based on experience to be 0.25, and the third threshold is determined based on experience to be 0.40.

[0122] Through this matching strategy, dynamic matching is performed first, and static matching is performed only when the dynamic matching result meets the set threshold conditions. Weighted calculation of matching values ​​is performed only when the static matching result meets the set threshold conditions. If the dynamic matching or static matching results do not meet the set threshold conditions, no subsequent calculation will be performed. In this way, the amount of calculation can be reduced and the detection efficiency can be improved.

[0123] In some other embodiments, step S4 may include:

[0124] The static space image corresponding to the image frame is matched with the static space template image to obtain a second matching value; if the second matching value is less than or equal to the second threshold, it is determined that there is no drone target in the image frame; if the second matching value is greater than the second threshold, the dynamic area image corresponding to the image frame is matched with the dynamic area template image to obtain a first matching value; if the first matching value is less than or equal to the first threshold, it is determined that there is no drone target in the image frame; if the first matching value is greater than the first threshold, the first matching value and the second matching value are weightedly calculated to obtain a third matching value; if the third matching value is less than or equal to the third threshold, it is determined that there is no drone target in the image frame; if the third matching value is greater than the third threshold, it is determined that there is a drone target in the image frame.

[0125] Through this matching strategy, static matching is performed first, and dynamic matching is performed only when the static matching result meets the set threshold conditions. Weighted calculation of matching values ​​is performed only when the dynamic matching result meets the set threshold conditions. If the static matching or dynamic matching result does not meet the set threshold conditions, no subsequent calculation will be performed. In this way, the amount of calculation can be reduced and the detection efficiency can be improved.

[0126] In some embodiments, a comprehensive judgment of image similarity may be performed based on data similarity at three levels: image level (pixel information), manual feature level (contour, texture, etc.), and deep feature (feature extraction based on convolutional neural network).

[0127] For example, in some other embodiments, step S4 may include:

[0128] Extracting the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area image; matching the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area image corresponding to the image frame with the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area template image respectively to obtain a first dynamic matching value, a second dynamic matching value and a third dynamic matching value; performing weighted calculation on the first dynamic matching value, the second dynamic matching value and the third dynamic matching value to obtain a first matching value;

[0129] It should be understood that the dynamic area image and the dynamic area template image are matched with the same feature type and feature dimension. For example, the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area image can be respectively the dynamic area image itself, the HOF feature of the dynamic area image and the full convolutional network feature of the dynamic area image; correspondingly, the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area template image are respectively the dynamic area template image itself, the optical flow direction information histogram feature of the dynamic area template image and the full convolutional network feature of the dynamic area template image. Of course, the features to be matched are not limited to the above-mentioned types of features.

[0130] Extracting the first static feature, the second static feature and the third static feature of the static space image; matching the first static feature, the second static feature and the third static feature of the static space image corresponding to the image frame with the first static feature, the second static feature and the third static feature of the static space template image respectively to obtain a first static matching value, a second static matching value and a third static matching value; performing weighted calculation on the first static matching value, the second static matching value and the third static matching value to obtain a second matching value;

[0131] It should be understood that the above-mentioned static space image and the static space template image have the same feature type and feature dimension for matching. For example, the first static feature, the second static feature and the third static feature of the static space image can be respectively taken from the static space image itself, the local binary pattern feature of the image and the full convolutional network feature of the image; correspondingly, the first static feature, the second static feature and the third static feature of the static space template image can be respectively taken from the static space template image itself, the local binary pattern feature of the static space template image and the full convolutional network feature of the static space template image;

[0132] The first matching value and the second matching value are weightedly calculated to obtain a third matching value; if the first matching value is greater than the fourth threshold, the second matching value is greater than the fifth threshold, and the third matching value is greater than the sixth threshold, it is determined that there is a drone target in the image frame.

[0133] In the technical solution provided by the above embodiment, the target to be matched and the feature library samples are matched one by one according to the similarity matching method, and the matching results are sorted to obtain the maximum matching value. If the threshold condition of the matching strategy is met, it is determined to be a drone target. The entire matching process includes static space template image matching and dynamic area template image matching, wherein the static space template image matching result is the comprehensive matching result of the first static feature, the second static feature and the third static feature, and the dynamic area template image matching result is the comprehensive matching result of the first dynamic feature, the second dynamic feature and the third dynamic feature, which are integrated according to the following calculation formula:

[0134] (2)

[0135] in, and are the first feature, second feature and third feature matching results of the target to be matched and the feature library sample, respectively. and The acquisition of is calculated according to the zero-mean normalized cross-correlation function; and They are the corresponding weights, which are set according to the requirements, but must meet (In this embodiment, 0.3, 0.3, 0.4 are taken).

[0136] The method provided in the embodiment of the present application has obvious advantages over the existing deep learning target detection method: first, it is simple to operate and does not require complex target labeling and long-term model training. The method provided in the embodiment of the present application only requires a small number of templates; second, it is less affected by scene changes; third, the detection efficiency is high. The method provided in the embodiment of the present application can effectively detect target drones with pixels less than 10×10 in 1920×1280 high-definition images.

[0137] The present application also provides a drone target detection system, including:

[0138] An image sequence acquisition module is used to obtain an image sequence according to the video data to be detected;

[0139] A dynamic area image extraction module is used to extract the dynamic area image corresponding to the image frame in the image sequence;

[0140] A static space image extraction module, used for extracting a static space image corresponding to an image frame in an image sequence;

[0141] The judgment module is used to match the dynamic area image corresponding to the image frame with the dynamic area template image, and to match the static space image corresponding to the image frame with the static space template image. If the matching result meets the preset conditions, it is determined that there is a drone target in the image frame.

[0142] It should be understood that the above-mentioned drone target detection system has the function of implementing the method in the first aspect and any possible implementation thereof. Its functional modules can be implemented by hardware, or by hardware executing corresponding software.

[0143] The embodiment of the present application also provides an electronic device, including: a memory and a processor;

[0144] Memory for storing computer programs;

[0145] The processor is used to call a computer program to execute the above-mentioned UAV target detection method.

[0146] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed on an electronic device, the electronic device implements the above-mentioned method for detecting a drone target.

[0147] An embodiment of the present application also provides a computer program product, including a computer program. When the computer program is run on an electronic device, the electronic device executes the above-mentioned drone target detection method.

[0148] Experiment and analysis

[0149] 1. Experimental setup

[0150] In order to verify the effectiveness of the method provided in the embodiment of the present application, target templates are first constructed according to the drone category, and then the template matching strategy, the scene, size, mode, etc. of the target are compared based on the real flight data of the drone in different scenarios. This experiment mainly uses the DJI Mavic 3 drone and the Phantom 4P drone as the detection objects. It should be noted that in the data set constructed in this experiment, both the platform and the target are in motion.

[0151] 2. Evaluation indicators

[0152] Detection accuracy and false alarm rate are selected as evaluation indicators. Given M Frame continuous video, count the number of target drones that appear N , the number of targets correctly detected by the method provided in the embodiment of the present application is N 0 , the number of targets detected incorrectly is N 1 , the calculation formulas for detection accuracy and false alarm rate are as follows:

[0153] (3)

[0154] (4).

[0155] 3. Ablation Experiment Analysis

[0156] In order to verify the effectiveness of the template matching strategy, an ablation analysis of the multi-template matching strategy was performed. As shown in Table 2, it is a comparison result of 4 different matching methods. It can be seen from the table that the method of comprehensive dynamic matching and static space matching provided in the embodiment of the present application has the highest detection accuracy and the lowest false alarm rate, while the method relying only on dynamic matching has the lowest detection accuracy, and the method without any matching has the highest false alarm rate. Without any matching, it is equivalent to relying only on dynamic target judgment to judge low, slow and small targets. Under the condition of platform motion or other moving targets, the false alarm rate is very high, and relying only on dynamic information matching will be greatly affected by the target's motion state, shooting distance, etc., which is in line with our expected judgment:

[0157] Table 2 Detection results of different template matching strategies .

[0158] 4.4 Detection results in different scenarios

[0159] Based on visible light modal data, the following tests are conducted on the detection results of low, slow, and small targets in different backgrounds such as the sky, mountains, and buildings using the method provided in the embodiments of the present application. Figure 4 As shown in the figure, the box marked is the drone target. The detection results under different backgrounds, including target detection accuracy and false alarm rate, are shown in Table 3.

[0160] It can be seen from Table 3 that good target detection accuracy and false alarm rate can be achieved under different backgrounds: in terms of accuracy, the sky background is the highest at 0.95, and the building background is the lowest at 0.78; in terms of false alarm rate, the sky background false alarm rate is 0, and the building background false alarm rate is 0.11. The results of the detection errors were traced back and analyzed, and the erroneous image frames mainly appeared in the following two situations: ① The target is hovering or the movement changes very little; ② The target is very similar to the background texture. This is in line with the expectations of the method of the present application. Since the method provided in the embodiment of the present application is based on dynamic area extraction, when the above situation occurs, the extracted dynamic information is weak and the target area is easily filtered out:

[0161] Table 3 Detection results under different backgrounds ,

[0162] In addition, the detection results under different backgrounds are related to the presence of suspected interference objects. The sky background is relatively pure, with the least suspected interference objects; while the mountain background can filter out some interference through spatial matching; road vehicles, pedestrians, and building lighting under the building background all become suspected targets. This is consistent with the detection results that the accuracy rate decreases and the false alarm rate increases in the sky, mountain, and building backgrounds.

[0163] 4.5. Detection results of different target sizes

[0164] In order to further analyze the detection results of small pixel targets at large viewing angles by the method provided in the embodiment of the present application, the data is divided according to the pixel ratio, and the detection results of different sizes are compared. As shown in Table 4, the target detection results under different pixel ratios are:

[0165] Table 4 Detection results under different pixel ratios ,

[0166] It can be seen from the table that the method provided in the embodiment of the present application has an accuracy rate of 0.93 and a false alarm rate of 0.03 for small pixel target detection, while the accuracy rate for extremely small pixel target detection can still reach 0.70, and the false alarm rate reaches 0.26. As the pixel size becomes smaller and smaller, the detection performance decreases significantly. On the one hand, this is because the dynamic information extraction of small-sized targets is easily affected, and on the other hand, the fewer pixels that match the spatial information, the lower the matching confidence. It should be noted that the above data are all relative to the image input of the original size of 1920×1280. According to previous analysis, the YOLOV8 algorithm can be trained for learning, but it cannot output valid detection box information. This to a certain extent illustrates the effectiveness of the method provided in the embodiment of the present application for detecting small-sized targets.

[0167] 4.6. Other modal data detection results

[0168] In order to further verify the adaptability of the method provided in the embodiment of the present application, a preliminary verification is performed on infrared modal data. Figure 5 As shown, the ones marked with boxes are drone targets. Through the detection results of infrared modal data in the backgrounds of sky, mountains, buildings, etc., all thresholds in the experiment are consistent with the visible light modal experimental process. From the experimental results, it can be seen that the method of the present application can also adapt well to the detection of low, slow and small targets in infrared modality, and can still obtain good detection effects under different backgrounds. Since infrared modal imaging is related to the temperature of the target, the target imaging is clearer in time periods with large temperature differences such as at night, and in theory it is more suitable to adopt the method provided in the embodiment of the present application.

[0169] 4.7. Typical low, slow and small target detection integrated application

[0170] A typical low-speed, slow, and small target detection scenario was constructed in a typical urban environment, including a test optoelectronic pod (experimental platform) and a target UAV. The UAV was a DJI Mavic 3 equipped with a positioning module to help effectively capture target data.

[0171] The method provided in the embodiment of the present application is deployed on a PC platform, and the environment is Intel(R) Core(TM) i7-8550UCPU @ 1.80GHz 1.99 GHz processor, 8G memory, and win10 operating system. The PC and the optoelectronic pod are connected via a network cable, the focal length of the optoelectronic pod remains unchanged, and the target drone moves randomly within a range of about 200m from the camera. It can be seen from the measured results that the method provided in the embodiment of the present application can effectively detect drone targets under different backgrounds. The real-time operation efficiency is statistically analyzed, and the average processing frame rate (frames per second, fps) is about 59.8, and the real-time performance meets the requirements.

[0172] The detection of low, slow and small targets provides important data support for low-altitude safety supervision and prevention and control. Due to their small size, weak texture, unclear features, complex motion background (mountains, buildings, sky), large background changes, and many interferences, the detection of low, slow and small targets is still a very challenging problem. The method provided in the embodiment of the present application aims at the shortcomings of existing deep learning methods in the detection of extremely small targets in a large field of view, and proposes a low, slow and small target detection method that integrates dynamic area information and static space information, effectively combining the dynamic characteristics and static space characteristics of drone targets to improve the detection performance of extremely small targets in dynamic and large viewing angle scenes. The above data set testing and real scene application verification were carried out. 102 drone video data were collected, and detection experiments were carried out on drone targets of different target sizes and different data modes under different backgrounds such as sky, mountains, and buildings, and preliminary application verification was carried out based on a ground-based fixed photoelectric detection platform. The results show that the effective detection rate of the method provided by the embodiment of the present application for low, slow and small drone targets is 0.81, the false alarm rate is 0.06, and the target detection rate can still reach 0.70 when the pixel ratio is not less than 0.01%. It can adapt to video data of different modes such as visible light and infrared, and can effectively make up for the shortcomings of deep learning methods in detecting extremely small pixels. The experimental results verify the feasibility and effectiveness of fusing dynamic area images and static space images for low, slow and small target detection in complex dynamic large field of view scenes.

[0173] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for detecting an unmanned aerial vehicle target, characterized in that: The method comprises: Obtaining an image sequence according to the video data to be detected; Extracting a dynamic region image corresponding to an image frame in the image sequence; Extracting an image corresponding to the position of the dynamic area image on the image frame as a static space image corresponding to the image frame; Performing template matching based on the dynamic area image and the static space image to determine whether there is a drone target in the image frame includes: Matching the dynamic area image corresponding to the image frame with the dynamic area template image to obtain a first matching value; Matching the static space image corresponding to the image frame with the static space template image to obtain a second matching value; Performing weighted calculation on the first matching value and the second matching value to obtain a third matching value; If the first matching value is greater than the first threshold, the second matching value is greater than the second threshold, and the third matching value is greater than the third threshold, it is determined that there is a drone target in the image frame.

2. The method according to claim 1, characterized in that Extracting the dynamic area image corresponding to the image frame in the image sequence includes: extracting the dynamic area image corresponding to the image frame in the image sequence based on an inter-frame difference method.

3. The method according to claim 2, characterized in that The extracting of the dynamic area image corresponding to the image frame in the image sequence based on the frame difference method includes: For an nth frame image in the image sequence, calculating a difference between the nth frame image and the (n+1)th frame image in the image sequence to obtain a first image; Binarizing the first image to obtain a second image; Performing corrosion and expansion processing on the second image to obtain a third image; Finding contours in the third image to obtain a fourth image marked with each contour; A dynamic area image corresponding to the n-th frame image is obtained based on the fourth image.

4. The method according to claim 3, characterized in that: The step of obtaining a dynamic region image corresponding to the nth frame image based on the fourth image includes: Gaussian filtering is performed on the fourth image to obtain a fifth image, and a dynamic area image corresponding to the n-th frame image is obtained based on the fifth image.

5. The method according to claim 4, characterized in that The step of obtaining a dynamic region image corresponding to the nth frame image based on the fifth image includes: Based on each contour marked in the fifth image, determining in turn whether the size thereof is within a preset range; If the size of the outline is within the preset range, it is retained; if the size of the outline is not within the preset range, it is discarded; The image within the contour retained on the fifth image is used as the dynamic area image corresponding to the n-th frame image.

6. The method according to any one of claims 1 to 5, characterized in that: Matching the dynamic area image corresponding to the image frame with the dynamic area template image to obtain a first matching value includes: Extracting a first dynamic feature, a second dynamic feature, and a third dynamic feature of a dynamic area image corresponding to the image frame; wherein the first dynamic feature, the second dynamic feature, and the third dynamic feature of the dynamic area image are respectively the dynamic area image itself, the optical flow direction information histogram feature of the dynamic area image, and the full convolutional network feature of the dynamic area image; Matching the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area image with the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area template image, respectively, to obtain a first dynamic matching value, a second dynamic matching value and a third dynamic matching value, respectively; wherein the first dynamic feature, the second dynamic feature and the third dynamic feature of the dynamic area template image are respectively the dynamic area template image itself, the optical flow direction information histogram feature of the dynamic area template image and the full convolution network feature of the dynamic area template image; Performing weighted calculation on the first dynamic matching value, the second dynamic matching value, and the third dynamic matching value to obtain a first matching value; Matching the static space image corresponding to the image frame with the static space template image to obtain a second matching value includes: Extracting a first static feature, a second static feature, and a third static feature of a static space image corresponding to the image frame; wherein the first static feature, the second static feature, and the third static feature of the static space image are respectively the static space image itself, a local binary pattern feature of the static space image, and a full convolutional network feature of the static space image; Matching the first static feature, the second static feature and the third static feature of the static space image with the first static feature, the second static feature and the third static feature of the static space template image, respectively, to obtain a first static matching value, a second static matching value and a third static matching value, respectively; wherein the first static feature, the second static feature and the third static feature of the static space template image are respectively the static space template image itself, the local binary pattern feature of the static space template image and the full convolution network feature of the static space template image; The first static matching value, the second static matching value, and the third static matching value are weightedly calculated to obtain a second matching value.

7. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer programs; The processor is configured to call the computer program to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on an electronic device, the electronic device implements the method according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed on an electronic device, the electronic device implements the method according to any one of claims 1 to 6.

Images