Unmanned aerial vehicle cluster overall target identification method based on deep learning

Abstract

The invention discloses an unmanned aerial vehicle cluster overall target identification method based on deep learning, and belongs to the field of target detection, and the method comprises the following steps: S1, collecting and screening original images of an unmanned aerial vehicle cluster; s2, identifying the overall position of the unmanned aerial vehicle cluster; s3, obtaining an overall image of the unmanned aerial vehicle cluster; s4, identifying the individual position of the unmanned aerial vehicle; and S5, combining the unmanned aerial vehicle cluster overall position identification result and the unmanned aerial vehicle individual position identification result to obtain the position of the unmanned aerial vehicle individual on the unmanned aerial vehicle cluster original image. According to the unmanned aerial vehicle cluster overall target recognition method based on deep learning, overall target recognition is introduced under the condition that unmanned aerial vehicle cluster individuals are distributed in a concentrated mode, unmanned aerial vehicle individual target recognition is conducted based on the overall target recognition, the overall target recognition result and the individual target recognition result are combined, and the overall target recognition result and the individual target recognition result are combined. According to the invention, on the premise of simplifying the steps, the accuracy of position identification of the unmanned aerial vehicle individuals in the cluster is improved.

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a method for overall target recognition of a drone cluster based on deep learning. Background Art

[0002] With the rapid development of modern technology, drone swarms are gradually being used in all aspects of people's daily lives, among which aerial drone swarm combat is a form of modern warfare. Target identification of aerial drone swarms helps to accurately locate and track drones, determine the changes in individual drone postures, and study the intentions of drone swarms, improve the safety of airspace management, enhance military defense capabilities, and ensure public safety. It plays an important role in maintaining air order, improving mission execution efficiency, and promoting the advancement of drone technology.

[0003] At present, the target recognition of aerial drone clusters is classified as the field of aerial small target recognition. And the small target in the image is defined as the relative area of ​​all target instances in the same category according to the relative scale, that is, the median ratio of the bounding box area to the image area is between 0.08% and 0.58%. The method of identifying aerial small targets based on visual images has difficulties such as weak light signals, complex recognition environment, and high real-time requirements.

[0004] In order to solve the above problems, the prior art has conducted research in the following directions:

[0005] For example, in "UAV optical small target detection algorithm based on dual-modal image fusion", Chang Kaixuan, Huang Jianhua, Sun Xiyan, etc. proposed a dual-channel image fusion detection algorithm based on back-end fusion in order to enhance the feature information of small target images and improve the accuracy of small target detection by fusing infrared and visible light images, and designed a back-end fusion detection framework (BFDN). This method is more complicated because it needs to match the visible light image and infrared image at the corresponding time to ensure that the corresponding images are as similar as possible.

[0006] In "Real-time Target Detection Based on Improved YOLOv3 Network", Sun Jia, Guo Dabo, Yang Tiantian, etc. disclosed that based on the YOLOv3 network, the appropriate initial cluster center is selected by setting a threshold to improve the selection effect of the detection anchor frame; for medium and small targets, feature maps of different scales are added to improve the detection accuracy. However, when the area occupied by the target is too small relative to the size of the original image, adjusting the initial cluster center of the anchor frame will only improve the recognition accuracy of small targets slightly.

[0007] In his article “Study on Small Target Detection of UAVs in Low-Altitude Airspace of Airports Based on Contrastive Learning”, Yuanzi Hao disclosed the use of contrastive learning to improve the feature differentiation of image background and target. Specifically, in the process of obtaining positive and negative samples, the target image and background image in the sample image are respectively intercepted by randomly cutting the image, and the sample image is enhanced and input into the contrastive learning network for training. However, this method is only for backgrounds with weak textures, such as clear blue sky backgrounds. Even after data enhancement, the features of negative samples are still small. Summary of the invention

[0008] To solve the above problems, the present invention provides a method for overall target recognition of a drone cluster based on deep learning. When the individuals in the drone cluster are concentrated, overall target recognition is introduced, and individual target recognition of the drones is performed based on the overall target recognition. The overall target recognition result and the individual target recognition result are combined to obtain the final result. While simplifying the steps, the accuracy of position recognition of individual drones in the cluster is improved.

[0009] To achieve the above object, the present invention provides a method for identifying an overall target of a drone cluster based on deep learning, comprising the following steps:

[0010] S1. Collecting and screening original images of drone clusters: Collecting original images, and screening the original images that meet the set relative relationship between the overall distribution range of the drone clusters and the original image size, as the original images of the drone clusters;

[0011] S2, calibrate the overall target of the drone cluster in the original image of the drone cluster and identify the overall position of the drone cluster;

[0012] S3, based on the overall position recognition result of the drone cluster obtained in step S2, the original image of the drone cluster is cropped along the overall distribution range of the drone cluster to obtain an overall image of the drone cluster;

[0013] S4, performing individual target calibration on the overall image of the drone cluster cropped in step S3 to identify the individual positions of the drones;

[0014] S5. Combining the overall position recognition result of the drone cluster obtained in step S2 and the individual position recognition result of the drone obtained in step S4, the final target recognition result is obtained according to the image cropping and transformation relationship, and the position of the individual drone on the original image of the drone cluster is obtained.

[0015] Preferably, step S1 specifically includes the following steps:

[0016] S11, using the visual sensor of the ground base station to shoot the drone cluster in the air to obtain the original image;

[0017] S12, randomly select image P from the original image i , i=1,2,···,10%N, N is the total number of original images;

[0018] S13, record the randomly selected i Image P i In the pixel coordinate system, the minimum lateral position u in the drone cluster min , maximum lateral position u max , the minimum vertical position v min , the maximum value of the vertical coordinate v max The drone individual uses the vertex coordinates (u min ,v min )、(u max ,v min )、(u min ,v max )、(u max ,v max ) to determine the distribution range of the drone cluster and calculate the area ratio of the rectangular detection box to the original image. Where S is the area of ​​the original image;

[0019] S14. Calculate the mean of the area ratio of the selected image If β<0.3, it is determined that the set relative relationship is satisfied, and the selected image that satisfies the set relative relationship is regarded as the original image of the drone cluster.

[0020] Preferably, step S2 specifically includes the following steps:

[0021] S21, divide the acquired drone cluster original images into training set P according to the set ratio train With the test set P test ;

[0022] S22, for the training set P train For each original image of the drone cluster in the image coordinate system, search for the drone cluster in the original image with the minimum lateral position u min , maximum lateral position u max , the minimum vertical position v min , the maximum value of the vertical coordinate v max The drone individual uses the vertex coordinates (u min ,v min )、(u max ,v min )、(u min ,v max )、(u max ,v max)’s rectangular detection frame frames the drone cluster, and the location of the rectangular detection frame is regarded as the overall location of the drone cluster;

[0023] S23, using the overall position of the drone cluster obtained to train the YOLOv5 deep learning algorithm model;

[0024] S24, the test set P test Input the YOLOv5 deep learning algorithm model trained in step S23, and output the overall image result of the drone cluster represented by a rectangular detection frame.

[0025] Preferably, in step S24, the output result (u min ,u max ,v min ,v max ), representing the four vertices of the rectangular detection box (u min ,v min )、(u max ,v min )、(u min ,v max )、(u max ,v max ).

[0026] Preferably, in step S3, according to (u min ,u max ,v min ,v max ) to crop the image, retain the rectangular detection frame part, and obtain the cropped image P where the drone cluster accounts for a large proportion of the entire image crop , and the image P crop As an overall image of the drone cluster.

[0027] Preferably, step S4 specifically includes the following steps:

[0028] S41, according to the set ratio, the image P crop Divide into training set With the test set

[0029] S42, training set Use the images in the image to calibrate individual targets of UAVs;

[0030] S43, using the image calibrated in step S42 to train the YOLOv5 deep learning algorithm model;

[0031] S44, the test set Input the YOLOv5 deep learning algorithm model to identify the individual positions of drones in the image.

[0032] Preferably, in step S42, all rectangular detection frames that frame the outer contour of the individual drone are searched, the smallest rectangular detection frame among all the rectangular detection frames is found, and the individual drone is calibrated using the smallest rectangular detection frame.

[0033] Preferably, the YOLOv5 deep learning algorithm models described in step S23 and step S43 include an input structure, a backbone network, a neck structure and a head structure arranged in sequence, wherein the input structure is used for image preprocessing, the backbone network is used for feature extraction, the neck structure is used for feature fusion, and the head structure is used to output detection results.

[0034] Preferably, in step S5, assuming that the original image P of the drone cluster i After the overall calibration of the drone cluster, the output rectangular detection box is After cropping, we get image P i ',P i 'After the drone individual calibration, the image P i 'The recognition result corresponding to the drone individual j is a rectangular detection box The image P is obtained by combining the results of the overall target recognition of the drone cluster and the individual target recognition of the drone. i The position of UAV individual j is

[0035] The present invention has the following beneficial effects:

[0036] When the individuals in a drone cluster are concentrated, overall target recognition is introduced, and individual drone target recognition is performed based on the overall target recognition. The final result can be obtained by combining the overall target recognition result with the individual target recognition result. This improves the accuracy of position recognition of individual drones in the cluster while simplifying the steps.

[0037] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of a method for overall target recognition of a drone cluster based on deep learning of the present invention;

[0039] Figure 2 This is an original image of an example of an aerial drone cluster in a simulation experiment of the present invention;

[0040] Figure 3 This is a diagram of the overall target recognition result of the drone cluster in the simulation experiment of the present invention;

[0041] Figure 4 This is a diagram showing the result of identifying individual drone targets in a drone cluster in a simulation experiment of the present invention;

[0042] Figure 5 This is a diagram of the final target recognition results of the drone cluster in the simulation experiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages disclosed in the embodiments of the present invention clearer, the embodiments of the present invention are further described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present invention and are not intended to limit the embodiments of the present invention. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions.

[0044] It should be noted that the terms "include", "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or devices.

[0045] Like reference numerals and letters denote similar items in the following drawings, and thus, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0046] Taking into account the accuracy and convenience requirements of aerial drone cluster target recognition, the present invention is designed as follows: Figure 1 As shown, a method for overall target recognition of a drone cluster based on deep learning includes the following steps:

[0047] S1. Collecting and screening original images of drone clusters: Collecting original images, and screening the original images that meet the set relative relationship between the overall distribution range of the drone clusters and the original image size, as the original images of the drone clusters;

[0048] Step S1 specifically includes the following steps:

[0049] S11. Use the visual sensor of the ground base station to capture the drone cluster in the air to obtain the original image. The image of the drone cluster in the air at a long distance generally has the characteristics of the drone cluster occupying a small area of ​​the entire image and the individual drones being densely distributed;

[0050] S12, randomly select image P from the original image i , i=1,2,···,10%N, N is the total number of original images;

[0051] S13, record the randomly selected i-th image P i In the pixel coordinate system, the minimum lateral position u in the drone cluster min , maximum lateral position u max , the minimum vertical position v min , the maximum value of the vertical coordinate v max The drone individual uses the vertex coordinates (u min ,v min )、(u max ,v min )、(u min ,v max )、(u max ,v max ) to determine the distribution range of the drone cluster and calculate the area ratio of the rectangular detection box to the original image. Where S is the area of ​​the original image;

[0052] S14. Calculate the mean of the area ratio of the selected image If β<0.3, it is determined that the set relative relationship is satisfied, and the selected image that satisfies the set relative relationship is regarded as the original image of the drone cluster.

[0053] S2, calibrate the overall target of the drone cluster in the original image of the drone cluster and identify the overall position of the drone cluster;

[0054] Step S2 specifically includes the following steps:

[0055] S21, divide the acquired drone cluster original images into training set P according to the set ratio train With the test set P test ;

[0056] S22, for the training set P train For each original image of the drone cluster in the image coordinate system, search for the drone cluster in the original image with the minimum lateral position u min , maximum lateral position u max , the minimum vertical position v min , the maximum value of the vertical coordinate v max The drone individual uses the vertex coordinates (u min ,v min )、(u max ,v min )、(u min ,v max )、(u max ,v max )’s rectangular detection frame frames the drone cluster, and the location of the rectangular detection frame is regarded as the overall location of the drone cluster;

[0057] S23, using the overall position of the drone cluster obtained to train the YOLOv5 deep learning algorithm model;

[0058] S24, the test set P test Input the YOLOv5 deep learning algorithm model trained in step S23, and output the overall image result of the drone cluster represented by a rectangular detection frame.

[0059] In step S24, the result (u min ,u max ,v min ,v max ), representing the four vertices of the rectangular detection box (u min ,v min )、(u max ,v min )、(u min ,v max )、(u max ,v max ).

[0060] S3, based on the overall position recognition result of the drone cluster obtained in step S2, the original image of the drone cluster is cropped along the overall distribution range of the drone cluster to obtain an overall image of the drone cluster;

[0061] In step S3, according to (u min ,u max ,v min ,v max ) to crop the image, retain the rectangular detection frame part, and obtain the cropped image P where the drone cluster accounts for a large proportion of the entire image crop , and the image P crop As an overall image of the drone cluster.

[0062] S4, performing individual target calibration on the overall image of the drone cluster cropped in step S3 to identify the individual positions of the drones;

[0063] Step S4 specifically includes the following steps:

[0064] S41, according to the set ratio, the image P crop Divide into training set With the test set

[0065] S42, training set Use the images in the image to calibrate individual targets of UAVs;

[0066] In step S42, all rectangular detection frames that frame the outer contour of the individual drone are searched, the smallest rectangular detection frame among all the rectangular detection frames is found, and the individual drone is calibrated using the smallest rectangular detection frame.

[0067] S43, using the image calibrated in step S42 to train the YOLOv5 deep learning algorithm model;

[0068] S44, the test set Input the YOLOv5 deep learning algorithm model to identify the individual positions of drones in the image.

[0069] The YOLOv5 deep learning algorithm models described in step S23 and step S43 include an input structure, a backbone network, a neck structure and a head structure which are arranged in sequence, wherein the input structure is used for image preprocessing, the backbone network is used for feature extraction, the neck structure is used for feature fusion, and the head structure is used to output detection results, which has the advantages of fast detection speed, lightweight model and simple training.

[0070] S5. Combining the overall position recognition result of the drone cluster obtained in step S2 and the individual position recognition result of the drone obtained in step S4, the final target recognition result is obtained according to the image cropping and transformation relationship, and the position of the individual drone on the original image of the drone cluster is obtained.

[0071] The relative position of the objects in the image is expressed in a pixel coordinate system. In the pixel coordinate system, the upper left corner of the image is the origin (0,0), the horizontal axis (u axis) increases from left to right, and the vertical axis (v axis) increases from top to bottom. Based on this, in step S5, it is assumed that the original image P of the drone cluster is i After the overall calibration of the drone cluster, the output rectangular detection box is After cropping, we get image P i ',P i 'After the drone individual calibration, the image P i 'The recognition result corresponding to the drone individual j is a rectangular detection box The image P is obtained by combining the results of the overall target recognition of the drone cluster and the individual target recognition of the drone. i The position of UAV individual j is

[0072] Simulation experiment

[0073] In this simulation experiment, the Figure 2 As shown, a drone cluster consisting of 56 drones is in a triangular formation with a relatively concentrated distribution. There are 99 images in total, and the resolution of each image is 1920*1080. The average area ratio of 10 randomly selected images is calculated. The result is 0.0094, which meets the requirements of overall target recognition, so overall target recognition can be performed on the selected 10 images.

[0074] Table 1 Area ratio of selected images

[0075]

[0076]

[0077] The selected images are processed using the UAV cluster overall target recognition method based on deep learning described in the present invention, and the results are as follows: Figure 3-Figure 5 shown.

[0078] Table 2 Target recognition results of example images using the algorithm of the present invention

[0079]

[0080]

[0081]

[0082] Combination Figure 3-Figure 5 As can be seen from Table 2, due to the fact that some drones are cropped at the border of the rectangular detection frame and some drones are close to each other in the picture, 54 drones are finally identified in the picture, which is within a reasonable range.

[0083] However, if we directly perform drone individual calibration on the same drone cluster image and use the YOLOv5 model for training and drone individual recognition, the model will fail to recognize the individual drones because the pixels occupied by the drone targets are too small.

[0084] Thereby verifying the effectiveness of the design of the present invention.

[0085] Therefore, the present invention adopts the above-mentioned drone cluster overall target recognition method based on deep learning. When the individuals in the drone cluster are concentrated, overall target recognition is introduced, and individual drone target recognition is performed based on the overall target recognition. The overall target recognition result and the individual target recognition result are combined to obtain the final result. While simplifying the steps, the accuracy of position recognition of individual drones in the cluster is improved, and it has a broad application prospect.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for overall target recognition of drone clusters based on deep learning, characterized by: The following steps are involved: S1. Collecting and screening original images of drone clusters: Collecting original images, and screening the original images that meet the set relative relationship between the overall distribution range of the drone clusters and the original image size, as the original images of the drone clusters; S2, calibrate the overall target of the drone cluster in the original image of the drone cluster and identify the overall position of the drone cluster; S3, based on the overall position recognition result of the drone cluster obtained in step S2, the original image of the drone cluster is cropped along the overall distribution range of the drone cluster to obtain an overall image of the drone cluster; S4, performing individual target calibration on the overall image of the drone cluster cropped in step S3 to identify the individual positions of the drones; S5. Combining the overall position recognition result of the drone cluster obtained in step S2 and the individual position recognition result of the drone obtained in step S4, the final target recognition result is obtained according to the image cropping and transformation relationship, and the position of the individual drone on the original image of the drone cluster is obtained.

2. The method for identifying overall targets of drone clusters based on deep learning according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11, using the visual sensor of the ground base station to shoot the drone cluster in the air to obtain the original image; S12, randomly select image P from the original image i , i=1,2,···,10%N, N is the total number of original images; S13, record the randomly selected i-th image P i In the pixel coordinate system, the minimum lateral position u in the drone cluster min , maximum lateral position u max , the minimum vertical position v min , the maximum value of the vertical coordinate v max The drone individual uses the vertex coordinates (u min ,v min )、(u max ,v min )、(u min ,v max )、(u max ,v max ) to determine the distribution range of the drone cluster and calculate the area ratio of the rectangular detection box to the original image. Where S is the area of ​​the original image; S14. Calculate the mean of the area ratio of the selected image If β<0.3, it is determined that the set relative relationship is satisfied, and the selected image that satisfies the set relative relationship is regarded as the original image of the drone cluster.

3. The method for overall target recognition of a drone cluster based on deep learning according to claim 2 is characterized in that: Step S2 specifically includes the following steps: S21, divide the acquired drone cluster original images into training set P according to the set ratio train With the test set P test ; S22, for the training set P train For each original image of the drone cluster in the image coordinate system, search for the drone cluster in the original image with the minimum lateral position u min , maximum lateral position u max , the minimum vertical position v min , the maximum value of the vertical coordinate v max The individual drones use vertex coordinates (u min ,v min )、(u max ,v min )、(u min ,v max )、(u max ,v max )’s rectangular detection frame frames the drone cluster, and the location of the rectangular detection frame is regarded as the overall location of the drone cluster; S23, using the overall position of the drone cluster obtained to train the YOLOv5 deep learning algorithm model; S24, the test set P test Input the YOLOv5 deep learning algorithm model trained in step S23, and output the overall image result of the drone cluster represented by a rectangular detection frame.

4. The method for identifying overall targets of a drone cluster based on deep learning according to claim 3 is characterized in that: In step S24, the result (u min ,u max ,v min ,v max ), representing the four vertices of the rectangular detection box (u min ,v min )、(u max ,v min )、(u min ,v max )、(u max ,v max ).

5. The method for overall target recognition of a drone cluster based on deep learning according to claim 4 is characterized in that: In step S3, according to (u min ,v max ,u min ,v max ) to crop the image, retain the rectangular detection frame part, and obtain the cropped image P where the drone cluster accounts for a large proportion of the entire image crop , and the image P crop As an overall image of a drone cluster.

6. The method for overall target recognition of a drone cluster based on deep learning according to claim 5, characterized in that: Step S4 The specific steps include: S41, according to the set ratio, the image P crop Divide into training set With the test set S42, training set Use the images in the image to calibrate individual targets of UAVs; S43, using the image calibrated in step S42 to train the YOLOv5 deep learning algorithm model; S44, the test set Input the YOLOv5 deep learning algorithm model to identify the individual positions of drones in the image.

7. The method for overall target recognition of a drone cluster based on deep learning according to claim 6, characterized in that: In step S42, all rectangular detection frames that frame the outer contour of the individual drone are searched, the smallest rectangular detection frame among all the rectangular detection frames is found, and the individual drone is calibrated using the smallest rectangular detection frame.

8. The method for identifying overall targets of drone clusters based on deep learning according to claim 7 is characterized in that: The YOLOv5 deep learning algorithm models described in step S23 and step S43 include an input structure, a backbone network, a neck structure and a head structure arranged in sequence, wherein the input structure is used for image preprocessing, the backbone network is used for feature extraction, the neck structure is used for feature fusion, and the head structure is used to output detection results.

9. The method for identifying overall targets of drone clusters based on deep learning according to claim 8, characterized in that: In step S5, assuming that the original image P of the drone cluster i After the overall calibration of the drone cluster, the output rectangular detection box is After cropping, we get image P i ',P i 'After the drone individual calibration, the image P i 'The recognition result corresponding to the drone individual j is a rectangular detection box The image P is obtained by combining the results of the overall target recognition of the drone cluster and the individual target recognition of the drone. i The position of UAV individual j is

Images