Multi-scale cascade aerial photography target detection method and system based on deep neural network

Abstract

The present application discloses a multi-scale cascade aerial target detection method and system based on a deep neural network. The first feature map and the third feature map are respectively input into a large-scale detection head and a small-scale detection head to obtain small target object prediction information and large target object prediction information, respectively. The corresponding small target object prediction state and large target object prediction state in the target array are updated according to the small target object prediction information and the large target object prediction information, and the large target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array; based on the updated target array, the loss function of each target object in the target array is calculated, and the target objects not predicted by the large-scale detection head and the small-scale detection head are reversely processed to obtain small target object detection information and large target object detection information, respectively. The technical problems such as repeated prediction of aerial image samples, uneven target distribution and missed detection, low recall rate and long detection time are solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and specifically to a multi-scale cascade aerial target detection method and system based on a deep neural network. Background Art

[0002] Object detection is one of the hot research directions in computer vision and is widely used in video surveillance, criminal tracking, autonomous driving, and cancer detection. Aerial object detection is an important aspect of object detection. Aerial images are mainly taken from the air by remotely controlling or automatically using equipment such as drones and satellites. Aerial images have the characteristics of wide viewing angle and high resolution, which can clearly show geographical features. Because they are less affected by terrain, they can accommodate a large amount of target information. Compared with equipment such as satellites, aerial images have the advantages of low image acquisition cost and high resolution. At present, object detection based on aerial images has been widely used in military and civilian fields. For example, computer vision algorithms are used in drones to detect and track targets on the entire battlefield and achieve precision strikes. In the civilian field, the detection technology of aerial targets is used in dangerous areas to identify and locate victims. It can also be applied to urban exploration, land planning, and the protection of wild animals in natural ecological areas. Most aerial image application fields require the identification of single target information to effectively help perceive the scene and assist in formulating strategies, so as to complete the detection tasks of specific scenes.

[0003] Object detection algorithms have achieved good generalization capabilities on natural image object detection datasets and solved many problems in practical object detection applications. These traditional detection algorithms can simultaneously locate and classify various objects (such as cars, airplanes, ships, bridges, and pedestrians) in aerial images. However, most natural image object detection algorithms will experience a sharp drop in detection performance when directly migrated to aerial image datasets, which is specifically manifested in problems such as missed detection, false detection, and misclassification. Compared with natural images, aerial images have the following challenges: (1) Aerial images have a larger resolution. The input size of detectors for natural images is generally 600*600 pixels, while the images in aerial datasets are generally 400 to 2000 pixels. Therefore, aerial images cannot be directly used for such object detectors. (2) The object size is small. Aerial images are generally taken at high altitudes with a wide field of view, so the proportion of objects in the generated images is relatively small. (3) Aerial background area interference. Because aerial images have a large resolution and uneven distribution of objects, foreground object detection will be affected by the interference background.

[0004] Object detection algorithms have been widely studied in the past decade, and various types of detectors have been proposed, such as Faster RCNN, YOLO, SSD and CornerNet. The two-stage detection network represented by the RCNN series has the characteristics of high accuracy but long running time. First, a large number of candidate boxes are generated using algorithms such as select search and RPN, and then some candidate boxes with higher scores are selected, and features are extracted using methods such as ROI Pooling. Finally, the features of the cropped area are classified and the positions of the candidate boxes are regressed. The one-stage detection network represented by the YOLO series has the characteristics of fast speed but poor detection performance. This type of method is an end-to-end network, that is, a series of anchor points are generated on the feature map, and then each anchor point is classified and regressed. The key point-based object detection algorithm uses multiple key points to replace the candidate box algorithm. This method extracts features from the input image through a deep convolutional neural network, and then passes it through a specially designed pooling layer to generate a heat map of the key point distribution, thereby avoiding setting too many hyperparameters. Key point-based object detection has become a research hotspot and mainstream algorithm for current object detection. The key point positions of the target contain rich semantic information. For example, the four extreme points of the target are at the edge of the target and the background, and their feature differences are quite obvious. Combined with the positioning information of the center point, a target frame is determined by three points, making the prediction of the entire candidate frame of the target more accurate and convenient, and the detection accuracy is also higher.

[0005] In aerial object detection, most algorithms are improved based on traditional one-stage and two-stage algorithms and applied to the aerial field. For example, R2CNN designs a multi-scale ROIPooling module to extract the features of candidate boxes for the feature extraction module in Faster RCNN. SCRDet adds channel attention mechanism and pixel attention mechanism on the basis of R2CNN to improve the effect of small object detection. Guiding Vertex integrates multi-scale features and introduces background attention mechanism to improve the accuracy of object detection in cluttered background. In aerial object detection, scale change is one of the main challenges faced by large-scale aerial images. Existing object detection pipelines usually detect objects of different scales at multiple scale layers. However, traditional detection methods based on multi-scale density prediction will lead to repeated detection of the same object. Due to the sparsity of aerial images, the same settings of multi-scale branches are generally used, which makes the predictions repeated for samples that are easy to detect and classify, and at the same time makes the recall rate low, which leads to increased detection time. In addition, some difficult-to-identify samples, such as small objects, are easily missed. Due to the uneven distribution and small size of targets in aerial images, these detection algorithms still have a large number of missed detection problems. Summary of the invention

[0006] The purpose of the embodiments of the present application is to propose a multi-scale cascade aerial target detection method and system based on a deep neural network, so as to solve the technical problems in the detection algorithms in the prior art caused by repeated prediction of aerial image samples, uneven target distribution, and small size, such as a large number of missed detections, low recall rate and long detection time.

[0007] In a first aspect, an embodiment of the present application provides a multi-scale cascade aerial target detection method based on a deep neural network, the method comprising:

[0008] Based on the deep neural network, the aerial pictures are converted into the first feature map and the third feature map respectively;

[0009] Constructing a target array recording the target objects predicted by the first feature map and the third feature map;

[0010] Input the first feature map and the third feature map into the large-scale detection head and the small-scale detection head of the deep neural network respectively to obtain small target object prediction information and large target object prediction information respectively, and update the corresponding small target object prediction state and large target object prediction state in the target array according to the small target object prediction information and the large target object prediction information respectively, wherein the large target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array;

[0011] Based on the updated target array, the loss function of each target object in the target array is calculated, and the target objects not predicted by the large-scale detection head and the small-scale detection head are reversely processed to obtain small target object detection information and large target object detection information respectively;

[0012] The small target object detection information and the large target object detection information are cascaded and fine-tuned and then fused to obtain the target detection results of the aerial image.

[0013] In some embodiments, the first feature map and the second feature map are respectively input into the detection heads corresponding to the multi-scales, a cascade operation is performed on the multi-scale detection, and the detection results of the features of each layer are used to improve the detection accuracy of the entire network.

[0014] In some embodiments, the length and position of the target array correspond one-to-one to the number and order of the target objects to be predicted in the annotation file. The target array is represented as: ispre=[0,0,…,0,0], and the formula of the loss function for each target object in the target array is represented as:

[0015]

[0016] Among them, λ1=0.8, λ2=1.5 represent the hyperparameters of the detection head, lbox represents the IOU loss function, ispre represents the target array, 0 represents the state where the target object to be predicted is not predicted, and 1 represents the state where the target object to be predicted is predicted.

[0017] In this method, for the target objects that have been detected, a weight of λ1=0.8 is used to reduce their losses when calculating the loss function. For other undetected targets, a weight of λ2=1.5 is used to increase their costs when calculating the losses. This is to weaken the processing of detected targets and strengthen the cost penalty of undetected targets, so that the network cascade optimization can be better performed.

[0018] In some embodiments, small target object prediction information is obtained by the following sub-steps:

[0019] The first feature map is input into the large-scale detection head, and the original prediction box (x, y, w, h) and classification confidence (C S100 , C S101 , C S102 , …, C S10k ) and the corresponding category, where x and y represent the center coordinates of the original prediction box, w and h represent the width and height of the original prediction box, respectively, and k represents the number of small target objects to be predicted;

[0020] Calculate the IOU threshold of all small target objects, and match the original predicted box with the actual box one-to-one. If the IOU threshold is greater than 0.5, the small target object prediction information is obtained.

[0021] In some embodiments, the classification confidence of a single small target object is obtained by sorting all the confidences of the single small target object from large to small, and obtaining the top-1 confidence of the single small target object as the classification confidence score C of the single small target object. s1xx .

[0022] In some specific embodiments, the small target object detection information and the large target object detection information are input into the Fine-Tune-Graph for cascade fine-tuning operation to obtain a fine-tuned prediction frame of the target object, and the fine-tuned prediction frame and the original prediction frame are fused and filtered to obtain the target detection result of the aerial image; wherein the cascade fine-tuning operation specifically includes:

[0023] The target objects of the same category in the small target object detection information and the large target object detection information are aggregated to obtain a set of target objects of the same category G i ;

[0024] The set G iThe classification confidence of the target objects of the same category is calculated by the maximum or average confidence fusion to obtain the fusion confidence value C of the target objects of the same category. i * , where the fusion confidence value C i * =max(C S100 , C S101 , ..., C S10k ) or, fusion confidence value C i * =avg(C S100 , C S101 , ..., C S10k );

[0025] The fusion confidence value C i * Multiply it by the coordinates of the original prediction box (x, y, w, h). The specific formula is:

[0026] x′=x*C o *[1±(C i * *C total )]

[0027] y′=y*C o *[1±(C i * *C total )]

[0028] w′=w*C o *[1±(C i * *C total )]

[0029] h′=h*C o *[1±(C i * *C total )]

[0030] Among them, x′ and y′ represent the center coordinates of the fine-tuned prediction box, w′ and h′ represent the width and height of the fine-tuned prediction box, respectively. o represents the original confidence value, C total It represents the sum of the confidences of all prediction boxes. When the original confidence is greater than 0.5, it is “+”; when the original confidence is less than 0.5, it is “-”.

[0031] In this method, after the target objects of the same category are determined, the classification confidence is used for fusion calculation in order to weight the original prediction box and obtain more accurate target object detection results.

[0032] In some embodiments, the method further comprises:

[0033] Based on the deep neural network, the aerial pictures are converted into the second feature map;

[0034] Inputting the second feature map into a mesoscale detection head of a deep neural network, outputting the middle target object prediction information of the mesoscale detection head, and further updating the corresponding middle target object prediction state in the target array according to the middle target object prediction information, wherein the large target object prediction information does not change the target prediction state corresponding to the middle target object prediction information in the target array, and the middle target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array;

[0035] Based on the updated target array, the loss function of each target object in the target array is calculated, and the target objects not predicted by the large-scale detection head, the medium-scale detection head, and the small-scale detection head are reversely processed to obtain small target object detection information, medium target object detection information, and large target object detection information, respectively;

[0036] The small target object detection information, medium target object detection information and large target object detection information are cascaded and fine-tuned and then fused to obtain the target detection results of the aerial image.

[0037] In this method, the multi-scale detection heads of aerial target detection are cascaded and optimized to improve the accuracy of repeated prediction and detection networks. Usually, three or more detection heads are used to generate candidate boxes, and then the cascade concept is used to predict objects of different scales as a whole. Finally, all detection boxes are fused for classification and position regression.

[0038] In some embodiments, the deep neural network includes a residual extraction feature block composed of a convolution layer and a pooling layer, and the residual extraction feature block obtains the first feature map and the third feature map through top-down forward calculation.

[0039] In a second aspect, an embodiment of the present application provides a multi-scale cascade aerial target detection system based on a deep neural network, the system comprising:

[0040] A feature map extraction module: configured to convert the aerial image into a first feature map and a third feature map respectively based on a deep neural network;

[0041] A target array module: configured to construct a target array recording target objects predicted by the first feature map and the third feature map;

[0042] An update target array module is configured to input the first feature map and the third feature map into a large-scale detection head and a small-scale detection head of a deep neural network respectively to obtain small target object prediction information and large target object prediction information respectively, and update the corresponding small target object prediction state and large target object prediction state in the target array according to the small target object prediction information and the large target object prediction information respectively, wherein the large target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array;

[0043] A target object detection module is configured to calculate a loss function for each target object in the target array based on the updated target array, and reversely process the target objects not predicted by the large-scale detection head and the small-scale detection head to obtain small target object detection information and large target object detection information, respectively;

[0044] The target detection result module is configured to fuse the small target object detection information and the large target object detection information after cascade fine-tuning operations to obtain the target detection result of the aerial image.

[0045] In some embodiments, the length and position of the target array correspond one-to-one to the number and order of the target objects to be predicted in the annotation file. The target array is represented as: ispre=[0,0,…,0,0], and the formula of the loss function for each target object in the target array is represented as:

[0046]

[0047] Among them, λ1=0.8, λ2=1.5 represent the hyperparameters of the detection head, lbox represents the IOU loss function, ispre represents the target array, 0 represents the state where the target object to be predicted is not predicted, and 1 represents the state where the target object to be predicted is predicted.

[0048] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement a method as described in any implementation method in the first aspect.

[0049] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0050] The multi-scale cascade aerial target detection method and system based on deep neural network provided in the embodiment of the present application, inputs the picture to obtain feature maps of different scales through the backbone network and feature extraction module, firstly, the original feature map is downsampled multiple times to obtain the feature map of the minimum scale, then the feature map of the minimum scale is upsampled by an even multiple, and a convolution layer is used to fuse the feature map of the corresponding scale, which is input into the detector of the corresponding scale for prediction, and the detection results of multiple scales are merged to obtain the final detection result, thereby eliminating repeated prediction samples and solving the problem of missed detection of special targets, which can effectively improve the detection accuracy and efficiency of target detection in aerial images, thereby improving the overall prediction performance of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0052] Figure 1 is a flowchart of an embodiment of a multi-scale cascade aerial target detection method based on a deep neural network according to the present application;

[0053] Figure 2 is a flowchart of another embodiment of a multi-scale cascade aerial target detection method based on a deep neural network according to the present application;

[0054] Figure 3 It is an overall flow chart of the multi-scale cascade aerial target detection method based on deep neural network according to the present application;

[0055] Figure 4 It is a structural schematic diagram of an embodiment of a multi-scale cascade aerial target detection system based on a deep neural network according to the present application;

[0056] Figure 5 This is a vehicle actual test effect diagram of the multi-scale cascade aerial target detection system based on deep neural network according to the present application;

[0057] Figure 6 This is a picture showing the actual test results of a ship using a multi-scale cascade aerial target detection system based on a deep neural network according to the present application;

[0058] Figure 7 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0059] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0060] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0061] Continue to refer Figure 1 , which shows a flowchart 100 of an embodiment of a multi-scale cascade aerial target detection method based on a deep neural network according to the present application. The method comprises the following steps:

[0062] Step S1: based on a deep neural network, convert the aerial image into a first feature map and a third feature map respectively.

[0063] In this embodiment, the first feature feature and the third feature map generation network mainly use a deep neural extraction feature network, which can be seen as a residual extraction block composed of different convolutional layers and pooling layers, and different scale feature extraction modules are used to extract corresponding information. Different feature information is extracted through the backbone network, so that the original aerial image is converted into the first feature map F1 (80*80*128) and the third feature map F3 (20*20*512).

[0064] In some specific embodiments, the input aerial image is passed through the backbone network and feature extraction module to obtain feature maps of different scales. The original aerial image feature map is first downsampled multiple times to obtain a third feature map F3 for prediction of the small-scale detection head. The third feature map F3 is then upsampled by an even multiple and fused into a first feature map F1 and / or a second feature map F2 of the corresponding scale using a convolutional layer.

[0065] Step S2, constructing a target array that records the target objects predicted by the first feature map and the third feature map.

[0066] In this embodiment, a target array ispre is set to record the target objects predicted by the large-scale detection head and the small-scale detection head, and the target array ispre is set according to the order of the target objects of the original aerial pictures in the annotation file, and its array length is consistent with the target number of the target objects in each original aerial picture.

[0067] Step S3, input the first feature map and the third feature map into the large-scale detection head and the small-scale detection head of the deep neural network respectively to obtain small target object prediction information and large target object prediction information, and update the corresponding small target object prediction state and large target object prediction state in the target array according to the small target object prediction information and the large target object prediction information, wherein the large target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array.

[0068] In this embodiment, the first feature map (80*80*128) is input into the large-scale detection head for predicting small target objects to obtain the prediction information of the small target objects of the large-scale detection head, wherein the prediction information includes the original prediction box of the small target object, the classification confidence score and the corresponding category, and the original prediction box of the small target object is matched one-to-one with the real box ground-truth. If the intersection-over-union ratio threshold is greater than 0.5, a small target object in the correct ground-truth is predicted, and the corresponding position of the small target object in the target array ispre is updated to 1, otherwise it is set to 0, and the corresponding category of the predicted correct small target object should also be satisfied.

[0069] The third feature map (20*20*512) is input into the small-scale detection head for predicting the large target object, and the prediction information of the large target object of the small-scale detection head is obtained, wherein the prediction information includes the original prediction box of the large target object, the classification confidence score and the corresponding category, and the original prediction box of the large target object is matched one-to-one with the ground-truth box. If the intersection-union ratio threshold is greater than 0.5, a large target object in the correct ground-truth is predicted, and the third feature map is input into the small-scale detection head to obtain the detection result of the small-scale detection head, wherein the detection result of the small-scale detection head does not change the predicted target position, but only modifies the unpredicted target position in the large-scale detection head, that is, does not modify the array position of the position "1", and only changes the value of the position "0". The corresponding position of the large target object in the target array ispre is updated to 1, otherwise it continues to be set to 0. In addition, the corresponding category of the correctly predicted large target object should also be satisfied at the same time.

[0070] In some specific embodiments, the small target object prediction information is obtained through the following sub-steps:

[0071] The first feature map is input into the large-scale detection head, and the original prediction box (x, y, w, h) and classification confidence (C S100 ,C S101 ,C S102 ,…,C S10k) and the corresponding category, where x and y represent the center coordinates of the original prediction box, w and h represent the width and height of the original prediction box, respectively, and k represents the number of small target objects to be predicted; the classification confidence of a single small target object is obtained by sorting all the confidences of a single small target object from large to small, and obtaining the top-1 confidence of a single small target object as the classification confidence score C of the single small target object. S1xx .

[0072] Calculate the IOU threshold of all small target objects, and match the original prediction box with the actual box one-to-one. If the intersection over union ratio threshold is greater than 0.5, the prediction information of the small target object is obtained. The intersection over union (IOU) represents the intersection over union ratio of the original prediction box and the actual box (Ground-Truth). The larger the IOU threshold, the closer the original prediction box is to the actual box. The IOU threshold greater than 0.5 represents a correct prediction.

[0073] Step S4, based on the updated target array, calculate the loss function of each target object in the target array, and reversely process the target objects not predicted by the large-scale detection head and the small-scale detection head to obtain small target object detection information and large target object detection information respectively.

[0074] In this embodiment, the small target object prediction state and the large target object prediction state of the updated target array are used to reduce the loss of the predicted small target object and / or large target object using a weight of λ1=0.8. For other undetected target objects in the target array, a weight of λ2=1.5 is used to increase their cost when calculating the loss, so that the predicted target objects are weakened while the cost penalty for unpredicted target objects is strengthened. This can better perform cascade optimization of the network, traverse the target prediction state update and target object loss function calculation of the target array, and obtain small target object detection information and / or large target object detection information.

[0075] In some specific embodiments, the length and position of the target array correspond one-to-one to the number and order of the target objects to be predicted in the annotation file. The target array is expressed as: ispre=[0,0,…,0,0], and the formula of the loss function of each target object in the target array is expressed as:

[0076]

[0077] Among them, λ1=0.8, λ2=1.5 represent the hyperparameters of the detection head, lbox represents the IOU loss function, ispre represents the target array, 0 represents the state where the target object to be predicted is not predicted, and 1 represents the state where the target object to be predicted is predicted.

[0078] Step S5, cascade fine-tuning the small target object detection information and the large target object detection information and fusing them to obtain the target detection result of the aerial image.

[0079] In this embodiment, the multi-scale detection heads of aerial target detection are cascaded for optimization to improve the accuracy of repeated prediction and detection network. Usually, at least two scales of detection heads are used to generate candidate frames, and then the cascade concept is used to predict targets of different scales as a whole. Finally, all detection frames are fused for classification and position regression to obtain the final aerial target object detection result and its corresponding type.

[0080] In some preferred embodiments, the multi-scale cascade aerial target detection method based on deep neural network also includes:

[0081] Based on the deep neural network, the aerial pictures are converted into the second feature map;

[0082] Inputting the second feature map into a mesoscale detection head of a deep neural network, outputting the middle target object prediction information of the mesoscale detection head, and further updating the corresponding middle target object prediction state in the target array according to the middle target object prediction information, wherein the large target object prediction information does not change the target prediction state corresponding to the middle target object prediction information in the target array, and the middle target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array;

[0083] Based on the updated target array, the loss function of each target object in the target array is calculated, and the target objects not predicted by the large-scale detection head, the medium-scale detection head, and the small-scale detection head are reversely processed to obtain small target object detection information, medium target object detection information, and large target object detection information, respectively;

[0084] The small target object detection information, medium target object detection information and large target object detection information are cascaded and fine-tuned and then fused to obtain the target detection results of the aerial image.

[0085] In this embodiment, in order to eliminate repeated prediction samples and solve the problem of missed detection of special targets, and improve the efficiency and accuracy of target detection, based on the characteristic that detection branches of different sizes are suitable for the detection of targets of different sizes, detectors of different scales are guided to focus on target objects of different scales to solve these problems. The input aerial picture is passed through the backbone network and feature extraction module to obtain feature maps of different scales. First, the original feature map is downsampled multiple times to obtain the feature map of the minimum scale, and then the feature map of the minimum scale is upsampled by an even multiple, and the convolution layer is used to fuse the feature map of the corresponding scale, which is input into the detector of the corresponding scale for prediction, and the detection results of multiple scales are merged to obtain the final detection result. Among them, the detection result predicted by the large-scale detection head is the detection information of small target objects; the detection result predicted by the medium-scale detection head is the detection information of medium target objects; and the detection result predicted by the small-scale detection head is the detection information of large target objects.

[0086] In a further preferred embodiment, feature maps of different scales are extracted through the backbone network and feature extraction module, and these three feature maps of different scales are input into the corresponding detection head to predict the target. Each detection head will receive the detection results of the previous detection heads of different scales, except for the large-scale detection head, and analyze and process them to improve the detection results of the current detection head, which can reduce the detection efficiency and effectively improve the detection accuracy and efficiency of target detection in aerial images, thereby improving the overall prediction performance of the algorithm.

[0087] refer to Figure 2 and Figure 3 , Figure 2 A flowchart of another embodiment of a multi-scale cascade aerial target detection method based on a deep neural network according to the present application is shown. Figure 3 The overall flow chart of the multi-scale cascade aerial target detection method based on deep neural network according to the present application is shown, which specifically includes the following sub-steps:

[0088] Step 100: input aerial photography data, which includes aerial video data and aerial picture data.

[0089] Step 101, using the deep network as the backbone network, three feature maps of different scales are generated. The aerial image is input into the backbone network and feature extraction module of the deep neural network, and the original image is converted into feature maps of F1 (80*80*128), F2 (40*40*256) and F3 (20*20*512). The deep neural feature extraction network consists of the original detector path, the revised detector path and the cascade fine-tuning map. The top-down feature pyramid is used as the forward calculation of the feature extraction module. It is a calculation feature stage, which consists of feature maps of multiple prior scales, with a scaling step of 2, so as to obtain three feature maps of different scales F1, F2 and F3. The output of the last layer is selected as the reference set of feature maps, which can be called the original detector path. Secondly, in order to enrich the reference performance, we select three levels of small scale, medium scale and large scale to enhance the feature information of each stage respectively. This part is called the improved detector path. Specifically, the feature map of each stage is first input into the 1×1 convolution layer and fused with the original feature map. In particular, the output of the last layer is the third feature map F3, which needs to be upsampled by two and four times in order to be concatenated with the modified feature map. In addition, the small-scale detection head S3 should also be able to detect large-scale target objects R3 in the aerial image. The medium-scale detection head S2 should predict these medium-scale target objects R2, while the large-scale detection head S1 looks for some minimum-scale target objects R1.

[0090] Step 102, large-scale prediction. Set a target array ispre to record the target objects predicted by the three different-scale detection heads. Set the target array ispre according to the order of the targets in the annotation file, and its array length is consistent with the number of targets in each picture. The three feature maps of different scales F1, F2 and F3 are predicted by a cascade optimization method, that is, the 80*80*128 large-scale feature map F1 is input into the large-scale detection head S1 unique to the deep neural network to predict small target objects, and the original prediction boxes (x, y, w, h) and classification confidence (C S100 ,C S101 ,C S102 ,……,C S10k ) and the corresponding category, where k is the number of predicted small target objects, 00 represents the first small target object, 01 represents the second small target object, and so on. Arrange the confidence of the original prediction box of each small target object from large to small, and take the top-1 of the confidence of each small target object as the classification confidence score C of the small target object S1xxTake out the original prediction box information of the small target object R1, and perform a one-to-one match with the original prediction box (x, y, w, h) of the ground-truth. If the IOU threshold is greater than 0.5, a small target object in the ground-truth is predicted correctly, and the corresponding position of the target array ispre is set to 1, otherwise it is set to 0.

[0091] In some specific embodiments, there are five target objects (00, 01, 02, 03, 04) in an original aerial image. The large-scale detection head S1 predicts the information of three small target objects 00, 02, and 04, that is, the categories and classification confidence of the small target objects are (C S100 ,C S102 ,C S104 ). At the same time, the confidence of the original prediction boxes of the above three small target objects is greater than 0.5 for subsequent real box matching. Among the five targets, only the two predicted targets 02 and 04 can correspond well to the ground-truth, and their categories are consistent and the confidence is greater than 0.5. Therefore, according to the above matching operation, the 02 and 04 positions of the target array ispre are set to 1, and the rest of the positions are set to 0, that is, ispre=[0,0,1,0,1].

[0092] Step 103, prediction of mesoscale. The deep neural network analyzes and processes the prediction information of the small target object R1, and fuses it with the detection result of R2 to obtain the merged detection result. Specifically, the deep neural network obtains the target array ispre of the large-scale detection head S1, and then calculates the loss function of each target object. For the target object with position "1", the weight λ1=0.8 is used for adjustment, and the target with position "0" is used for weight correction processing with the weight λ2=1.5 to obtain the loss function of the small target object R1. According to the loss function of the small target object R1, the target objects that were not detected at the previous scale are reversely processed. Similarly, the second feature map F2 is input into the mesoscale detection head S2 to obtain the original prediction box of all the mesoscale target objects R2 (R2 does not change the predicted target position, but only modifies the target position that is not predicted in the large-scale detection head S1, that is, the array position with position "1" is not modified, and only the value of the position "0" is changed). For example, ispre=[0,0,1,0,1] is predicted by the large-scale detection head S1. Then the 40*40*256 medium feature map F2 is input into the corresponding mesoscale detection head S2 of the deep network for prediction, and the original prediction box and category information of all the medium target objects R2 are output. First, the mesoscale target objects with prediction confidence greater than 0.5 and consistent categories are matched with the target objects in ispre. If the mesoscale detection head S2 predicts the three targets 01, 02, and 03, the prediction information of these three targets is matched with the ground-truth (and the category and confidence are matched at the same time). When these three prediction information correspond to the ground-truth, the target array is updated to ispre = [0, 1, 1, 1, 1]. According to the position of the ispre array, the weight λ1=0.8 is used to reduce the loss of the targets corresponding to 01, 02, 03, and 04. For other undetected targets (such as the target 00), the weight λ2=1.5 is used to increase their cost when calculating the loss. This is to weaken the processing of the detected targets and strengthen the cost penalty of the undetected targets, so that the cascade optimization of the network can be better performed. The loss function formula is expressed as:

[0093]

[0094] Among them, λ1=0.8, λ2=1.5 represent the hyperparameters of the detection head, lbox represents the IOU loss function, and ispre represents the target array.

[0095] Step 104, small-scale prediction. The deep neural network analyzes and processes the prediction information of the medium target object R2, and fuses it with the detection result of R3 to obtain the merged detection result. Specifically, the deep neural network obtains the target array ispre of the medium-scale detection head S2, and then calculates the loss function of each target object. For the target object with position "1", the weight λ1=0.8 is used for adjustment, and the target with position "0" is used for weight correction processing with the weight λ2=1.5 to obtain the loss function of the medium target object R2. According to the loss function of the medium target object R2, the target objects that were not detected at the previous scale are reversely processed. Similarly, the third feature map F3 is input into the small-scale detection head S3 to obtain the original prediction box of all large target objects R3 (R3 does not change the predicted target position, but only modifies the unpredicted target position in the large-scale detection head S1 and the medium-scale detection head S2, that is, the array position with position "1" is not modified, and only the value of the position "0" is changed). For example, ispre=[0,1,1,1,1] is predicted by the large-scale detection head S1 and the medium-scale detection head S2. The third feature map F3 of the last layer is combined with the features input into the convolutional layer and input into the small-scale detection head S3. Specifically, the 20*20*512 third feature map F3 is input into the small-scale detection head S3 corresponding to the deep network for prediction, and the original prediction box and category information of all large target objects R3 are output. First, the medium-scale target objects with prediction confidence greater than 0.5 and consistent categories are matched with the target objects in ispre. If the small-scale detection head S3 predicts the two targets 00 and 02, the prediction information of these two targets is matched with the ground-truth (and the category and confidence are matched at the same time). When these two prediction information correspond to the ground-truth, the target array is updated to ispre=[1,1,1,1,1].

[0096] Step 105, fine-tune and fuse all prediction results. The prediction information of the small target object R1, the prediction information of the medium target object R2, and the prediction information of the large target object R3 are fine-tuned and then fused, and all detection frames are fused for classification and position regression to obtain the final aerial photography target detection result.

[0097] In some specific embodiments, the target category and rectangular box information corresponding to the output result of the large-scale detection head S1 are input into the Fine-Tune-Graph for storage. Similarly, the second feature map F2 and the third feature map F3 are respectively input into the medium-scale detection head S2 and the small-scale detection head S3 for detection after mapping processing. Similarly, the output target prediction category and original prediction box information are input into the Union-Graph for further cascade fine-tuning graph operation.

[0098] Preferably, the detailed steps of cascading fine-tuning the graph are as follows:

[0099] 1. Category aggregation: First, aggregate the target objects with the same category in the prediction results to form a set G i (i=1,2,…,n);

[0100] 2. Confidence fusion calculation: By performing the maximum or average fusion operation on the confidence in the target, a new fusion confidence value C is calculated. i * , the formula is: C i * =max(C S100 ,C S101 ,…,C S10k ) or C i * =avg(C S100 ,C S101 ,…,C S10k );

[0101] 3. Coordinate transformation: transform the fusion confidence value C i * Multiply it by the coordinates of the original predicted target box (x, y, w, h), the formula is:

[0102] x′=x*C o *[1±(C i * *C total )]

[0103] y′=y*C o *[1±(C i * *C total )]

[0104] w'=w*C o *[1±(C i * *C total )]

[0105] h'=h*C o *[1±(C i * *C total )]

[0106] Among them, x′ and y′ represent the center coordinates of the fine-tuned prediction box, w′ and h′ represent the width and height of the fine-tuned prediction box, respectively. o represents the original confidence value, C totalIt represents the sum of the confidences of all prediction boxes. When the original confidence is greater than 0.5, it is "+"; when the original confidence is less than 0.5, it is "-". The transformed fine-tuned prediction target box and the original prediction target box before the transformation are saved. All the fine-tuned prediction target boxes and the original prediction target boxes are subsequently fused and filtered to obtain the final target detection result.

[0107] Continue to refer Figure 4 , which shows a schematic diagram of the structure of an embodiment of a multi-scale cascade aerial target detection system based on a deep neural network according to the present application, such as Figure 4 As shown, the multi-scale cascade aerial target detection system 400 includes the following units.

[0108] Feature map extraction module 401: configured to convert the aerial image into a first feature map and a third feature map respectively based on a deep neural network;

[0109] Target array module 402: configured to construct a target array recording target objects predicted by the first feature map and the third feature map;

[0110] The target array update module 403 is configured to input the first feature map and the third feature map into the large-scale detection head and the small-scale detection head of the deep neural network respectively to obtain small target object prediction information and large target object prediction information respectively, and update the corresponding small target object prediction state and large target object prediction state in the target array according to the small target object prediction information and the large target object prediction information respectively, wherein the large target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array;

[0111] A target object detection module 404 is configured to calculate a loss function for each target object in the target array based on the updated target array, and reversely process the target objects not predicted by the large-scale detection head and the small-scale detection head to obtain small target object detection information and large target object detection information, respectively;

[0112] The target detection result module 405 is configured to fuse the small target object detection information and the large target object detection information after cascade fine-tuning operations to obtain the target detection result of the aerial image.

[0113] In some specific embodiments, the length and position of the target array correspond one-to-one to the number and order of the target objects to be predicted in the annotation file. The target array is expressed as: ispre=[0,0,…,0,0], and the formula of the loss function of each target object in the target array is expressed as:

[0114]

[0115] Among them, λ1=0.8, λ2=1.5 represent the hyperparameters of the detection head, lbox represents the IOU loss function, ispre represents the target array, 0 represents the state where the target object to be predicted is not predicted, and 1 represents the state where the target object to be predicted is predicted.

[0116] Continue to refer Figure 5 and Figure 6 , Figure 5 The figure shows the actual vehicle test effect diagram of an embodiment of the multi-scale cascade aerial target detection system based on deep neural network according to the present application, Figure 6 This is a picture of the actual test effect of a ship according to an embodiment of the multi-scale cascade aerial target detection system based on a deep neural network of the present application, such as Figure 5 and Figure 6 As shown, the small-scale detection head, medium-scale detection head and large-scale detection head of the multi-scale cascade aerial target detection system based on deep neural network of the present application can intuitively and accurately identify cars, buses and ships, and display the category of the target object in the corresponding prediction box at the edge of the corresponding prediction box. Since the target objects are too dense, the category of the target object is not shown in the figure. The present application can not only effectively avoid false detection and missed detection, improve the accuracy of target object detection, but also avoid repeated detection of the same target object, improve detection efficiency. In addition, the final effect diagram is intuitive and clear and is suitable for various video surveillance and autonomous driving fields.

[0117] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 suitable for implementing an electronic device of an embodiment of the present application. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0118] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 701 or a program loaded from a storage part 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the system 700 are stored in the RAM 703. The CPU 701, the ROM 701, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0119] The following components are connected to the I / O interface 705: an input section 706, an output section 707, a storage section 708, a communication section 709, a drive 710, and a removable medium 711. The drive 710 can be connected to the I / O interface 705 as needed, and the removable medium 711 can be installed on the drive 710 as needed, so that a computer program read therefrom can be installed into the storage section 708 as needed.

[0120] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. Computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof. The program code can be executed entirely on the trainer computer, partially on the trainer computer, as a separate software package, partially on the trainer computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the trainer computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0121] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0122] The modules involved in the embodiments of the present application may be implemented by software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including a feature map extraction module, a target array module, an update target array module, a target object detection module, and a target detection result module. Among them, the names of these modules do not constitute a limitation on the unit itself in some cases, for example, the feature map extraction module may also be described as "used to convert aerial images into first feature maps and third feature maps respectively based on a deep neural network".

[0123] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: based on a deep neural network, converts the aerial image into a first feature map and a third feature map respectively; constructs a target array that records the target objects predicted by the first feature map and the third feature map; inputs the first feature map and the third feature map into the large-scale detection head and the small-scale detection head of the deep neural network respectively to obtain small target object prediction information and large target object prediction information, and updates the corresponding small target object prediction state and large target object prediction state in the target array according to the small target object prediction information and the large target object prediction information, wherein the large target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array; based on the updated target array, calculates the loss function of each target object in the target array, and reversely processes the target objects not predicted by the large-scale detection head and the small-scale detection head to obtain small target object detection information and large target object detection information respectively; cascades the small target object detection information and the large target object detection information and fuses them to obtain the target detection result of the aerial image.

[0124] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A multi-scale cascade aerial target detection method based on deep neural network, characterized in that: The method comprises: Based on the deep neural network, the aerial pictures are converted into a first feature map and a third feature map respectively; Constructing a target array of records including target objects predicted by the first feature map and the third feature map; Input the first feature map into the large-scale detection head of the deep neural network for predicting small target objects, obtain small target object prediction information of the large-scale detection head, input the third feature map into the small-scale detection head of the deep neural network for predicting large target objects, obtain large target object prediction information of the small-scale detection head, and update the corresponding small target object prediction state and large target object prediction state in the target array according to the small target object prediction information and the large target object prediction information, respectively, wherein the large target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array; Based on the updated target array, a loss function of each target object in the target array is calculated, and the target objects not predicted by the large-scale detection head and the small-scale detection head are reversely processed to obtain small target object detection information and large target object detection information respectively; The small target object detection information and the large target object detection information are cascaded and fine-tuned and then fused to obtain the target detection result of the aerial image; The small target object prediction information is obtained by the following sub-steps: The first feature map is input into the large-scale detection head, and the original prediction box (x, y, w, h) and classification confidence (C S100 ,C S101 ,C S102 ,…,C S10k ) and the corresponding category, where x and y represent the center coordinates of the original prediction box, w and h represent the width and height of the original prediction box, respectively, and k represents the number of small target objects to be predicted; Calculate the IOU threshold of all the small target objects, and perform one-to-one matching between the original prediction box and the actual box. If the IOU threshold is greater than 0.5, obtain the prediction information of the small target object; The small target object detection information and the large target object detection information are input into the Fine-Tune-Graph to perform the cascade fine-tuning operation, to obtain a fine-tuning prediction frame of the target object, to fuse and filter the fine-tuning prediction frame and the original prediction frame, to obtain the target detection result of the aerial image; wherein the cascade fine-tuning operation specifically includes: The target objects of the same category in the small target object detection information and the large target object detection information are aggregated to obtain a set G of target objects of the same category. i ; The set G i The classification confidence of the target objects of the same category is calculated by the maximum or average confidence fusion to obtain the fusion confidence value C of the target objects of the same category. i * , where the fusion confidence value C i * =max(C S100 ,C S101 ,…,C S10k ) or, fusion confidence value C i * =avg(C S100 ,C S101 ,…,C S10k ); The fusion confidence value C i * Multiply it by the coordinates of the original prediction box (x, y, w, h). The specific formula is: x′=x*C o *[1±(C i * *C total )] y′=y*C o *[1±(C i * *C total )] w′=w*C o *[1±(C i * *C total )] h′=h*C o *[1±(C i * *C total )] Among them, x′ and y′ represent the center coordinates of the fine-tuned prediction box, w′ and h′ represent the width and height of the fine-tuned prediction box, respectively. o represents the original confidence value, C total It represents the sum of the confidences of all prediction boxes. When the original confidence is greater than 0.5, it is "+"; when the original confidence is less than 0.5, it is "-".

2. The method according to claim 1, characterized in that The length and position of the target array correspond one-to-one to the number and order of the target objects to be predicted in the annotation file. The target array is expressed as: ispre=[0,0,…,0,0]. The formula of the loss function of each target object in the target array is expressed as: Among them, λ1=0.8, λ2=1.5 represent the hyperparameters of the detection head, lbox represents the IOU loss function, ispre represents the target array, 0 represents the state where the target object to be predicted is not predicted, and 1 represents the state where the target object to be predicted is predicted.

3. The method according to claim 1, characterized in that The classification confidence of a single small target object is obtained by sorting all the confidences of the single small target object from large to small, and obtaining the top-1 confidence of the single small target object as the classification confidence score C of the single small target object. S1xx .

4. The method according to claim 1, characterized in that: The method further comprises: Based on the deep neural network, convert the aerial picture into a second feature map; Inputting the second feature map into the mesoscale detection head of the deep neural network, outputting the mesoscale target object prediction information of the mesoscale detection head, and further updating the corresponding mesoscale target object prediction state in the target array according to the mesoscale target object prediction information, wherein the large target object prediction information does not change the target prediction state in the target array corresponding to the mesoscale target object prediction information, and the mesoscale target object prediction information does not change the target prediction state in the target array corresponding to the small target object prediction information; Based on the updated target array, a loss function of each target object in the target array is calculated, and the target objects not predicted by the large-scale detection head, the medium-scale detection head, and the small-scale detection head are reversely processed to obtain small target object detection information, medium target object detection information, and large target object detection information, respectively; The small target object detection information, the medium target object detection information and the large target object detection information are cascaded and fine-tuned and then fused to obtain the target detection result of the aerial image.

5. A multi-scale cascade aerial target detection system based on deep neural network, characterized in that: The system comprises: A feature map extraction module: configured to convert the aerial image into a first feature map and a third feature map respectively based on the deep neural network; A target array module: configured to construct a target array recording target objects predicted by the first feature map and the third feature map; A target array update module is configured to input the first feature map into a large-scale detection head of the deep neural network for predicting small target objects, obtain small target object prediction information of the large-scale detection head, input the third feature map into a small-scale detection head of the deep neural network for predicting large target objects, obtain large target object prediction information of the small-scale detection head, and update the corresponding small target object prediction state and large target object prediction state in the target array according to the small target object prediction information and the large target object prediction information, respectively, wherein the large target object prediction information does not change the target prediction state corresponding to the small target object prediction information in the target array; A target object detection module is configured to calculate a loss function for each target object in the target array based on the updated target array, and reversely process the target objects not predicted by the large-scale detection head and the small-scale detection head to obtain small target object detection information and large target object detection information respectively; A target detection result module is configured to fuse the small target object detection information and the large target object detection information after cascade fine-tuning operations to obtain a target detection result of the aerial image; The small target object prediction information is obtained by the following sub-steps: The first feature map is input into the large-scale detection head, and the original prediction box (x, y, w, h) and classification confidence (C S100 ,C S101 ,C S102 ,…,C S10k ) and the corresponding category, where x and y represent the center coordinates of the original prediction box, w and h represent the width and height of the original prediction box, respectively, and k represents the number of small target objects to be predicted; Calculate the IOU threshold of all the small target objects, and perform one-to-one matching between the original prediction box and the actual box. If the IOU threshold is greater than 0.5, obtain the prediction information of the small target object; The small target object detection information and the large target object detection information are input into the Fine-Tune-Graph to perform the cascade fine-tuning operation, to obtain a fine-tuning prediction frame of the target object, to fuse and filter the fine-tuning prediction frame and the original prediction frame, to obtain the target detection result of the aerial image; wherein the cascade fine-tuning operation specifically includes: The target objects of the same category in the small target object detection information and the large target object detection information are aggregated to obtain a set G of target objects of the same category. i ; The set G i The classification confidence of the target objects of the same category is calculated by the maximum or average confidence fusion to obtain the fusion confidence value C of the target objects of the same category. i * , where the fusion confidence value C i * =max(C S100 ,C S101 ,…,C S10k ) or, fusion confidence value C i * =avg(C S100 ,C S101 ,…,C S10k ); The fusion confidence value C i * Multiply it by the coordinates of the original prediction box (x, y, w, h). The specific formula is: x′=x*C o *[1±(C i * *C total )] y′=y*C o *[1±(C i * *C total )] w′=w*C o *[1±(C i * *C total )] h′=h*C o *[1±(C i * *C total )] Among them, x′ and y′ represent the center coordinates of the fine-tuned prediction box, w′ and h′ represent the width and height of the fine-tuned prediction box, respectively. o represents the original confidence value, C total It represents the sum of the confidences of all prediction boxes. When the original confidence is greater than 0.5, it is "+"; when the original confidence is less than 0.5, it is "-".

6. The system according to claim 5, characterized in that The length and position of the target array correspond one-to-one to the number and order of the target objects to be predicted in the annotation file. The target array is expressed as: ispre=[0,0,…,0,0]. The formula of the loss function of each target object in the target array is expressed as: Among them, λ1=0.8, λ2=1.5 represent the hyperparameters of the detection head, lbox represents the IOU loss function, ispre represents the target array, 0 represents the state where the target object to be predicted is not predicted, and 1 represents the state where the target object to be predicted is predicted.

7. An electronic device comprising: one or more processors; a storage system for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images