A target detection method for unmanned aerial vehicle aerial photographs

Abstract

The application relates to the technical field of unmanned system target detection, in particular to a target detection method for aerial photos of unmanned aerial vehicles, which comprises the following steps: 1, constructing a training set; 2, constructing a target detection model; 3, inputting an image into a backbone network to obtain four types of backbone network feature maps with different scales; 4, performing feature fusion on the output results of step 3 to obtain four types of fusion feature maps; 5, inputting the fourth type of fusion feature map into a CVAN module to finally obtain a fourth type of output feature map; 6, inputting the output results of step 5 into a target detection head to calculate the classification information and other information of the target; repeating steps 3 to 5 until iteration reaches a set number of times, selecting the best set of weight results, and obtaining the trained target detection model. The application introduces a transformer, the unmanned aerial vehicle image contains many dense micro objects, the introduction of the transformer can effectively enhance the semantic distinguishability of each micro object, and effectively reduce missed detection and false detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned system target detection, and particularly relates to a target detection method for unmanned aerial vehicle aerial photograph. BACKGROUND

[0002] Micro target detection in unmanned aerial vehicle images is widely used in maritime rescue, large-scale monitoring, agriculture, and environmental protection. The main purpose of this paper is to enhance the target detection performance of unmanned aerial vehicle captured images, and in addition, to provide valuable insights for the above-mentioned applications.

[0003] Due to the different flight heights of the images captured by the unmanned aerial vehicle, the scales of the targets may vary greatly, which brings challenges to the accurate detection of the target detection model. In addition, the micro objects in the image contain little information and are difficult to distinguish from the background. For the detection of micro targets in images, many traditional detection methods have been proposed, including sliding window-based detection methods, histogram of oriented gradients, and deformable part models. However, these traditional target detection algorithms have large computational complexity, slow processing speed, and low accuracy, making them difficult to apply in real-world scenarios. In recent years, many target detection algorithms based on convolutional neural networks (CNN) have been proposed and successfully applied to target detection in unmanned aerial vehicle images. Specifically, target detection methods can be mainly divided into two categories: two-stage methods and one-stage methods. Two-stage methods first generate candidate blocks containing possible detection targets, and then classify and correct these candidate blocks, such as R-CNN, Fast R-CNN, and Faster R-CNN. However, this method has high detection accuracy but low efficiency. Single-stage methods achieve end-to-end target detection through regression operations, such as SSD, YOLO series, which have faster detection speed and can meet real-time requirements. SUMMARY

[0004] The present application provides a target detection method for unmanned aerial vehicle aerial photographs to solve the technical problem of low detection efficiency of existing unmanned aerial vehicle image target detection methods.

[0005] To achieve the above-mentioned purposes, the technical solution of the present application is as follows:

[0006] The present application provides a target detection method for unmanned aerial vehicle aerial photographs, comprising the following steps:

[0007] S1, obtaining multiple unmanned aerial vehicle aerial images, dividing the multiple unmanned aerial vehicle aerial images into a training set, a test set, and a validation set, pre-processing the images in the training set to obtain a pre-processed training set;

[0008] S2, a target detection model is constructed, including a backbone network, an intermediate layer and a detection head connected in sequence; the intermediate layer includes a transformer and a CVAN module, wherein the transformer is connected with the bottom layer of the backbone network, and the CVAN module is connected with the detection head;

[0009] S3, an image is selected from the preprocessed training set and input into the backbone network to obtain four types of backbone network feature maps of different scales, namely a first type of backbone network feature map to a fourth type of backbone network feature map, and the first type of backbone network feature map is input into the transformer to obtain a transformer output feature map;

[0010] S4, the transformer output feature map and the second type of backbone network feature map to the fourth type of backbone network feature map are input into the intermediate layer of the target detection model for feature fusion to obtain a first type of fusion feature map, a second type of fusion feature map, a third type of fusion feature map and a fourth type of fusion feature map;

[0011] S5, the fourth type of fusion feature map is input into the CVAN module to finally obtain a fourth type of output feature map ;

[0012] S6, the first type of fusion feature map, the second type of fusion feature map, the third type of fusion feature and the fourth type of output feature map are input into respective target detection heads to calculate the classification information, confidence information and bounding box position information of the target; S3 to S5 are repeated, the validation set is used to verify the current target detection model, the confidence loss, classification loss and bounding box prediction regression loss are calculated, and then the best set of weight results in the training set is selected and saved until the iteration reaches the set number of times, and then a trained target detection model is obtained.

[0013] Further, the preprocessing in S1 includes annotation file generation, data set reading, image size adjustment, image enhancement, image normalization, image padding, image conversion and image dimension expansion.

[0014] The image enhancement includes random cropping, random flipping, random rotation, random scaling, and brightness and contrast adjustment.

[0015] Further, the backbone network in S2 is the backbone network of YOLOv7, specifically including four convolution normalization activation layers, three maximum pooling layers and four first layer aggregation networks.

[0016] The three maximum pooling layers are connected in pairs with the three first layer aggregation networks in the first group in sequence; the four convolution normalization activation layers are connected in sequence, then connected with another first layer aggregation network, and connected with the maximum pooling layer and the first layer aggregation network in the first group.

[0017] Further, the step of inputting the first type of backbone network feature map into the transformer in S3 to obtain a transformer output feature map specifically comprises the following steps:

[0018] S31, inputting the first type of backbone network feature map into the transformer through a trainable query weight matrix , a keyword weight matrix , and a linear transformation to obtain a query matrix and a key matrix ;

[0019] S32, calculating the attention weight by dot product operation between the query matrix and the key matrix ;

[0020] S33, performing weighted calculation on the value matrix of the first type of backbone network feature map using the attention weight to obtain a plurality of self-attention values ;

[0021] S34, concatenating the plurality of self-attention values to obtain multi-head self-attention MSA, and then inputting the multi-head self-attention MSA into a linear layer to obtain an output ;

[0022] S35, performing residual A connection and layer normalization on the first type of backbone network feature map and the obtained multi-head self-attention MSA to obtain an intermediate result ;

[0023] S36, inputting the output results of S34 and S35 into a multi-layer perceptron and performing residual A connection to obtain a transformer output feature map .

[0024] Further, the S31 to S36 are specifically expressed by the following formulas:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] wherein, represents a query weight matrix, represents a keyword weight matrix, represents a value weight matrix, D represents a vector dimension; LN represents layer normalization, represents a multi-head self-attention weight matrix, k represents the number of self-attention values .

[0035] Further, the S5 specifically comprises the following steps:

[0036] S51, standardizing the fourth type of fusion feature map to obtain a standardized feature map ;

[0037] S52, inputting the standardized feature map into the attention mechanism of the CVAN module to obtain an attention map , which is used to focus on the location of the set target;

[0038] S53, the attention map and the standardized feature map pass through residual A connection to obtain a feature map that can capture the location and feature information of the set target;

[0039] S54, performing an element-wise product on the feature map obtained in S53 and a trainable parameter , and performing residual A connection on the obtained result and the fourth type of fusion feature map to obtain a fourth type of output feature map .

[0040] Further, the feature map in S53 and the fourth type of output feature map in S54 can be represented by the formula:

[0041] ;

[0042] ;

[0043] wherein, represents an element-wise product, This represents the attention module.

[0044] Furthermore, the target detection method further includes the following steps:

[0045] S7. Select any drone aerial image from the test set and input it into the trained target detection model to obtain the final target detection result. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the target detection model in this invention;

[0047] Figure 2 for Figure 1 A magnified schematic diagram of the backbone network structure in the diagram;

[0048] Figure 3 for Figure 1 An enlarged schematic diagram of the intermediate layer structure;

[0049] Figure 4 for Figure 1 A schematic diagram of a medium-efficiency layered aggregation network;

[0050] Figure 5 for Figure 1 A schematic diagram of the structure of the maximum pooling layer;

[0051] Figure 6 for Figure 1 A schematic diagram of the structure of the mid-layer aggregation network;

[0052] Figure 7 for Figure 1 A schematic diagram of the structure of a convolutional normalized activation layer;

[0053] Figure 8 for Figure 1 Schematic diagram of the upper sampling layer;

[0054] Figure 9 This is a schematic diagram of the CVAN module in this invention;

[0055] Figure 10 This is a comparison chart showing the effect of the target detection method used in this invention and YOLOv7 on real aerial images. Detailed Implementation

[0056] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0057] In addition, the terms "first", "second", etc. are used only for descriptive purposes and should not be construed as indicating or implying relative importance or an indicated number of technical features. Therefore, the features defined as "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0058] Referring to Figures 1 to 8 The embodiment of the present application provides a target detection method for unmanned aerial vehicle aerial photographs, comprising the following steps:

[0059] S1, a plurality of unmanned aerial vehicle aerial images are acquired, and the plurality of unmanned aerial vehicle aerial images are divided into a training set, a test set and a verification set. The training set is used for training of a model, the verification set is used for adjusting hyperparameters and monitoring model performance, the test set is used for evaluating final performance of the model, then images in the training set are preprocessed to obtain a preprocessed training set. The preprocessing includes data enhancement, and a K-means clustering algorithm is used to obtain sizes of anchor boxes.

[0060] S2, a target detection model is constructed, comprising a backbone network of YOLOv7, an intermediate layer and a detection head connected in sequence; the intermediate layer comprises a transformer model and a CVAN (corresponding to a convolutional visual attention network in Figure 1 ) module, wherein the transformer model is connected with a bottom layer of the backbone network of YOLOv7, and the CVAN module (for a specific structure, refer to Figure 9 ) is connected with the detection head.

[0061] The CVAN module comprises a normalization layer and an attention network, wherein an input end of the normalization layer is split into two branches, which are a first branch and a second branch respectively, the first branch is directly connected with the input of the normalization layer, an output end of the normalization layer forms two branches, which are a third branch and a fourth branch respectively, the third branch is connected with an input end of the attention network, the fourth branch is connected with an output end of the attention network through a residual connection to obtain a summation result, the summation result is multiplied by a trainable weight element by element to obtain a product result, the product result is connected with the second branch through a residual connection to obtain an output of the CVAN module; the attention network comprises a 1x1 convolution layer, an activation function, a large convolution kernel attention and another 1x1 convolution layer connected in sequence.

[0062] The present application integrates the transformer model into the output of the last layer of the backbone network of YOLOv7 to enhance its ability to capture global context. This enables the target detection model to have a larger dynamic receptive field and stronger robustness while also having the ability to link the context of the image.

[0063] S3, an image is selected from the pre-processed training set and input into the backbone network to obtain four types of backbone network feature maps of different scales, namely a first type of backbone network feature map to a fourth type of backbone network feature map, and the first type of backbone network feature map is input into the transformer to obtain a transformer output feature map;

[0064] The first type of backbone network feature map to the fourth type of backbone network feature map are Figure 1 The four layers of the efficient layer aggregation network in the backbone network are from top to bottom.

[0065] The first type, the second type, the third type and the fourth type in the first type of backbone network feature map to the fourth type of backbone network feature map correspond to large targets, medium targets, small targets and tiny targets respectively. The first type, the second type, the third type and the fourth type appearing in the following text are the same, and will not be described in detail below.

[0066] S4, the transformer output feature map and the second type of backbone network feature map to the fourth type of backbone network feature map are input into the middle layer of the target detection model for feature fusion to obtain a first type of fusion feature map, a second type of fusion feature map, a third type of fusion feature map and a fourth type of fusion feature map;

[0067] S5, the fourth type of fusion feature map is input into the CVAN module to finally obtain a fourth type of output feature map ;

[0068] S6, the first type of fusion feature map, the second type of fusion feature map, the third type of fusion feature and the fourth type of output feature map are input into the respective target detection head to calculate the classification information, the confidence information and the bounding box position information of the target; S3 to S5 are repeated, the validation set is used to verify the current target detection model, the confidence loss, the classification loss and the bounding box prediction regression loss are calculated, and the iteration is performed to a set number of times, and then a best set of weight results in the training set is selected and saved to obtain the trained target detection model.

[0069] In some embodiments, the preprocessing in S1 includes annotation file generation, data set reading, image size adjustment, image enhancement, image normalization, image padding, image conversion and image dimension expansion; wherein the image enhancement includes random cropping, random flipping, random rotation, random scaling, and brightness and contrast adjustment, which are specifically as follows:

[0070] (1) Annotation file generation: first, an annotation file needs to be created for each image in the training set. The annotation file is usually in txt or xml format, which contains the class and bounding box information of each target object (i.e. real label). Each line represents a target object, which contains the class index and the coordinate information of the bounding box.

[0071] (2) Dataset reading: Write code to read the dataset and annotation file.

[0072] (3) Adjust image size: Adjust the input image to the size required by the model. The input image is changed to 1536x1536 size. The advantage of this is that high-resolution image input helps to improve the visual quality and feature representation of the target, thereby improving the target detection accuracy.

[0073] (4) Image enhancement: Perform image enhancement operations to improve the performance and robustness of the model. The image enhancement methods used include random cropping, random flipping, random rotation, random scaling, and brightness and contrast adjustment.

[0074] (5) Image normalization: Normalize the pixel values of the image from 0 to 255 to the range of 0 to 1. Each pixel value can be divided by 255 to achieve normalization.

[0075] (6) Pad image: If the aspect ratio of the image does not match the input size of the model, you can choose to add padding around the image to maintain the aspect ratio.

[0076] (7) Image conversion: Convert the normalized and padded image to the format required by the model.

[0077] (10) Image expansion dimension: Expand the dimension of the converted image to match the input requirements of the model. The dimension of the image is expanded to the form (batch_size, channels, height, width).

[0078] In some embodiments, the backbone network of YOLOv7 in S2 includes four convolution normalization activation layers, three max pooling layers, and four first layer aggregation networks (corresponding to the high-efficiency layer aggregation network in Figure 1 );

[0079] The three max pooling layers are connected in turn in pairs with the three first layer aggregation networks among them; the four convolution normalization activation layers are connected in turn, and then connected with another first layer aggregation network, and connected with the max pooling layers and first layer aggregation networks of the first group.

[0080] In some embodiments, the input of the first type of backbone network feature map into the transformer in S3 to obtain the transformer output feature map specifically includes the following steps:

[0081] S31, the first type of backbone network feature map is linearly transformed by a trainable query weight matrix , a keyword weight matrix to obtain a query matrix and key matrix ;

[0082] S32, the value matrix of the first type of backbone network feature map is calculated by querying the matrix and key matrix Dot product operation between ;

[0083] S33, the value matrix of the first type of backbone network feature map is calculated by querying the matrix Weighted calculation is performed on the value matrix of the first type of backbone network feature map to obtain a plurality of self-attention values ;

[0084] S34, the plurality of self-attention values are spliced together to obtain multi-head self-attention MSA, and then input into a linear layer to obtain output ;

[0085] S35, the obtained multi-head self-attention MSA and the first type of backbone network feature map Residual A connection and layer normalization are performed to obtain an intermediate result ;

[0086] S36, the output results of S34 and S35 are input into a multi-layer perceptron and residual A connection is performed to obtain a transformer output feature map .

[0087] The S31 to S36 are specifically expressed by the following formulas:

[0088] ;

[0089] ;

[0090] ;

[0091] ;

[0092] ;

[0093] ;

[0094] ;

[0095] ;

[0096] ;

[0097] wherein, query weight matrix​ denotes a keyword weight matrix, denotes a value weight matrix, D denotes a vector dimension; LN denotes layer normalization, denotes a multi-head self-attention weight matrix, k denotes a plurality of self-attention values .

[0098] In some embodiments, the S5 specifically comprises the following steps:

[0099] S51, the fourth type of fusion feature map is standardized to obtain a standardized feature map ;

[0100] S52, the standardized feature map is input into the attention mechanism of the CVAN module to obtain an attention map , which is used to focus on the location of the set target;

[0101] S53, the attention map and the standardized feature map are connected through residual A to obtain a feature map that can capture the location and feature information of the set target;

[0102] S54, the feature map obtained in S53 is multiplied element by element with a trainable parameter , and the obtained result is connected with the fourth type of fusion feature map through residual A to obtain the fourth type of output feature map .

[0103] The feature map in the S53 and the fourth type of output feature map in the S54 can be expressed by the following formula:

[0104] ;

[0105] ;

[0106] wherein, denotes element-wise multiplication, denotes an attention module.

[0107] In some embodiments, the target detection method further comprises the following steps:

[0108] S7, selecting any one of the unmanned aerial vehicle aerial images from the test set and inputting it into the trained target detection model to obtain the final target detection result.

[0109] The micro target detector (TOD) provided by the application can effectively extract the features of micro targets and distinguish the micro target features from the image background. Figure 1 As shown in the figure, the micro target detector (TOD) comprises a second layer aggregation network, a convolutional visual attention network (CVAN), a repeated convolution layer and a convolution layer connected in sequence, and the micro target detector (TOD) can better focus on the area of micro objects.

[0110] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application. Furthermore, the technical solutions of each embodiment of the application can be combined with each other, but it must be based on the realization of the ordinary skilled in the art, when the combination of the technical solutions appears contradictory or unachievable, it should be considered that the combination of the technical solutions does not exist, nor within the protection scope required by the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A target detection method for drone aerial photographs, characterized in that: The method comprises the following steps: S1, obtaining a plurality of unmanned aerial vehicle aerial images, dividing the plurality of unmanned aerial vehicle aerial images into a training set, a test set and a verification set, and preprocessing images in the training set to obtain a preprocessed training set; S2, constructing a target detection model, comprising a backbone network, an intermediate layer and a detection head connected in turn; the intermediate layer comprises a transformer and a CVAN module, wherein the transformer is connected with the bottom layer of the backbone network, and the CVAN module is connected with the detection head; S3, inputting an image selected from the preprocessed training set into the backbone network to obtain four types of backbone network feature maps of different scales, namely a first type of backbone network feature map to a fourth type of backbone network feature map, and inputting the first type of backbone network feature map into the transformer to obtain a transformer output feature map; S4, inputting the transformer output feature map and the second type of backbone network feature map to the fourth type of backbone network feature map into the intermediate layer of the target detection model for feature fusion to obtain a first type of fusion feature map, a second type of fusion feature map, a third type of fusion feature map and a fourth type of fusion feature map; S5, input the fourth type of fusion feature map into the CVAN module to finally obtain a fourth type of output feature map ; S6、the first type of fusion feature map, the second type of fusion feature map, the third type of fusion feature and the fourth type of output feature map The input is input into the respective target detection head, and the classification information, confidence information and bounding box position information of the target are calculated; S3 to S5 are repeated, the current target detection model is verified using the verification set, the confidence loss, classification loss and bounding box prediction regression loss are calculated, and the iteration is performed until the set number of times, and then the best set of weight results in the training set is selected and saved, to obtain the trained target detection model; The CVAN module comprises a normalization layer and an attention network, wherein the input end of the normalization layer is split into two branches, namely a first branch and a second branch, the first branch is directly connected with the input of the normalization layer, the output end of the normalization layer forms two branches, namely a third branch and a fourth branch, the third branch is connected with the input end of the attention network, the fourth branch is connected with the output end of the attention network through a residual connection to obtain a summation result, the summation result is multiplied element by element with a trainable weight to obtain a product result, the product result is connected with the second branch through a residual connection to obtain the output of the CVAN module; The attention network comprises a 1×1 convolution layer, an activation function, a large kernel attention and another 1×1 convolution layer connected in turn.

2. The object detection method of claim 1, wherein, The preprocessing in S1 comprises annotation file generation, data set reading, image size adjustment, image enhancement, image normalization, image padding, image conversion and image dimension expansion. The image enhancement comprises random cropping, random flipping, random rotation, random scaling, brightness adjustment and contrast adjustment.

3. The object detection method of claim 1, wherein, The backbone network in S2 is the backbone network of YOLOv7, and specifically comprises four convolution normalization activation layers, three maximum pooling layers and four first layer aggregation networks. The three maximum pooling layers are connected in turn in groups of two with the three first layer aggregation networks, and the four convolution normalization activation layers are connected in turn, then connected with another first layer aggregation network, and connected with the maximum pooling layers and the first layer aggregation networks in the first group.

4. The object detection method of claim 1, wherein, In S3, inputting the first type of backbone network feature map into the transformer to obtain the transformer output feature map specifically comprises the following steps: S31、the first type of backbone network feature map by a trainable query weight matrix , a keyword weight matrix to obtain a query matrix and a key matrix ; S32, calculate the attention weight by querying the matrix and the key matrix between the dot product operation ; S33, utilize attention weight The value matrix of the first type of backbone network feature map The weighted calculation is performed on the value matrix of the first type of backbone network feature map ;​ S34, values of multiple self-attention are concatenated to get multi-head self-attention MSA, which is then passed into a linear layer to get the output ; S35. Obtain the multi-head self-attention MSA and the first type of backbone network feature map. Perform residual A-joins and layer normalization to obtain intermediate results. ; S36, the output results of S34 and S35 are transmitted into a multi-layer perception machine and residual A connection are obtained .

5. The object detection method of claim 4, wherein, In S31 to S36, the formula is specifically as follows: ； ； ； ； ； ； ； ； ； wherein, denotes a query weight matrix, denotes a keyword weight matrix, denotes a value weight matrix, D denotes a vector dimension; LN denotes layer normalization, denotes a multi-head self-attention weight matrix, k denotes a number of values of a plurality of self-attentions .

6. The object detection method of claim 5, wherein, In S5, the formula is specifically as follows: S51, the fourth type of fusion feature map standardization, to obtain a normalized feature map ; S52, the standardized feature map input into the attention mechanism of the CVAN module to obtain an attention map for focusing on the location of the target setting S53, attention map and the normalized feature map a residual A connection to obtain a feature map that can capture the set target position and the set target feature information ; S54, the feature map obtained in S53 is taken as input of a convolutional layer with one trainable parameter an element-wise product is performed, and the obtained result is taken as input of a fourth fusion feature map a residual A connection is made, and a fourth output feature map is obtained .

7. The object detection method of claim 6, wherein, the feature map in S53 and the fourth type of output feature map in S54 which can be expressed by the formula ； ； wherein, denotes an element-wise multiplication, denotes an attention module.

8. The object detection method of claim 7, wherein, Further comprising the following steps: S7, selecting an arbitrary unmanned aerial vehicle aerial image from the test set and inputting it into the trained target detection model to obtain a final target detection result.

Images