Small Target Detection Method for UAV Platform

Abstract

The present disclosure provides a small target detection method for a drone platform. The method includes: obtaining an image set including a training set and a test set; preprocessing the images in the training set; inputting the preprocessed images into a network and training the network until the network is fitted. The step of training the network includes: obtaining values representing the probabilities that each target is discriminated as various categories, coordinate prediction values of each target adjusted in the image, and coordinate prediction values without adjustment, calculating a loss function based on the above parameters, and adjusting the network structure parameters to the optimal by minimizing the loss function; using the optimal parameters to output detection results, where the detection results include values representing the probabilities that each target is discriminated as various categories, coordinate prediction values of each target adjusted in the image, and coordinate prediction values without adjustment.

Description

Technical Field

[0001] The present invention relates to an image processing technology, and more specifically, to a small target detection method for an unmanned aerial vehicle (UAV) platform. Background Art

[0002] Target detection is an important computer vision task. In recent years, with the rapid expansion of the UAV market and the decline in the cost of hardware devices, target detection based on the UAV platform has become a research hotspot. Against the background of the rapid development of deep learning technology, target detection on the UAV platform has gradually shifted from traditional manually designed features to deep neural network recognition.

[0003] The target detection task can be decoupled into a regression localization sub-task and a classification sub-task. For the regression localization sub-task, the model should have an inductive preference for edges and shapes, while for the classification sub-task, the model should have an inductive preference for internal textures. In view of its characteristics, current deep learning-based target detection algorithms are divided into two-stage and single-stage target detection algorithms. The two-stage algorithm first predicts the positions of many regions where targets may exist and preliminarily classifies them as foreground or background regions, and then in the second stage, it makes a fine prediction of the target positions and distinguishes the object categories for the selected foreground regions. The single-stage algorithm directly predicts and outputs the position information and category information of the targets.

[0004] Although deep learning has brought great development to target detection, there are still many problems in small target detection. Among different categories of general target detection, some target textures and postures are similar and easy to be confused, and there are large differences within the same category, such as shape postures, sizes, and color textures, which are prone to misjudgment. In addition to these conventional characteristics, UAV aerial images also contain a large number of small targets. Among them, the proportion of small targets in the image is small, the information contained is less, and it is difficult for the network to extract enough information for training. In addition to the small amount of information of small targets themselves, due to the inherent characteristics of network neural convolution, the feature information of small targets decreases with the layer-by-layer convolution of the network, which also leads to the problem of poor small target detection effect.

[0005] Typical small target detection algorithms optimize the final prediction by adjusting the output results in multiple stages to improve the detection accuracy. However, due to the high flight speed of UAVs and the high real-time requirement for target detection, it is necessary to ensure both recognition accuracy and speed at the same time. Traditional multi-stage algorithms cannot be actually deployed on UAVs because of their long inference time. Single-stage target detection algorithms are fast but have low accuracy, and for UAVs in specific scenarios, they cannot reach the detection accuracy, so the deployment effect is poor.

[0006] In addition, from the perspective of code security, when deploying a model on an embedded device, loading a pre-trained model usually requires relying on the source code of the model structure. However, this deployment method is prone to source code leakage. Therefore, it is urgent to load the pre-trained model without source code.

[0007] Therefore, an efficient and accurate method for detecting small targets is needed. Summary of the Invention

[0008] The present invention is proposed to solve the above problems, and its purpose is to provide a small target detection method for an unmanned aerial vehicle (UAV) platform.

[0009] To achieve the above object of the invention, according to one aspect of the present invention, there is provided a small target detection method for an unmanned aerial vehicle platform, the method comprising: obtaining an image set including a training set and a test set; preprocessing the images in the training set; inputting the preprocessed images into a network and training the network until the network converges. The step of training the network includes: obtaining values representing the probabilities that each target is classified into various categories, values representing the predicted coordinates of each target in the image after adjustment, and predicted coordinates of each target in the image without adjustment, calculating a loss function based on the above parameters, and adjusting the network structure parameters to the optimal by minimizing the loss function; outputting a detection result using the optimal parameters, the detection result including values representing the probabilities that each target is classified into various categories, values representing the predicted coordinates of each target in the image after adjustment, and predicted coordinates of each target in the image without adjustment.

[0010] Preprocessing the images in the training set includes: restricting the height and width of the images so that they are divisible by 8, subtracting the average image of the entire data set from each image and normalizing, and randomly flipping and shearing the images in the training set, and adding the images that have been randomly flipped and sheared to the training set to increase the number of images in the training set.

[0011] The loss function is calculated by the following formula:

[0012]

[0013] where DFL(S)=-((y i+1 -y)log(S i )+(y-y i )log(S i+1 )),

[0014] y is the coordinate value of the target in the image, y i and y i+1 represent the coordinates of the two adjacent points on the left and right of the target, obtained by rounding up and rounding down the coordinates respectively, S i and S i+1 represent the probabilities that the two adjacent points on the left and right of the target are judged as the true coordinates,

[0015]

[0016]

[0017]

[0018]

[0019] The Bbox is a set of adjusted coordinate prediction values, the Bbox’ is a set of unadjusted coordinate prediction values, the GT is a set of coordinates of the target in the image, the Cs is the minimum closed region of the Bbox and the GT, and the Cs’ is the minimum closed region of the Bbox’ and the GT.

[0020]

[0021] θ = 0.1×(epoch - 100) 2 +eps

[0022] The epoch is the number of training iterations, eps is a constant, and the area represents the true size of each target. As the number of training iterations increases, the weights of targets of different sizes are changed.

[0023] QFL(C) = -|z - C| β ((1 - z)log(1 - C)+z log(C))

[0024] z is the IoU (Intersection over Union) between the Bbox and the GT, and C is a set of values of the probability that each target is classified as various classes.

[0025] The network includes a backbone sub-network, a first feature pyramid sub-network, a second feature pyramid sub-network, a main detection head sub-network, and an auxiliary detection head sub-network. Among them, the main detection head sub-network includes 4 main detection heads, and the auxiliary detection head sub-network includes 4 auxiliary detection heads. The backbone sub-network processes the input image into 3 features and inputs them into the first feature pyramid sub-network and the second feature pyramid sub-network respectively. The first feature pyramid sub-network and the second feature pyramid sub-network process the input 3 features into 4 features respectively and output them to the main detection head sub-network and the auxiliary detection head sub-network respectively. The 4 main detection heads of the main detection head sub-network respectively output: values representing the probability that each target is classified as various classes, adjusted coordinate prediction values of each target in the image, and unadjusted coordinate prediction values. The 4 auxiliary detection heads of the auxiliary detection head sub-network respectively output: values representing the probability that each target is classified as various classes, adjusted coordinate prediction values of each target in the image, and unadjusted coordinate prediction values.

[0026] Use the values representing the probabilities of each target being classified into various categories output by the main detection head subnet, the adjusted coordinate prediction values of each target in the image, and the unadjusted coordinate prediction values to calculate the loss function of the main detection head.

[0027] Use the values representing the probabilities of each target being classified into various categories output by the auxiliary detection head subnet, the adjusted coordinate prediction values of each target in the image, and the unadjusted coordinate prediction values to calculate the loss function of the auxiliary detection head.

[0028] Minimize the loss function of the main detection head and the loss function of the auxiliary detection head respectively, and adjust the network structure parameters to the optimal.

[0029] According to another aspect of the present invention, there is provided a computer program product encoded on a computer storage medium, including instructions that, when executed by one or more computers, will cause the one or more computers to execute the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flowchart showing a small target detection method for an unmanned aerial vehicle platform according to an embodiment of the present disclosure;

[0031] Figure 2 A network structure diagram showing an embodiment of the present disclosure;

[0032] Figure 3 A flowchart showing a network training processing method according to an embodiment of the present disclosure;

[0033] Figure 4 A structure diagram showing Stage-1 and Stage-2 according to an embodiment of the present disclosure;

[0034] Figure 5 Describe the structures of the first feature pyramid subnet and the second feature pyramid subnet;

[0035] Figure 6 Describe the structure of the main detection head of the main detection head subnet;

[0036] Figure 7 Describe the structure of the auxiliary detection head of the auxiliary detection head subnet; and

[0037] Figure 8 Is an example result of detecting an image through the network of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] Next, with reference to the accompanying drawings, embodiments of the present disclosure will be described. In the drawings, the same reference numerals are used for the same or corresponding structural elements. In addition, in the following description of the embodiments, repeated descriptions of the same or corresponding structural elements may be omitted. However, even if the description of a structural element is omitted, it does not mean that those structural elements are not included in a certain embodiment.

[0039] Although the process steps, method steps, algorithms, etc. are described in sequence in the flowcharts illustrated in this specification, these processes, methods, and algorithms can be set to operate in any suitable order. In other words, the steps of the processes, method steps, and algorithms described in various embodiments of the present disclosure do not need to be executed in the order described in the present disclosure.

[0040] In addition, even if some steps are described as not being executed simultaneously, in other embodiments, these partial steps can be executed simultaneously. In addition, the examples of the processes described through the drawings do not mean that the illustrated processes exclude other variations and modifications of them, and do not mean that any one of the illustrated processes or their steps is essential in more than one embodiment of the various embodiments of the present disclosure, nor does it mean that the illustrated process is preferred.

[0041] Figure 1 A flowchart showing a small target detection method for a drone platform according to an embodiment of the present disclosure.

[0042] In step S101, a data set is obtained. In the step of obtaining the data set, first, a publicly available data set is downloaded. The publicly available data set includes a pre-divided training set and a test set. The training set and the test set are independent of each other and are both subsets of the publicly available data set.

[0043] Then, in step S102, the images in the training set are preprocessed. By preprocessing the images in the training set, the image data can better adapt to the network of the present disclosure and, to a certain extent, reduce the computational cost of the algorithm.

[0044] In the step of preprocessing the images, first, the height and width of the images are restricted so that they can be divisible by 8. Then, the average image of the entire data set is subtracted from each image and normalized to ensure that each pixel value falls within [0, 1].

[0045] To expand the number of images in the training set, the images in the training set need to be randomly flipped and sheared. Specifically, the images in the training set are horizontally flipped as a whole with a probability of 0.5. A position is randomly selected in the image, and while maintaining the original aspect ratio of the image, a region of a random size is cropped. The part exceeding the original image is filled with 0. Then, the images that have been randomly flipped and sheared are added to the training set to expand the training set, increasing the number of images in the training set.

[0046] Next, hybrid splicing operations can be performed on multiple images in the extended training set. Specifically, 4 images can be randomly selected from the extended training set with a probability of 0.7 for random size scaling, and then spliced into an image with a size of 640×640. If hybrid splicing is not performed, the original image is scaled to 640×640. The above operations are performed on all images in the extended training set, so that the images in the extended training set are all 640×64 size images. The images processed through the above are more beneficial to the processing of the algorithm proposed in this disclosure. This size is only a preferred example of the algorithm proposed in this disclosure and does not limit the image size. Other sizes can be selected according to various situations.

[0047] After preprocessing the images in the training set in step S102, step S103 is entered. A batch of images in the extended training set are selected as sample images, and the selected sample images are input into the network, and the network is trained until the network fits.

[0048] The overall structure of the network is as Figure 2 shown. According to different positions of the deep and shallow layers of the network, the network can include a backbone sub-network, a first feature pyramid sub-network and a second feature pyramid sub-network, a main detection head sub-network, and an auxiliary detection head sub-network, where the backbone sub-network can adopt ShuffleNetV2. Refer to Figure 2 the network structure diagram and Figure 3 the network training processing method flow chart to describe the processing of the training network.

[0049] According to the network training processing method of the present disclosure, the selected sample images are input into the network, and the input images are represented in the form of features. Since the input image RGB is an image and has been preprocessed as above, the input feature size is 640×640, and the number of channels is 3. In operation S301, the input feature passes through the backbone sub-network, and 3 features P1, P2, and P3 are output, and the number of channels is 176, 352, and 704 respectively.

[0050] Specifically, in Figure 1 the structure diagram of the backbone sub-network is shown. The backbone sub-network includes 5 layers, and its specific structure is as follows:

[0051] The first convolutional layer Conv: 320×320, 24

[0052] The max pooling layer Maxpooling: 160×160, 24

[0053] The third layer Stage2: 80×80, 176

[0054] The fourth layer Stage3: 40×40, 352

[0055] The fifth layer, Stage4: 20×20, 704

[0056] The first layer of the backbone sub-network is the first convolutional layer. The input features pass through the first convolutional layer and output features with a size of 320×320 and 24 channels. The second layer of the backbone sub-network is the max pooling layer. The features output by the first convolutional layer become features with a size of 160×160 and 24 channels after being processed by the max pooling layer. Next, the features output by the max pooling layer enter the third layer, Stage2, of the backbone sub-network. After being processed by the third layer, Stage2, features P1 with a size of 80×80 and 176 channels are output. Then, they enter the fourth layer, Stage3, of the backbone sub-network, and features P2 with a size of 40×40 and 352 channels are output. Then, they enter the fifth layer, Stage4, of the backbone sub-network, and features P3 with a size of 20×20 and 704 channels are output. The sizes and numbers of channels of the features processed by each layer are only a preferred example of the network disclosed in the present application, and can be adjusted by those skilled in the art according to the situation under the teaching of the present disclosure.

[0057] The first convolutional layer and the max pooling layer are operations well-known to those skilled in the art and will not be elaborated here.

[0058] The third layer, Stage2, includes a first Stage-1 with 176 channels and a cascaded first to third Stage-2, each with 176 channels. The first Stage-1 is directly cascaded with the first to third Stage-2. The first Stage-1 outputs features with a size of 80×80 and 176 channels, which are input to the first Stage-2. The first Stage-2 outputs features with a size of 80×80 and 176 channels, which are input to the second Stage-2. The second Stage-2 outputs features with a size of 80×80 and 176 channels, which are input to the third Stage-2, and it outputs features P1 with a size of 80×80 and 176 channels.

[0059] The fourth layer, Stage3, includes a second Stage-1 with 352 channels and a cascaded fourth to tenth Stage-2, each with 352 channels. The second Stage-1 is directly cascaded with the fourth to tenth Stage-2. The second Stage-1 outputs features with a size of 40×40 and 352 channels, which are input to the fourth Stage-2. The fourth Stage-2 outputs features with a size of 40×40 and 352 channels, which are input to the fifth Stage-2, and so on. Finally, the tenth Stage-2 outputs features P2 with a size of 40×40 and 352 channels.

[0060] The fifth layer, Stage4, includes the third Stage-1 with 704 channels, and the cascaded eleventh to thirteenth Stage-2s, each with 704 channels. The third Stage-1 is directly cascaded with the eleventh to thirteenth Stage-2s. The third Stage-1 outputs features with a size of 20×20 and 704 channels, which are input to the eleventh Stage-2. The eleventh Stage-2 outputs features with a size of 20×20 and 704 channels, which are input to the twelfth Stage-2. The twelfth Stage-2 outputs features with a size of 20×20 and 704 channels, which are input to the thirteenth Stage-2. The thirteenth Stage-2 outputs features P3 with a size of 20×20 and 704 channels.

[0061] Now, the specific structures of Stage-1 and Stage-2 will be described with reference to Figure 4 Describe the specific structures of Stage-1 and Stage-2.

[0062] Specifically, the structures of the first to third Stage-1s are exactly the same, only the feature sizes and the number of channels are different. Therefore, the first to third Stage-1s are described together, and the specific structure is as follows:

[0063] stage-1: The features input to stage-1 are evenly divided into two groups according to the channels. The first group of features pass through the first conventional convolutional layer, the first BN layer, the first LeakRelu activation function, the DW convolution, the second BN layer, the second conventional convolutional layer, the third BN layer, the second LeakRelu activation function, and then are concatenated with the second group of features in the channel dimension, so that the output channels are the same as the input channels.

[0064] The structures of the first to third Stage-2s, the fourth to tenth Stage-2s, and the eleventh to thirteenth Stage-2s are exactly the same, only the number of channels is different. Therefore, the first to thirteenth Stage-2s are described together, and the specific structure is as follows:

[0065] stage-2: The features input to stage-2 are evenly divided into two groups according to the channels. The first group of features pass through the first conventional convolutional layer, the first BN layer, the first LeakRelu activation function, the first DW convolution, the second BN layer, the second conventional convolutional layer, the second BN layer, the second LeakRelu activation function, and then the second group passes through the second DW convolution, the third conventional convolutional layer, the third BN layer, the fourth conventional convolutional layer, the fourth BN layer, the third LeakRelu activation function. Finally, the two groups of features are concatenated in the channel dimension, so that the output channels are the same as the input channels.

[0066] In step S302, the three features P1, P2, and P3 output by the backbone sub-network are respectively input into the first feature pyramid sub-network and the second feature pyramid sub-network. The first feature pyramid sub-network outputs four features FL1, FL2, FL3, and FL4, with the number of channels being 128, 128, 128, and 128 respectively; the second feature pyramid sub-network outputs four features FA1, FA2, FA3, and FA4, with the number of channels being 256, 256, 256, and 256 respectively.

[0067] Refer to Figure 5 Describe the structures of the first feature pyramid sub-network and the second feature pyramid sub-network.

[0068] The structures of the first feature pyramid sub-network and the second feature pyramid sub-network are exactly the same, only the processed feature sizes and the number of channels are different. Therefore, the first feature pyramid sub-network and the second feature pyramid sub-network are described uniformly, and the specific structure is as follows:

[0069] The feature pyramid sub-network includes 20 layers:

[0070] The first Ghostconv layer

[0071] The first upsampling operation layer (Upsample)

[0072] The first channel concatenation operation layer (Cat)

[0073] The second Ghostconv layer

[0074] The second upsampling operation layer

[0075] The second channel concatenation operation layer

[0076] The third Ghostconv layer

[0077] The third upsampling operation layer

[0078] The fourth Ghostconv layer

[0079] The first downsampling operation layer (Downsample)

[0080] The third channel concatenation operation layer

[0081] The fifth Ghostconv layer

[0082] The second downsampling operation layer

[0083] The fourth channel concatenation operation layer

[0084] The sixth Ghostconv layer

[0085] The third downsampling operation layer

[0086] The fifth channel concatenation operation layer

[0087] The seventh Ghostconv layer

[0088] The fourth downsampling operation layer

[0089] The eighth Ghostconv layer

[0090] Ghostconv represents the Ghost convolution operation.

[0091] The feature pyramid sub-network first enriches features of different scales through upsampling and concatenation, and then gradually increases feature diversity through downsampling and concatenation, so as to obtain target features of different sizes, as well as local and global feature fusion.

[0092] Feature P3 is input into the first Ghostconv layer of the first feature pyramid sub-network. Through the first Ghostconv layer, a feature with a size of 20×20 and 128 channels is output. After upsampling by the first upsampling operation layer, a feature with a size of 40×40 and 128 channels is output. Then, in the first channel concatenation operation layer, it is concatenated with feature P2 in channels to output a feature with a size of 40×40 and 480 channels.

[0093] The output of the first channel concatenation operation layer enters the second Ghostconv layer, and a feature with a size of 40×40 and 128 channels is output. After upsampling by the second upsampling operation layer, a feature with a size of 80×80 and 128 channels is output. Then, in the second channel concatenation operation layer, it is concatenated with feature P1 in channels to output a feature with a size of 80×80 and 304 channels.

[0094] The output of the second channel concatenation operation layer enters the third Ghostconv layer, and a feature with a size of 80×80 and 128 channels is output. After upsampling by the third upsampling operation layer, a feature with a size of 160×160 and 128 channels is output.

[0095] The output of the third upsampling operation layer enters the fourth Ghostconv layer, and a feature with a size of 160×160 and 128 channels is output. After downsampling by the first downsampling operation layer, a feature with a size of 80×80 and 128 channels is output. Then, in the third channel concatenation operation layer, it is concatenated with the output of the third Ghostconv layer in channels to output a feature with a size of 80×80 and 256 channels.

[0096] The output of the third channel concatenation operation layer enters the fifth Ghostconv layer, outputting a feature FL4 with a size of 80×80 and 128 channels. After downsampling by the second downsampling operation layer, a feature with a size of 40×40 and 128 channels is output. Then, in the fourth channel concatenation operation layer, it is concatenated with the output of the second Ghostconv layer in channels, outputting a feature with a size of 40×40 and 256 channels.

[0097] The output of the fourth channel concatenation operation layer enters the sixth Ghostconv layer, outputting a feature FL3 with a size of 40×40 and 128 channels. After downsampling by the third downsampling operation layer, a feature with a size of 20×20 and 128 channels is output. Then, in the fifth channel concatenation operation layer, it is concatenated with the output of the first Ghostconv layer in channels, outputting a feature with a size of 20×20 and 256 channels.

[0098] The output of the fifth channel concatenation operation layer enters the seventh Ghostconv layer, outputting a feature FL2 with a size of 20×20 and 128 channels. After downsampling by the fourth downsampling operation layer, a feature with a size of 10×10 and 128 channels is output.

[0099] The output of the fourth downsampling operation layer enters the eighth Ghostconv layer, outputting a feature FL1 with a size of 10×10 and 128 channels.

[0100] Similarly, the feature P3 is input into the first Ghostconv layer of the second pyramid sub-network, outputting a feature with a size of 20×20 and 256 channels through the first Ghostconv layer. After upsampling by the first upsampling operation layer, a feature with a size of 40×40 and 256 channels is output. Then, in the first channel concatenation operation layer, it is concatenated with P2 in channels, outputting a feature with a size of 40×40 and 608 channels.

[0101] The output of the first channel concatenation operation layer enters the second Ghostconv layer, outputting a feature with a size of 40×40 and 256 channels. After upsampling by the second upsampling operation layer, a feature with a size of 80×80 and 256 channels is output. Then, in the second channel concatenation operation layer, it is concatenated with the feature P1 in channels, outputting a feature with a size of 80×80 and 432 channels.

[0102] The second channel concatenation operation layer enters the third Ghostconv layer, outputting a feature with a size of 80×80 and 256 channels. After upsampling by the third upsampling operation layer, a feature with a size of 160×160 and 256 channels is output.

[0103] The output of the third upsampling operation layer enters the fourth Ghostconv layer, and features with a size of 160×160 and 256 channels are output. After downsampling by the first downsampling operation layer, features with a size of 80×80 and 256 channels are output. Then, at the third channel concatenation operation layer, channel concatenation is performed with the output of the third Ghostconv layer, and features with a size of 80×80 and 512 channels are output.

[0104] The output of the third channel concatenation operation layer enters the fifth Ghostconv layer, and features FA4 with a size of 80×80 and 256 channels are output. After downsampling by the second downsampling operation layer, features with a size of 40×40 and 256 channels are output. Then, at the fourth channel concatenation operation layer, channel concatenation is performed with the output of the second Ghostconv layer, and features with a size of 40×40 and 512 channels are output.

[0105] The output of the fourth channel concatenation operation layer enters the sixth Ghostconv layer, and features FA3 with a size of 40×40 and 256 channels are output. After downsampling by the third downsampling operation layer, features with a size of 20×20 and 256 channels are output. Then, at the fifth channel concatenation operation layer, channel concatenation is performed with the output of the first Ghostconv layer, and features with a size of 20×20 and 512 channels are output.

[0106] The output of the fifth channel concatenation operation layer enters the seventh Ghostconv layer, and features FA2 with a size of 20×20 and 256 channels are output. After downsampling by the fourth downsampling operation layer, features with a size of 10×10 and 256 channels are output.

[0107] The output of the fourth downsampling operation layer enters the eighth Ghostconv layer, and features FA1 with a size of 10×10 and 256 channels are output.

[0108] In step S303, the four features FL1, FL2, FL3, and FL4 output by the first feature pyramid sub-network are respectively input to the four main detection heads of the main detection head sub-network. The regression branches of each main detection head respectively output CL1 / CL2 / CL3 / CL4, and their values represent the probabilities that each target is classified as various categories. In addition, the regression branches of each main detection head respectively output the adjusted coordinate prediction values RLd1 / RLd2 / RLd3 / RLd4. Among them, the adjusted coordinates RLd1 / RLd2 / RLd3 / RLd4 are obtained by multiplying the unadjusted coordinate prediction values RL1 / RL2 / RL3 / RL4 by the predicted coordinate offsets RL1’ / RL2’ / RL3’ / RL4’.

[0109] Now refer to Figure 6Describe the structure of the main detection head of the main detection head subnet.

[0110] The main detection head subnet includes four main detection heads. Each main detection head consists of two branches: classification and regression. The difference between each main detection head lies only in the parameters of the convolutional layer. The specific structure is as follows:

[0111] Classification branch:

[0112] First convolutional layer Conv: 128

[0113] Second convolutional layer Conv: 128

[0114] Third convolutional layer Conv: 10

[0115] Regression branch:

[0116] Fourth convolutional layer Conv: 128

[0117] Fifth convolutional layer Conv: 128

[0118] Sixth convolutional layer Conv: 32

[0119] Deformable convolutional operation layer (DCN)

[0120] Element multiplier

[0121] For the first main detection head, the classification branch includes two convolutional layers with an output channel number of 128 (i.e., the first convolutional layer and the second convolutional layer, which are depthwise separable convolutions with a kernel size of 5) and a convolutional layer with an output channel number of 10 (i.e., the third convolutional layer, which is a conventional convolution with a kernel size of 1). The feature FL1 output by the first feature pyramid subnet is input into its first convolutional layer, and a feature with a size of 10×10 and a channel number of 128 is output, then enters its second convolutional layer, and a feature with a size of 10×10 and a channel number of 128 is output. Finally, it is input into its third convolutional layer to output a feature CL1 with a size of N and a channel number of 10, and its value represents the probability that each target is classified into various categories, where N is the number of targets and changes dynamically with the input.

[0122] The regression branch includes a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a deformable convolutional operation (DCN) layer, and an element multiplier. Among them, the fourth convolutional layer and the fifth convolutional layer are depthwise separable convolutions with a convolutional kernel size of 5, and the sixth convolutional layer is a conventional convolution with a convolutional kernel size of 1. The feature FL1 output by the first feature pyramid subnetwork is input into its fourth convolutional layer, and a feature with a size of 10×10 and 128 channels is output, then input into its fifth convolutional layer to output a feature with a size of 10×10 and 128 channels, and finally input into its sixth convolutional layer to output a feature RL1 with a size of N and 32 channels. Its value represents the unadjusted coordinate prediction value of each target in the image, where N is the number of targets and changes dynamically with the input. At the same time, the output of its fifth convolutional layer is input into the DCN to output a feature RL1' with a size of N and 32 channels, which is the predicted output coordinate offset. At the element multiplier, the output feature RL1 of its sixth convolutional layer is multiplied element-wise with the output feature RL1' of the DCN to output an adjusted coordinate prediction value RLd1, which is a feature with a size of N and 32 channels.

[0123] Similarly, for the second main detection head, the classification branch includes two convolutional layers with 128 output channels (i.e., the first convolutional layer and the second convolutional layer, which are depthwise separable convolutions with a convolutional kernel size of 5) and a convolutional layer with 10 output channels (i.e., the third convolutional layer, which is a conventional convolution with a convolutional kernel size of 1). The feature FL2 output by the first feature pyramid subnetwork is input into its first convolutional layer, and a feature with a size of 20×20 and 128 channels is output, then enters its second convolutional layer to output a feature with a size of 20×20 and 128 channels, and finally input into its third convolutional layer to output a feature CL2 with a size of N and 10 channels. Its value represents the probability that each target is classified as various classes, where N is the number of targets and changes dynamically with the input.

[0124] The regression branch includes a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a deformable convolutional operation (DCN) layer, and an element multiplier. Among them, the fourth convolutional layer and the fifth convolutional layer are depthwise separable convolutions with a convolutional kernel size of 5, and the sixth convolutional layer is a conventional convolution with a convolutional kernel size of 1. The feature FL2 output by the first feature pyramid subnetwork is input to its fourth convolutional layer, and a feature with a size of 20×20 and 128 channels is output, then input to its fifth convolutional layer to output a feature with a size of 20×20 and 128 channels, and finally input to its sixth convolutional layer to output a feature RL2 with a size of N and 32 channels. Its value represents the unadjusted coordinate prediction value of each target in the image, where N is the number of targets and changes dynamically with the input. At the same time, the output of its fifth convolutional layer is input to the DCN to output a feature RL2' with a size of N and 32 channels, which is the predicted output coordinate offset. At the element multiplier, the output feature RL2 of its sixth convolutional layer is multiplied element-wise with the output feature RL2' of the DCN to output an adjusted coordinate prediction value RLd2, which is a feature with a size of N and 32 channels.

[0125] Similarly, for the third main detection head, the classification branch includes two convolutional layers with 128 output channels (i.e., the first convolutional layer and the second convolutional layer, which are depthwise separable convolutions with a convolutional kernel size of 5) and a convolutional layer with 10 output channels (i.e., the third convolutional layer, which is a conventional convolution with a convolutional kernel size of 1). The feature FL3 output by the first feature pyramid subnetwork is input to its first convolutional layer, and a feature with a size of 40×40 and 128 channels is output, then enters its second convolutional layer to output a feature with a size of 40×40 and 128 channels, and finally input to its third convolutional layer to output a feature CL2 with a size of N and 10 channels. Its value represents the probability that each target is classified as various classes, where N is the number of targets and changes dynamically with the input.

[0126] The regression branch includes a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a deformable convolutional operation (DCN) layer, and an element multiplier. Among them, the fourth convolutional layer and the fifth convolutional layer are depthwise separable convolutions with a convolutional kernel size of 5, and the sixth convolutional layer is a conventional convolution with a convolutional kernel size of 1. The feature FL3 output by the first feature pyramid subnetwork is input to its fourth convolutional layer, and a feature with a size of 40×40 and 128 channels is output, then input to its fifth convolutional layer to output a feature with a size of 40×40 and 128 channels, and finally input to its sixth convolutional layer to output a feature RL3 with a size of N and 32 channels. Its value represents the unadjusted coordinate prediction value of each target in the image, where N is the number of targets and changes dynamically with the input. At the same time, the output of its fifth convolutional layer is input to the DCN, and a feature RL3' with a size of N and 32 channels is output, which is the predicted output coordinate offset. At the element multiplier, the output feature RL3 of its sixth convolutional layer is multiplied element-wise with the output feature RL3' of the DCN to output the adjusted coordinate prediction value RLd3, which is a feature with a size of N and 32 channels.

[0127] Similarly, for the fourth main detection head, the classification branch includes two convolutional layers with 128 output channels (i.e., the first convolutional layer and the second convolutional layer, which are depthwise separable convolutions with a convolutional kernel size of 5) and a convolutional layer with 10 output channels (i.e., the third convolutional layer, which is a conventional convolution with a convolutional kernel size of 1). The feature FL4 output by the first feature pyramid subnetwork is input to its first convolutional layer, and a feature with a size of 40×40 and 128 channels is output, then enters its second convolutional layer to output a feature with a size of 40×40 and 128 channels, and finally input to its third convolutional layer to output a feature CL2 with a size of N and 10 channels. Its value represents the probability that each target is classified into various categories, where N is the number of targets and changes dynamically with the input.

[0128] The regression branch includes a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a deformable convolutional operation (DCN) layer, and an element multiplier. Among them, the fourth convolutional layer and the fifth convolutional layer are depthwise separable convolutions with a convolutional kernel size of 5, and the sixth convolutional layer is a conventional convolution with a convolutional kernel size of 1. The feature FL4 output by the first feature pyramid subnetwork is input to its fourth convolutional layer, and a feature with a size of 80×80 and 128 channels is output, then input to its fifth convolutional layer to output a feature with a size of 80×80 and 128 channels, and finally input to its sixth convolutional layer to output a feature RL4 with a size of N and 32 channels, whose value represents the unadjusted coordinate prediction value of each target in the image, where N is the number of targets and changes dynamically with the input. At the same time, the output of its fifth convolutional layer is input to the DCN, and a feature RL4' with a size of N and 32 channels is output, which is the predicted output coordinate offset. At the element multiplier, the output feature RL4 of its sixth convolutional layer is multiplied element-wise with the output feature RL4' of the DCN to output the adjusted coordinate prediction value RLd4, which is a feature with a size of N and 32 channels.

[0129] In the main detection head subnetwork, the 4 features output by the first feature pyramid subnetwork are respectively input to 4 main detection heads, where the side lengths of these 4 features are halved one by one. Usually, the regression branch of a single-stage detection head directly outputs the position coordinates of the predicted target, but the regression accuracy of the predicted coordinate position is low after only one prediction, and the prediction effect is worse than that of the two-stage. Learning from the idea of two-stage object detection to improve the single-stage object detection algorithm, this algorithm adopts a coordinate adjustment strategy to further improve the detection accuracy without significantly increasing the inference time.

[0130] In step S304, the 4 features FA1, FA2, FA3, and FA4 output by the second feature pyramid subnetwork are respectively input to the 4 auxiliary detection heads of the auxiliary detection head subnetwork. The classification branches of each auxiliary detection head respectively output CA1 / CA2 / CA3 / CA4, whose values represent the probabilities of each target being classified into various categories. The regression branches of each auxiliary detection head respectively output features RA1 / RA2 / RA3 / RA4, whose values represent the unadjusted coordinate prediction values of each target in the image, and the adjusted coordinate prediction values RAd1 / RAd2 / RAd3 / RAd4 are output. Among them, the adjusted coordinates RAd1 / RAd2 / RAd3 / RAd4 are obtained by multiplying the unadjusted coordinate prediction values RA1 / RA2 / RA3 / RA4 with the predicted coordinate offsets RA1' / RA2' / RA3' / RA4'.

[0131] Now refer to Figure 7 Describe the structure of the auxiliary detection head of the auxiliary detection head subnetwork.

[0132] The auxiliary detection head subnet includes four auxiliary detection heads. Each auxiliary detection head consists of two branches: a classification branch and a regression branch. The only difference between each auxiliary detection head lies in the parameters of the convolutional layer. The specific structure is as follows:

[0133] Classification branch:

[0134] Conv: 256

[0135] Conv: 256

[0136] Conv: 10

[0137] Regression branch:

[0138] Conv: 256

[0139] Conv: 256

[0140] Conv: 32

[0141] Deformable Convolutional Network (DCN) layer

[0142] Element-wise multiplier

[0143] Among them, the classification branch and the regression branch have two Conv: 256, which are two stacked depthwise separable convolutional layers with 256 channels. And the two Conv: 256 in the classification branch and the regression branch can be shared. Conv: 10 is a conventional convolutional layer with 10 channels, and Conv: 32 is a conventional convolutional layer with 32 channels.

[0144] Since the processing in the auxiliary detection head is similar to that in the main detection head, it will not be elaborated here. The processing flow of the auxiliary detection head is briefly described as follows:

[0145] For the classification branch, the features FA1 / FA2 / FA3 / FA4 are used as inputs. The channels of the features FA1 / FA2 / FA3 / FA4 are 256. Through two stacked depthwise separable convolutional layers with a kernel size of 5, and then through one conventional convolutional layer with a kernel size of 1, the output features CA1 / CA2 / CA3 / CA4 are obtained. Their values represent the probabilities of each target being classified into various classes, with a size of N and 10 channels, where N is the number of targets and changes dynamically with the input.

[0146] For the regression branch, the features FA1 / FA2 / FA3 / FA4 are used as inputs. The number of channels of the features FA1 / FA2 / FA3 / FA4 is 256. Through 2 stacked depthwise separable convolutions with a kernel size of 5, the features FA1’ / FA2’ / FA3’ / FA4’ are obtained. Then, through 1 conventional convolution with a kernel size of 1, the output features RA1 / RA2 / RA3 / RA4 are obtained. Their values represent the unadjusted coordinate prediction values of each target in the image, and the number of channels is 32. At the same time, FA1’ / FA2’ / FA3’ / FA4’ are passed through the deformable convolution operation layer to predict the output coordinate offsets RA1’ / RA2’ / RA3’ / RA4’. RA1’ / RA2’ / RA3’ / RA4’ are multiplied element-wise with RA1 / RA2 / RA3 / RA4, and the adjusted coordinate prediction values RAd1 / RAd2 / RAd3 / RAd4 are output, with a size of N and a channel number of 32.

[0147] In step S305, the network structure parameters are adjusted to the optimal by minimizing the loss function to achieve network fitting.

[0148] Specifically, the network training strategy: Aiming at the problem of low accuracy of small target detection in the general object detection algorithm, this method adopts the training method of curriculum learning. At different stages of network training, the loss weights of different-sized targets are adjusted to improve the detection accuracy of small targets. Specifically, since large targets have rich information and are easy to train, the weights of large targets are reduced in the initial stage of training, and the training cycle of small targets is extended.

[0149] When training the network of this algorithm, the loss function consists of two parts: the main detection head loss and the auxiliary detection head loss. The calculation methods of both are the same. The following takes the main detection head as an example for detailed description. As shown in Equation (1), CL1 / CL2 / CL3 / CL4 are combined into 1 set C, RLd1 / RLd2 / RLd3 / RLd4 are combined to obtain the set Bbox of the adjusted coordinate prediction values, RL1 / RL2 / RL3 / RL4 are combined to obtain the set Bbox’ of the unadjusted coordinate prediction values, and the network structure parameters are adjusted to the optimal by minimizing the loss function.

[0150] Represents the main detection head loss, Represents the auxiliary detection head loss, and the calculation method is the same. As shown in Equation (1),

[0151]

[0152] DFL(S) = -((y i+1 - y) log(S i ) + (y - y i ) log(S i+1)) (2)

[0153] Where DFL represents the distribution probability loss function of the adjusted coordinate prediction value Bbox, y is the coordinate of the target's true position, y i and y i+1 Indicates the coordinates of the two points on the left and right of the target, which are obtained by rounding up and rounding down the Bbox coordinates respectively. i and S i+1 Indicates the probability that the target's left and right neighbors are identified as true coordinates.

[0154]

[0155]

[0156] GIoU b Represents the GIoU loss function of the unadjusted target position, GIoU r Represents the GIoU loss function of the adjusted target position.

[0157]

[0158]

[0159] Bbox is a set of adjusted coordinate prediction values, Bbox' is a set of unadjusted coordinate prediction values, GT is a set of coordinates of the target in the image, Cs is the minimum closure area of Bbox and GT, and Cs' is the minimum closure area of Bbox and GT.

[0160]

[0161] θ=0.1×(epoch-100) 2 +eps (6)

[0162] GIoU c Represents the loss function of course learning, where epoch is the number of training iterations, which can be selected based on experience before training. eps is a constant, which can be 324 here, or other values can be selected based on the situation. area represents the actual size of each target. As the number of training iterations increases, the weights of targets of different sizes can be changed.

[0163] The weight is

[0164] In the training strategy of curriculum learning, the weight of large-size targets is reduced to less than 1 at the beginning of training. As the training generations increase, the weight of large targets is gradually increased to close to 1, while the weight of small targets remains unchanged at 1, which in disguise prolongs the training time of small targets and thus improves the detection accuracy of small targets.

[0165] QFL(C) = -|z - C| β ((1 - z)log(1 - C) + z log(C)) (7)

[0166] QFL represents the loss function of the classification branch prediction output, and z represents the IoU (Intersection over Union) between the Bbox and the GT.

[0167] Auxiliary detection head loss The calculation formula of is exactly the same as that of the main detection head loss However, the parameter values are different, so the final loss values are also different. Here, the calculation formula of the auxiliary detection head loss

[0168] By minimizing the main detection head loss and the auxiliary detection head loss respectively, the parameters are adjusted to the optimal, and thus the network training is completed.

[0169] In step S104, after completing the network training, using the optimal parameters, the detection results are output through the network. The detection results include the values representing the probabilities that each target is discriminated as various classes, the predicted values of the adjusted coordinates of each target in the image, and the predicted values of the unadjusted coordinates. The detection results can be expressed as [category, abscissa of the upper - left corner point, ordinate of the upper - left corner point, abscissa of the lower - right corner point, ordinate of the lower - right corner point], and then rectangular boxes are drawn on the original image according to the coordinates to represent the detected targets.

[0170] After completing the network training, the images in the test set can be input into the network for testing. When the test set is input into the network of this algorithm for object detection, the detection results of all targets in each image are output in the format of [category, abscissa of the upper - left corner point, ordinate of the upper - left corner point, abscissa of the lower - right corner point, ordinate of the lower - right corner point], and then rectangular boxes are drawn on the original image according to the coordinates to represent the detected targets.

[0171] Figure 8 An example result of detecting an image through the network of the present disclosure is shown. The first column is the input image, and the second column is the detection result image. It can be seen that the accuracy of detecting targets through the network of the present disclosure is very high.

[0172] The present disclosure draws on the fine - tuning idea of the two - stage object detection algorithm. Based on the single - stage object detection model, deformable convolutions are used to fine - tune the target positions of the predicted output, making the prediction accuracy from coarse to fine, and improving the target localization accuracy and accuracy rate.

[0173] To address the problem that small targets have little information and are difficult to train, a curriculum learning training strategy is adopted. In the early stage of training, the loss weight of large-sized targets is reduced to less than 1. As the training generations increase, the weight of large targets is gradually increased to close to 1, while the weight of small targets remains unchanged at 1, thereby extending the training time of small targets and improving the detection accuracy of small targets.

[0174] To address the security issue of model loading relying on source code, this algorithm can use the TensorRT format to encapsulate all weight files, and load the model using the TensorRT library loading method. The traditional weight file format pt is only used as an intermediate form in this algorithm and does not participate in deployment. The final deployment does not need to rely on the model structure source code.

[0175] The specific feature sizes and channel numbers involved in the description of this article are only for the convenience of describing the technical solutions. Ordinary technicians in this field will adopt various other feature sizes and channel numbers based on the description of this article, and these changes are included in the scope of the attached claims.

[0176] The embodiments of the present disclosure can be written as computer programs and can be implemented in general-use digital computers that execute the programs using a computer readable recording medium.

[0177] In this state, the medium can continuously store programs that can be executed by the computer, or can temporarily store programs for execution or downloading. In addition, the medium can be a variety of recording devices or storage devices that combine single or multiple hardware, is not limited to media that are directly connected to the computer system, and can exist in a distributed manner on the network. Examples of media include magnetic storage media (such as floppy disks or hard disks) configured to store program instructions, optical recording media (such as CD-ROMs or DVDs), magneto-optical media (such as floppy disks), and Rom, RAM, flash memory, etc. In addition, examples of other media can include: application stores for distributing applications, sites for providing or distributing various other software, and recording media or storage media managed at servers.

[0178] Although the present disclosure has been particularly shown and described with reference to preferred embodiments using specific terminology, the embodiments and terminology should be considered in a descriptive sense only and not for purposes of limitation. Therefore, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. A small target detection method for an unmanned aerial vehicle platform, the method comprising: Obtaining an image set including a training set and a test set; Preprocessing the images in the training set; Inputting the preprocessed images into a network and training the network until the network is fitted. The steps of training the network include: obtaining values representing the probabilities that each target is classified into various categories, coordinate prediction values of each target adjusted in the image, and unadjusted coordinate prediction values, calculating a loss function based on the above parameters, and adjusting the network structure parameters to the optimal by minimizing the loss function; Outputting a detection result using the optimal parameters, the detection result including values representing the probabilities that each target is classified into various categories, coordinate prediction values of each target adjusted in the image, and unadjusted coordinate prediction values; wherein the loss function is calculated by the following formula: where DFL(S) = -((y i+1 - y) log(S i )) + (y - y i ) log(S i+1 )) y is the value of the coordinate of the target in the image, y i and y i+1 represent the coordinates of the two points adjacent to the target on the left and right, which are obtained by rounding up and rounding down the coordinates respectively, S i and S i+1 represent the probabilities that the two points adjacent to the target on the left and right are judged as true coordinates Bbox is a set of adjusted coordinate prediction values, Bbox’ is a set of unadjusted coordinate prediction values, GT is a set of coordinates of the target in the image, Cs is the minimum closed region of Bbox and GT, Cs’ is the minimum closed region of Bbox’ and GT; θ = 0.1×(epoch - 100) 2 + eps Epoch is the number of training iterations, eps is a constant, and area represents the true size of each target. As the number of training iterations increases, the weights of targets with different sizes are changed QFL(C) = -|z - C| β ((1 - z)log(1 - C) + zlog(C)) z is the IoU (Intersection over Union) between Bbox and GT, and C is a set of values representing the probabilities that each target is classified into various categories.

2. The method according to claim 1, characterized in that Preprocessing the images in the training set includes: Limiting the height and width of the images so that they can be divisible by 8, subtracting the average image of the entire data set from each image and normalizing, and randomly flipping and shearing the images in the training set, and adding the images that have been randomly flipped and sheared to the training set to increase the number of images in the training set.

3. The method according to claim 1, characterized in that, The network includes a backbone sub-network, a first feature pyramid sub-network and a second feature pyramid sub-network, a main detection head sub-network and an auxiliary detection head sub-network. Among them, the main detection head sub-network includes 4 main detection heads, and the auxiliary detection head sub-network includes 4 auxiliary detection heads; The backbone sub-network processes the input image into 3 features and inputs them into the first feature pyramid sub-network and the second feature pyramid sub-network respectively. The first feature pyramid sub-network and the second feature pyramid sub-network respectively process the 3 input features into 4 features and output them to the main detection head sub-network and the auxiliary detection head sub-network respectively. The 4 main detection heads of the main detection head sub-network respectively output: values representing the probabilities that each target is classified into various categories, coordinate prediction values of each target adjusted in the image, and unadjusted coordinate prediction values. The 4 auxiliary detection heads of the auxiliary detection head sub-network respectively output: values representing the probabilities that each target is classified into various categories, coordinate prediction values of each target adjusted in the image, and unadjusted coordinate prediction values.

4. The method according to claim 3, wherein Calculating the loss function of the main detection head using the values representing the probabilities that each target is classified into various categories, coordinate prediction values of each target adjusted in the image, and unadjusted coordinate prediction values output by the main detection head sub-network.

5. The method according to claim 4, wherein Using the values representing the probabilities of each target being discriminated into various categories output by the auxiliary detection head subnet, the adjusted coordinate prediction values of each target in the image, and the unadjusted coordinate prediction values, calculate the loss function of the auxiliary detection head.

6. The method according to claim 5, characterized in that, Minimize the loss function of the main detection head and the loss function of the auxiliary detection head respectively, and adjust the network structure parameters to the optimal.

7. A computer program product encoded on a computer storage medium, including instructions that, when executed by one or more computers, will cause the one or more computers to execute the method according to any one of claims 1-6.

Images