A directional target detection method based on midpoint offset expression

Abstract

The present invention provides a directed target detection method expressed by midpoint offset. A directed target detection network expressed by midpoint offset is constructed, which mainly includes a backbone network, a directed candidate box generation network and a directed target detection head. The backbone network is used for feature extraction of input images, the directed candidate box generation network is used for generating high-quality candidate boxes, and the directed target detection head performs fine regression and classification on the directed candidate boxes. The network is trained in an end-to-end manner, and the output detection results are post-processed using a non-maximum suppression method to obtain the final detection results. The present invention can solve the problem of inefficient generation of existing directed candidate boxes and realize accurate and fast directed target detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a directional target detection method expressed by midpoint offset, which can simultaneously improve the accuracy and speed of directional target detection. Background Art

[0002] Directed object detection is an important part of computer vision and a key technology in the fields of aerial image object detection, scene text detection, densely packed object detection, etc. It has received extensive attention. After decades of development, directed object detection has made great progress, especially driven by deep learning.

[0003] At present, directed object detection based on deep learning can be roughly divided into two categories: one is two-stage directed object detection based on candidate generation, and the other is single-stage directed object detection based on direct regression. In terms of detection accuracy, two-stage directed object detection based on candidate generation occupies a dominant position, but the generation of directed candidate boxes in two-stage directed object detection is very time-consuming, which seriously restricts the speed of two-stage directed object detection.

[0004] In the early stage, the two-stage oriented object detection mainly generates oriented candidate boxes by placing preset boxes of different scales, aspect ratios and angles. For example, Rotated RPN places 54 preset boxes (3 scales × 3 aspect ratios × 6 angles) at each position of the feature map, and then regresses the preset boxes to obtain oriented candidate boxes. Although Rotated RPN improves the recall rate of oriented object detection, it also increases the computational cost and memory usage. Later, in order to avoid placing dense oriented preset boxes, the two-stage oriented object detection generates oriented candidate boxes through RoI Transformer. RoITransformer first regresses horizontal candidate boxes from preset boxes, and then generates oriented candidate boxes with horizontal candidate boxes. RoITransformer significantly improves the accuracy of oriented object detection, but the entire generation process is still time-consuming. Therefore, generating high-quality oriented candidate boxes in a low-computational cost manner is the key to breaking through the computational bottleneck of oriented object detection. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a directed target detection method expressed by a midpoint offset. The midpoint offset expression is a directed target representation method, which adds two parameters, namely, the top edge of the horizontal frame and the right midpoint offset, on the basis of the horizontal frame expression (center coordinates, width and height). The present invention constructs a directed target detection network expressed by a midpoint offset, which mainly includes a backbone network, a directed candidate frame generation network and a directed target detection head. The backbone network is used for feature extraction of the input image, the directed candidate frame generation network is used for generating high-quality candidate frames, and the directed target detection head performs fine regression and classification on the directed candidate frames; the network is trained in an end-to-end manner, and the output detection results are post-processed using a non-maximum suppression method to obtain the final detection results. The present invention can solve the problem of inefficient generation of existing directed candidate frames and realize accurate and fast directed target detection.

[0006] A directed target detection method using midpoint offset expression, characterized by the following steps:

[0007] Step 1, construct a directed target detection network expressed by midpoint offset: it mainly includes three parts: backbone network, directed candidate box generation network and directed target detection head. Among them, the backbone network adopts ResNet50 residual network, the directed candidate box generation network contains a fully convolutional neural network composed of 1 3×3 convolutional layer and two parallel 1×1 convolutional layers, one 1×1 convolutional layer is used for classification, and the other 1×1 convolutional layer is used for regression. The directed target detection head contains a fully connected network composed of 2 fully connected layers with 1024 dimensions and 2 parallel fully connected layers of different dimensions. One hidden layer dimension of the parallel fully connected layer is C+1, which is used for classification, and the other hidden layer dimension is 5×(C+1), which is used for regression. C is the number of target detection categories contained in the image dataset;

[0008] Input any image, get the feature map of the image through the backbone network, pass the feature map through the directed candidate box generation network, output the directed candidate box of the target, pass the feature map and the directed candidate box of the target through the directed target detection head, output the position and category of the directed candidate box of the target, the specific processing process is as follows:

[0009] Step a: Input image I passes through the ResNet50 residual network to obtain four levels of feature maps P2, P3, P4, and P5, where P i Represents the feature map of the i-th level, i = 2, 3, 4, 5;

[0010] Step b: The feature maps P2, P3, P4, and P5 are input into the directed candidate box generation network. First, the feature map P5 is downsampled by 2 times to obtain the feature map P6. Then, three preset boxes with aspect ratios of 1:1, 1:2, and 2:1 are placed on each feature point of the feature maps P2, P3, P4, P5, and P6. Each feature map is then passed through a full convolutional network to obtain the score and regression offset of the preset box belonging to the foreground. The regression offset includes the preset box parameters and the horizontal prediction box parameters, which are recorded as δ = (δ x ,δ y ,δ w ,δ h ,δ α ,δ β ), where (δ x ,δ y ) is the offset of the center point coordinates of the preset frame, δ w and δ h Respectively represent the width and height offset of the preset box, δ α and δ β Respectively represent the offset of the top edge and the right midpoint of the horizontal prediction box;

[0011] According to the regression offset, decode as follows to get other parameters of the horizontal prediction box:

[0012]

[0013] Where (x, y) is the coordinate of the midpoint of the horizontal prediction frame, w and h are the width and height of the horizontal prediction frame respectively, Δα is the horizontal offset of the midpoint of the top edge of the horizontal prediction frame, and Δβ is the vertical offset of the midpoint of the right edge of the horizontal prediction frame; (a x ,a y ) is the center point coordinate of the preset frame, a w and a h Respectively represent the width and height of the preset box;

[0014] The horizontal prediction box is decoded according to the midpoint offset expression formula to obtain the coordinate vectors v = {v1, v2, v3, v4} of the four vertices of the directed parallelogram candidate box:

[0015]

[0016] Among them, v1 represents the upper left vertex of the directed parallelogram candidate box, v2 represents the upper right vertex of the directed parallelogram candidate box, v3 represents the lower right vertex of the directed parallelogram candidate box, and v4 represents the lower left vertex of the directed parallelogram candidate box;

[0017] Extend the length of the short diagonal of the directed parallelogram candidate box to make its short diagonal and long diagonal the same length, and get the directed rectangular candidate box (xo ,y o ,w o ,h o ,θ), where (x0, y0) is the coordinate of the midpoint of the directed rectangular candidate box, w0 and h0 are the width and height of the directed rectangular candidate box, respectively, and θ∈[-90°, 90°] is the angle between the long side of the directed rectangular candidate box and the horizontal axis;

[0018] Step c: Input the directed rectangular candidate box and feature maps P2, P3, P4, and P5 into the directed detection head. First, project the directed rectangular candidate box onto the feature map P k superior, k0 is equal to 4; then, the directed rectangular candidate box projected onto the feature map is subjected to a directed RoIAlign operation to obtain the RoI feature of the directed rectangular candidate box; the RoI feature is passed through a fully connected network to obtain the score vector q = {q0, q1, L, q C} and its regression offset vector q0 represents the score of the directed rectangular candidate belonging to the background, q c Indicates the score of the directed rectangular candidate box belonging to the cth category; is the center point coordinate offset of the directed rectangular candidate box corresponding to the cth category, is the width offset of the directed rectangular candidate box corresponding to the cth class, is the offset of the height of the directed rectangular candidate box corresponding to the cth class, is the angle offset of the directed rectangular candidate box corresponding to the cth category;

[0019] Step 2, network training: Input the training image dataset into the directed object detection network expressed by the midpoint offset constructed in step 1, train the network in an end-to-end manner, and obtain a trained network; during the training, the network loss function L is set as follows:

[0020] L=L 1st +L 2nd (3)

[0021] Among them, L 1st is the loss of the directed candidate box generation network, L 2nd The loss of the detection head for oriented targets;

[0022] The loss L of the directed candidate box generation network 1st The calculation formula is:

[0023]

[0024] Among them, N1 represents the total number of positive and negative samples in a training batch of the directed candidate box generation network, and N P1is the number of positive samples, i represents the index of the sample, L ce (·) is the cross entropy loss function, L reg (·) is the Smooth L1 loss function, p i The score of the i-th sample output by the directed candidate box generation network belonging to the foreground, p i ′ is the true value of the i-th sample. If the i-th sample is a positive sample, then p i ′ is equal to 1. If the i-th sample is a negative sample, then p i ′ is equal to 0; δ i is the regression offset of the i-th preset box, t i ′ is the true value of the regression offset of the i-th preset box; the positive sample refers to a preset box whose intersection-and-union ratio with any external rectangular box is greater than 0.7, or a preset box with the maximum intersection-and-union ratio with the external rectangular box and the value of the intersection-and-union ratio is greater than 0.3; the negative sample refers to a preset box whose intersection-and-union ratio with the external rectangular box is less than 0.3; the external rectangular box refers to the horizontal external rectangle of the target directional annotation box in the image data set;

[0025] The loss L of the oriented object detection head 2nd The calculation formula is:

[0026]

[0027] Among them, N2 represents the total number of positive and negative samples in a training batch of the directional detection head, N P2 is the number of positive samples; u represents the category of the i-th sample, u=0,1,2,...,C; q i is the score vector of the i-th sample belonging to each class, output by the directed object detection head, q i ′ is the true value vector of the score of the i-th sample belonging to each class; when the i-th sample is a negative sample, the value of [u≥1] is 0, and when the i-th sample is a positive sample, the value of [u≥1] is 1; represents the regression offset true value of the i-th sample with respect to category u, σ i (u) is the prediction offset of the i-th sample with respect to category u; the positive sample mentioned here refers to a directed rectangular candidate box whose intersection-with-unit ratio with the target directed labeled box is greater than 0.5, and the negative sample refers to a directed rectangular candidate box whose intersection-with-unit ratio with the target directed labeled box is less than 0.5;

[0028] Step 3, directed target detection expressed by midpoint offset: Input the image to be detected into the directed candidate box generation network trained in step 2, output the position and foreground score of the directed rectangular candidate box, then use the non-maximum suppression method to process the directed rectangular candidate box, set the intersection-over-union ratio threshold of non-maximum suppression to 0.8, select the first 1000 directed rectangular candidate boxes with the largest scores after processing and input them into the trained directed target detection head, and output the score vector q = {q0, q1, L, q C} and the regression offset corresponding to each class The classification scores are normalized as follows to obtain the confidence level for each category:

[0029]

[0030] Among them, c Represents the confidence that the directed rectangular candidate box belongs to the cth category, c = 0, 1, 2, ..., C;

[0031] The class corresponding to the maximum confidence value is taken as the classification result of the directed rectangular candidate box, and its regression offset is decoded as follows to obtain the final directed detection box:

[0032]

[0033] in, is the center point coordinate of the oriented detection box, and The width and height of the detection box are respectively, is the angle between the long side of the directed detection box and the horizontal axis; (x0, y0) is the coordinate of the midpoint of the directed rectangular candidate box, w0 and h0 are the width and height of the directed rectangular candidate box, respectively, and θ is the angle between the long side of the directed rectangular candidate box and the horizontal axis;

[0034] The polygon non-maximum suppression method is used to process the final directional detection box of each class. The intersection-over-union ratio threshold of polygon non-maximum suppression is set to 0.5, and the processed directional detection box and its corresponding category are used as the final output detection result.

[0035] The beneficial effects of the present invention are as follows: since the midpoint offset expression method is adopted in the directed candidate box generation network, the complicated design of the directed candidate box generation network is avoided, and high-quality directed candidate boxes can be generated at a low computational cost; since the directed candidate box generation network based on the midpoint offset expression is used for directed target detection, the accuracy and speed of two-stage directed target detection can be significantly improved, and the directed target detection method based on the midpoint offset expression can be applied to the fields of aerial image target detection, scene text detection and dense target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the structure of the directional target detection network expressed by the midpoint offset of the present invention;

[0037] Figure 2 It is a schematic diagram of expressing the midpoint offset of the present invention;

[0038] In the figure, (a) the principle diagram of midpoint offset expression, (b) the example diagram of midpoint offset expression;

[0039] Figure 3 It is a result image of directed target detection using the method of the present invention;

[0040] In the figure, (a) large vehicle detection result, (b) small vehicle detection result, (c) ship detection result, and (a) aircraft detection result. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0042] The operating environment of this embodiment: 10 Karsten Heli GPU servers (CPU is Intel Xeon E5-2698, GPU is 12G RTX 2080Ti), the server operating system is Ubuntu 16.04.5LTS, and the experimental code is developed based on mmdetection2.2. The experiment uses the public directed target detection data set DOTA1-0 (containing 2806 pictures, 15 categories and 188282 instances), and the format of the annotation box is a directed rectangular box. By cropping the original DOTA1-0 image, image blocks of size 1024×1024 are obtained. The sliding window size used for cropping is 1024×1024, and the step size is 824. These image blocks are used as network training images.

[0043] The specific implementation process of the present invention is as follows:

[0044] 1. Build a network

[0045] The overall framework of the directional target detection network expressed by midpoint offset is as follows Figure 1As shown in the figure, it mainly includes three parts: backbone network, directed candidate box generation network and directed object detection head. Among them, the backbone network is a ResNet50 residual network with a feature pyramid (FPN) structure. The directed candidate region generation network contains a fully convolutional network, which is built on the RPN network: one 3×3 convolutional layer is followed by two parallel 1×1 convolutional layers (for regression and foreground / background classification, respectively). The number of feature map channels output by the 1×1 foreground / background classification convolutional layer is consistent with that of the RPN network, both of which are 3, and the number of feature map channels output by the 1×1 regression convolutional layer is changed from 12 in RPN to 18. The directed object detection head contains a fully connected network, which is built on the basis of Fast R-CNN (adding angle regression parameters to the regression fully connected layer): two hidden layers are both 1024-dimensional fully connected layers followed by two parallel fully connected layers for classification and regression, where the hidden layer dimension of the classification fully connected layer is 16, and the hidden layer dimension of the regression fully connected layer is 80.

[0046] The specific processing process of the entire network is as follows:

[0047] (1) Input any image and pass it through the ResNet50 residual network to obtain four levels of feature maps P2, P3, P4, and P5, where P i Represents the feature map of the i-th level, i = 2, 3, 4, 5.

[0048] (2) The feature maps P2, P3, P4, and P5 are input into the directed candidate box generation network. First, the feature map P5 is downsampled by 2 times to obtain the feature map P6. Then, three preset boxes with aspect ratios of 1:1, 1:2, and 2:1 are placed on each feature point of the feature maps P2, P3, P4, P5, and P6. Each feature map is then passed through a full convolutional network to obtain the preset box a=(a x ,a y ,a w ,a h ) belongs to the foreground score p and the regression offset δ = (δ x ,δ y ,δ w ,δ h ,δ α ,δ β ), where (a x ,a y ) is the center point coordinate of the preset frame, a w and a h Respectively represent the width and height of the preset box, (δ x ,δ y ) is the offset of the center point coordinates of the preset frame, δ w and δ hRespectively represent the width and height offset of the preset frame; δ α and δ β Represents the offset of the top and right midpoints of the horizontal prediction box respectively.

[0049] According to the regression offset, decode as follows to get other parameters of the horizontal prediction box:

[0050]

[0051] Where (x, y) is the coordinate of the midpoint of the horizontal prediction box, w and h are the width and height of the horizontal prediction box, respectively, Δα is the horizontal offset of the midpoint of the top edge of the horizontal prediction box, and Δβ is the vertical offset of the midpoint of the right edge of the horizontal prediction box.

[0052] The horizontal prediction box is decoded according to the midpoint offset expression formula to obtain the coordinate vectors v = {v1, v2, v3, v4} of the four vertices of the directed parallelogram candidate box:

[0053]

[0054] Among them, v1 represents the upper left vertex of the directed parallelogram candidate box, v2 represents the upper right vertex of the directed parallelogram candidate box, v3 represents the lower right vertex of the directed parallelogram candidate box, and v4 represents the lower left vertex of the directed parallelogram candidate box.

[0055] Figure 2 (a) shows the schematic diagram of the midpoint offset expression. Figure 2 (b) shows an example image of the midpoint offset expression.

[0056] Extend the length of the short diagonal of the directed parallelogram candidate box to make its short diagonal and long diagonal the same length, and get the directed rectangular candidate box (x o ,y o ,w o ,h o ,θ), where (x0, y0) are the coordinates of the midpoint of the directed rectangular candidate box, w0 and h0 are the width and height of the directed rectangular candidate box, respectively, and θ∈[-90°, 90°] is the angle between the long side of the directed rectangular candidate box and the horizontal axis.

[0057] (3) The directed rectangular candidate box and feature maps P2, P3, P4, and P5 are input to the directed detection head. First, the directed rectangular candidate box is projected onto the feature map P k superior, k0 is equal to 4; then, the directed rectangular candidate box projected onto the feature map is subjected to a directed RoIAlign operation to obtain the RoI feature of the directed rectangular candidate box, whose size is 7×7×256; the RoI feature is passed through a fully connected network to obtain the score vector q={q0,q1,L,q C} and its regression offset vector c=0,1,2,…,C, q0 represents the score of the directed rectangular candidate belonging to the background, q c Indicates the score of the directed rectangular candidate box belonging to the cth category; is the center point coordinate offset of the directed rectangular candidate box corresponding to the cth category, is the width offset of the directed rectangular candidate box corresponding to the cth class, is the offset of the directed rectangular candidate box corresponding to the cth class, is the angle offset of the oriented rectangular candidate box corresponding to the cth class.

[0058] For the oriented RoIAlign operation, see the document "Jian Ding, Nan Xue, Yang Long, Gui-Song Xia, and Qikai Lu. Learning roi transformer for oriented object detection in aerial images. In Proceedings of the IEEE Conference on Computer Vision and PatternRecognition, pages 2849–2858, 2019.".

[0059] 2. Network training

[0060] The training images are input into the network for training, and the network is trained in an end-to-end manner to obtain a trained network.

[0061] The loss function L of the network is set during training as follows:

[0062] L=L 1st +L 2nd (10)

[0063] Among them, L 1st is the loss of the directed candidate box generation network, L 2nd The loss of the detection head for oriented targets;

[0064] The loss L of the directed candidate box generation network 1st The calculation formula is:

[0065]

[0066] Where N1 represents the total number of positive and negative samples in a training batch of the directed candidate box generation network. In this embodiment, the value is 256. P1 is the number of positive samples, i represents the index of the sample, p i The score of the i-th sample output by the directed candidate box generation network belonging to the foreground, p i ′ is the true value of the i-th sample. If the i-th sample is a positive sample, then p i ′ is equal to 1. If the i-th sample is a negative sample, then p i ′ is equal to 0; δ i is the regression offset of the i-th preset box, t i ′ is the true value of the regression offset of the i-th preset box; the positive sample refers to a preset box whose intersection-and-union ratio with any external rectangular box is greater than 0.7, or a preset box with the maximum intersection-and-union ratio with the external rectangular box and the intersection-and-union ratio value is greater than 0.3; the negative sample refers to a preset box whose intersection-and-union ratio with the external rectangular box is less than 0.3; the external rectangular box refers to the horizontal external rectangle of the target directed annotation box in the training DOTA1-0 dataset.

[0067] Cross entropy loss function L ce (·) is defined as:

[0068] L ce =-[p′·log(p)+(1-p′)·log(1-p)] (12)

[0069] Among them, p and p′ are two input variables of the function.

[0070] L reg (·) is the Smooth L1 loss function, defined as:

[0071]

[0072] Among them, x is the input variable of the Smooth L1 loss function.

[0073] The loss L of the oriented object detection head 2nd The calculation formula is:

[0074]

[0075] Where N2 represents the total number of positive and negative samples in a training batch of the directional detection head. In this embodiment, the value is 512. P2 is the number of positive samples; u represents the category of the i-th sample, u=0,1,2,...,C; q i is the score vector of the i-th sample belonging to each class, output by the directed object detection head, qi ′ is the true value vector of the score of the i-th sample belonging to each class; when the i-th sample is a negative sample, the value of [u≥1] is 0, and when the i-th sample is a positive sample, the value of [u≥1] is 1; represents the regression offset true value of the i-th sample with respect to category u, σ i (u) is the predicted offset of the i-th sample with respect to category u; the positive sample mentioned here refers to a directed rectangular candidate box whose intersection-with-union ratio with the target directed labeled box is greater than 0.5, and the negative sample refers to a directed rectangular candidate box whose intersection-with-union ratio with the target directed labeled box is less than 0.5.

[0076] 3. Directed Target Detection Using Midpoint Offset Expression

[0077] The image to be detected is input into the trained directed candidate box generation network described above, and the position and foreground score of the directed rectangular candidate box are output. Then, the directed rectangular candidate box is processed by the non-maximum suppression method, and the intersection-over-union ratio threshold of the non-maximum suppression is set to 0.8. The first 1000 directed rectangular candidate boxes with the largest scores after processing are selected and input into the trained directed object detection head, and the score vector q = {q0, q1, L, q C} and the regression offset corresponding to each class The classification scores are normalized as follows to obtain the confidence level for each category:

[0078]

[0079] Among them, c Represents the confidence that the directed rectangular candidate box belongs to the cth category, c = 0, 1, 2, ..., C;

[0080] The class corresponding to the maximum confidence value is taken as the classification result of the directed rectangular candidate box, and its regression offset is decoded as follows to obtain the final directed detection box:

[0081]

[0082] in, is the center point coordinate of the oriented detection box, and The width and height of the detection box are respectively, is the angle between the long side of the directed detection box and the horizontal axis; (x0, y0) is the midpoint coordinate of the directed rectangular candidate box output by the trained directed candidate box generation network, w0 and h0 are the width and height of the directed rectangular candidate box output by the trained directed candidate box generation network, respectively, and θ is the angle between the long side of the directed rectangular candidate box output by the trained directed candidate box generation network and the horizontal axis.

[0083] The polygon non-maximum suppression method is used to process the final directional detection box of each class. The intersection-over-union ratio threshold of polygon non-maximum suppression is set to 0.5, and the processed directional detection box and its corresponding category are used as the final output detection result.

[0084] Since the input image is obtained by cropping the original image in DOTA1-0, the detection result cannot be effectively evaluated. The detection results of the cropped image blocks are merged. Since there are overlaps between adjacent image blocks, overlapping detection frames will be introduced after the image block detection results are merged. After the merging operation, polygon non-maximum suppression is used to post-process the merged results and output the detection results of the original image. The detection results of the original image are submitted to the DOTA1-0 test server for evaluation. The results of directed target detection using the method of the present invention are shown in Figure 2. Figure 3 .

[0085] mAP (mean Average Precision) is used to evaluate the effectiveness of the method of the present invention, and its definition is as follows:

[0086]

[0087] Where C represents the total number of categories contained in the training data set. In this example, C = 15, c represents the target category index, and AP c Represents the detection accuracy of different categories. The method of the present invention is compared with the oriented target detection algorithm based on Rotated RPN and RoI Transformer, and the backbone network used for the comparison is ResNet50. The comparison results are shown in Table 1. It can be seen that the present invention can achieve higher detection accuracy.

[0088] Table 1

[0089] method mAP Rotated RPN 60.01％ RoI Transformer 74.61％ Method of the present invention 75.87％

Claims

1. A directed target detection method using midpoint offset expression, characterized in that Here are the steps: Step 1, construct a directed target detection network expressed by midpoint offset: it mainly includes three parts: backbone network, directed candidate box generation network and directed target detection head. Among them, the backbone network adopts ResNet50 residual network, the directed candidate box generation network contains a fully convolutional neural network composed of 1 3×3 convolutional layer and two parallel 1×1 convolutional layers, one 1×1 convolutional layer is used for classification, and the other 1×1 convolutional layer is used for regression. The directed target detection head contains a fully connected network composed of 2 fully connected layers with 1024 dimensions and 2 parallel fully connected layers of different dimensions. One hidden layer dimension of the parallel fully connected layer is C+1, which is used for classification, and the other hidden layer dimension is 5×(C+1), which is used for regression. C is the number of target detection categories contained in the image dataset; Input any image, get the feature map of the image through the backbone network, pass the feature map through the directed candidate box generation network, output the directed candidate box of the target, pass the feature map and the directed candidate box of the target through the directed target detection head, output the position and category of the directed candidate box of the target, the specific processing process is as follows: Step a: Input image I passes through the ResNet50 residual network to obtain four levels of feature maps P2, P3, P4, and P5, where P i Represents the feature map of the i-th level, i = 2, 3, 4, 5; Step b: The feature maps P2, P3, P4, and P5 are input into the directed candidate box generation network. First, the feature map P5 is downsampled by 2 times to obtain the feature map P6. Then, three preset boxes with aspect ratios of 1:1, 1:2, and 2:1 are placed on each feature point of the feature maps P2, P3, P4, P5, and P6. Each feature map is then passed through a full convolutional network to obtain the score and regression offset of the preset box belonging to the foreground. The regression offset includes the preset box parameters and the horizontal prediction box parameters, which are recorded as δ = (δ x ,δ y ,δ w ,δ h ,δ α ,δ β ), where (δ x ,δ y ) is the offset of the center point coordinates of the preset frame, δ w and δ h Respectively represent the width and height offset of the preset box, δ α and δ β Respectively represent the offset of the top edge and the right midpoint of the horizontal prediction box; According to the regression offset, decode as follows to get other parameters of the horizontal prediction box: Where (x, y) is the coordinate of the midpoint of the horizontal prediction frame, w and h are the width and height of the horizontal prediction frame respectively, Δα is the horizontal offset of the midpoint of the top edge of the horizontal prediction frame, and Δβ is the vertical offset of the midpoint of the right edge of the horizontal prediction frame; (a x ,a y ) is the center point coordinate of the preset frame, a w and a h Respectively represent the width and height of the preset box; The horizontal prediction box is decoded according to the midpoint offset expression formula to obtain the coordinate vectors v = {v1, v2, v3, v4} of the four vertices of the directed parallelogram candidate box: Among them, v1 represents the upper left vertex of the directed parallelogram candidate box, v2 represents the upper right vertex of the directed parallelogram candidate box, v3 represents the lower right vertex of the directed parallelogram candidate box, and v4 represents the lower left vertex of the directed parallelogram candidate box; Extend the length of the short diagonal of the directed parallelogram candidate box to make its short diagonal and long diagonal the same length, and get the directed rectangular candidate box (x o ,y o ,w o ,h o ,θ), where (x0, y0) is the coordinate of the midpoint of the directed rectangular candidate box, w0 and h0 are the width and height of the directed rectangular candidate box, respectively, and θ∈[-90°, 90°] is the angle between the long side of the directed rectangular candidate box and the horizontal axis; Step c: Input the directed rectangular candidate box and feature maps P2, P3, P4, and P5 into the directed detection head. First, project the directed rectangular candidate box onto the feature map P k superior, k0 is equal to 4; then, the directed rectangular candidate box projected onto the feature map is subjected to a directed RoIAlign operation to obtain the RoI feature of the directed rectangular candidate box; the RoI feature is passed through a fully connected network to obtain the score vector q = {q0, q1, L, q C } and its regression offset vector c=0,1,2,…,C, q0 represents the score of the directed rectangular candidate belonging to the background, q c Indicates the score of the directed rectangular candidate box belonging to the cth category; is the center point coordinate offset of the directed rectangular candidate box corresponding to the cth category, is the width offset of the directed rectangular candidate box corresponding to the cth class, is the offset of the height of the directed rectangular candidate box corresponding to the cth class, is the angle offset of the directed rectangular candidate box corresponding to the cth category; Step 2, network training: Input the training image dataset into the directed object detection network expressed by the midpoint offset constructed in step 1, train the network in an end-to-end manner, and obtain a trained network; during the training, the network loss function L is set as follows: L=L 1st +L 2nd (3) Among them, L 1st is the loss of the directed candidate box generation network, L 2nd The loss of the detection head for oriented targets; The loss L of the directed candidate box generation network 1st The calculation formula is: Among them, N1 represents the total number of positive and negative samples in a training batch of the directed candidate box generation network, and N P1 is the number of positive samples, i represents the index of the sample, L ce (·) is the cross entropy loss function, L reg (·) is the Smooth L1 loss function, p i The score of the i-th sample output by the directed candidate box generation network belonging to the foreground, p′ i is the true value of the i-th sample. If the i-th sample is a positive sample, then p′ i is equal to 1. If the i-th sample is a negative sample, then p′ i =0; i is the regression offset of the i-th preset box, t′ i is the true value of the regression offset of the i-th preset box; the positive sample refers to a preset box whose intersection-and-union ratio with any external rectangular box is greater than 0.7, or a preset box with the maximum intersection-and-union ratio with the external rectangular box and whose intersection-and-union ratio is greater than 0.3; the negative sample refers to a preset box whose intersection-and-union ratio with the external rectangular box is less than 0.3; the external rectangular box refers to the horizontal external rectangle of the target directional annotation box in the image data set; The loss L of the oriented object detection head 2nd The calculation formula is: Among them, N2 represents the total number of positive and negative samples in a training batch of the directional detection head, N P2 is the number of positive samples; u represents the category of the i-th sample, u=0,1,2,...,C; q i is the score vector of the i-th sample belonging to each class, output by the directed object detection head, q i ′ is the true value vector of the score of the i-th sample belonging to each class; when the i-th sample is a negative sample, the value of [u≥1] is 0, and when the i-th sample is a positive sample, the value of [u≥1] is 1; represents the regression offset true value of the i-th sample with respect to category u, σ i (u) is the prediction offset of the i-th sample with respect to category u; the positive sample mentioned here refers to a directed rectangular candidate box whose intersection-with-unit ratio with the target directed labeled box is greater than 0.5, and the negative sample refers to a directed rectangular candidate box whose intersection-with-unit ratio with the target directed labeled box is less than 0.5; Step 3, directed target detection expressed by midpoint offset: Input the image to be detected into the directed candidate box generation network trained in step 2, output the position and foreground score of the directed rectangular candidate box, then use the non-maximum suppression method to process the directed rectangular candidate box, set the intersection-over-union ratio threshold of non-maximum suppression to 0.8, select the first 1000 directed rectangular candidate boxes with the largest scores after processing and input them into the trained directed target detection head, and output the score vector q = {q0, q1, L, q C } and the regression offset corresponding to each class The classification scores are normalized as follows to obtain the confidence level for each category: Among them, c Represents the confidence that the directed rectangular candidate box belongs to the cth category, c = 0, 1, 2, ..., C; The class corresponding to the maximum confidence value is taken as the classification result of the directed rectangular candidate box, and its regression offset is decoded as follows to obtain the final directed detection box: in, is the center point coordinate of the oriented detection box, and The width and height of the detection box are respectively, is the angle between the long side of the directed detection box and the horizontal axis; (x0, y0) is the coordinate of the midpoint of the directed rectangular candidate box, w0 and h0 are the width and height of the directed rectangular candidate box, respectively, and θ is the angle between the long side of the directed rectangular candidate box and the horizontal axis; The polygon non-maximum suppression method is used to process the final directional detection box of each class. The intersection-over-union ratio threshold of polygon non-maximum suppression is set to 0.5, and the processed directional detection box and its corresponding category are used as the final output detection result.

Images