A multi-factor fusion learning strategy for rotating small target detection

Abstract

To address the boundary issues that arise when optimizing the target angle during the learning process of a rotating small target detection model, this invention proposes a multi-factor fusion of Rotated IoU (MFRIoU) learning strategy. This strategy provides the network with the correct optimization direction by comprehensively considering factors such as overlap rate, center point distance, bounding box shape, and bounding box area. The shape penalty term in this invention accelerates the fitting of the predicted box shape to the ground truth box shape, while the bounding box area penalty term reduces the loss caused by the information difference corresponding to the area difference. This results in a rotating small target detection model with higher accuracy, faster speed, and lower error.

Description

Technical fields:

[0001] This invention relates to a learning strategy for rotating small target detection models. Existing technology:

[0002] With the development of drone technology, targets in aerial images often have arbitrary rotation angles, are small in scale, and are densely distributed. However, during the learning process of rotating small target detection models, boundary problems arise when optimizing the target angle, resulting in large loss values ​​and making it impossible to optimize model parameters according to the most ideal prediction results. The root cause of these boundary problems is the inconsistency between the objective function and the IoU (Intersection over Union) based evaluation metric. Therefore, rotational IoU (RIoU) loss can be used to learn the target detection model.

[0003] Current detection models (such as deep learning networks) employ learning strategies that approximate the RIoU. Reference 1, "Yang X, Yang J, Yan J, et al. Towards more robust detection for small, cluttered and rotated objects[C]. IEEE International Conference on Computer Vision. 2019, 27: 8232-8241," utilizes the Smooth IoU (SIoU) loss. This loss approximates the RIoU loss by combining the direction of gradient propagation of the Smooth L1 loss with the absolute value of the IoU, and then using the Smooth L1 loss to backpropagate the gradient of the IoU. The pixel cross-union ratio (PIoU) loss in reference 2, “Chen Z, Chen K, Lin W, et al. Piouloss: Towards accurate-oriented object detection in complex environments[C]. European Conference on Computer Vision. Springer, 2020:195-211.”, approximates the size of RIoU by using the number of shared and unique pixels between two rotated boxes.The Gaussian Wasserstein Distance (GWD) loss in reference 3, “Yang X, Yan J, Ming Q, et al. Rethinking rotated object detection with gaussian wasserstein distance loss[C]. International Conference on Machine Learning. 2021: 11830-11841,” and the Kullback-Leibler Divergence (KLD) loss in reference 4, “Yang X, Yang J, et al. Learning high-precision bounding box for rotated object detection via kullback-leibler divergence[C]. Advances in Neural Information Processing Systems, 2021, 34,” both convert any rotated box (x, y, w, h, θ) into a two-dimensional Gaussian distribution, and then perform different transformations on the two Gaussian distributions (calculating the GW distance; calculating the KL divergence) to approximate the function of RIoU.

[0004] Purpose of the invention:

[0005] Current deep object detection models do not fully consider the rotational geometry of the target object during learning. This invention proposes a multi-factor fusion of RotatedIoU (MFRIoU) learning strategy based on RIoU. Taking into account factors such as overlap rate, center point distance, bounding box shape, and bounding box area, the RotatedRetinaNet model is used as the feature extraction backbone network. The designed learning strategy is tested on the public test set DOTAv1, achieving a MAP50 score of 70.42 and a MAP75 score of 46.14, outperforming references 1-4. Summary of the Invention:

[0006] The learning strategy proposed in this invention involves designing a multi-factor fusion loss function, MFRioU, which is calculated based on the complete Rotated Intersection over Union (CIoU) loss. It incorporates rotation angle loss into the shape loss to address the boundary issues of angles and adds an area penalty term to optimize the information difference between different area differences under the same RIoU condition. The MFRioU loss integrates overlap rate, center point distance, bounding box shape, and bounding box area factors. The overlap rate penalty term aligns the objective function with the evaluation metric; the center point penalty term provides the correct optimization direction for the model when the predicted and ground truth boxes do not intersect; the shape penalty term accelerates the fitting of the predicted and ground truth box shapes in the later stages of training; and the bounding box area penalty term is the loss caused by matching the information difference corresponding to the area difference.

[0007] (1) Calculation of CIoU

[0008] For a complete calculation of the rotational intersection-to-union ratio loss, four factors should be considered simultaneously: overlap rate, center point distance, shape parameters, and rotation angle.

[0009] 1) Overlap rate penalty

[0010] L overlap Its main function is to align the objective function and the evaluation metric, where the accuracy evaluation metric mAP is calculated based on the RIoU between the two bounding boxes, and L overalp Also based on RIOU, the specific formula is as follows:

[0011] L overlap =L RIoU =1-RIoU (1)

[0012] Where RIoU represents the area crossover ratio between the predicted and actual rotated boxes.

[0013] 2) Center point distance penalty

[0014] L center Its main function is to provide the correct optimization direction for the depth model when the predicted bounding box and the ground truth bounding box do not intersect. The specific formula is as follows:

[0015]

[0016] Where ρ 2 (b,b gt ) represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c represents the diagonal length of the smallest circumscribed rotating rectangle that can simultaneously cover both boxes.

[0017] 3) Shape parameter penalty term

[0018] Lshape Its main function is to accelerate the fitting of the predicted bounding box shape and the true bounding box shape in the later stages of training. The specific formula is as follows:

[0019]

[0020] Where α represents a dynamic, overlap-based weight balancing factor that adjusts the sensitivity of the shape parameters; v represents a shape parameter penalty term used to measure the shape similarity between the predicted and ground truth boxes; w gt h gt w and h represent the long and short sides of the ground truth bounding box; w and h represent the long and short sides of the predicted bounding box.

[0021] 4) Rotation angle penalty item

[0022] L θ Its main function is to accelerate the reduction of the angle deviation between the fitted predicted box and the ground truth box at the end of the training period. The specific formula is as follows:

[0023] L θ =|θ-θ gt | (4)

[0024] Where θ and θ gt These are the angles of the predicted bounding box and the ground truth bounding box, respectively.

[0025] In summary, the complete calculation of the rotational crossover ratio loss is as follows:

[0026]

[0027] (2) Calculation of MFRIOU

[0028] A complete rotational intersection-union ratio (CIoU) loss is constructed using the relevant geometric factors of the rotated frame (x, y, w, h, θ), but its penalty term still has shortcomings. This invention proposes the MFRioU loss to improve it by calculating the implicitly included geometric parameters in (x, y, w, h, θ).

[0029] 1) Rotation angle penalty item

[0030] In Equation (5), the CRIoU loss optimizes the rotation angle separately, which leads to boundary issues in the model. Therefore, the rotation angle penalty term cannot be used to optimize the angle difference between the predicted box and the ground truth box, allowing L... θ It equals 0. Based on RIOU, a metric for measuring the overall regression performance, L can be used. overlap To implicitly optimize the angle.

[0031] 2) Shape parameter penalty term

[0032] In formula (3), L shapeIt is optimized based on aspect ratio, but this optimization method can only reflect the difference in aspect ratio, and cannot truly reflect w and w'. gt h and h gt The true relationship between the predicted bounding box and the ground truth bounding box. When the aspect ratios of the predicted bounding box and the ground truth bounding box are similar, L... shape This will slow down the model's further optimization of length and width. And when {(w=kw) gt h = kh gt )|k∈R - When v = 0, L shape =0, which means the shape parameter penalty term has no effect. Furthermore, it can be seen from formula (6) for the derivative of v with respect to w and h. and The signs are opposite, meaning that the long and short sides of the predicted box cannot increase or decrease at the same time during the model optimization process. For example, when the model increases w to fit the predicted box and the true box, h will decrease. However, an excessively small h will in turn constrain the model optimization. This contradictory fitting method will make the later optimization of the model very difficult.

[0033]

[0034] The shape parameters are optimized using the following formula:

[0035]

[0036] Where β is inherited from α, allowing the model to have different optimization priorities at different training epochs; u represents the bounding box shape parameter penalty factor, used to measure the difference in shape parameters between the predicted box and the ground truth box; a is a constant 1 / 2, which balances the value of u to prevent excessively large loss values ​​from causing training instability; w gt h gt w and h represent the long and short sides of the ground truth bounding box; w and h represent the long and short sides of the predicted bounding box; w c h c This represents the long and short sides of the smallest bounding rotated rectangle that can simultaneously cover both the predicted and ground truth bounding boxes.

[0037] L shape The method of directly optimizing the long and short side parameters using Euclidean distance is simple, efficient, and highly constraining, and it also overcomes the limitations of L... shape Contradictions in the parameters of long and short sides and {(w=kw) gt h = kh gt )|k∈R - The drawback of failure at that time.

[0038] 3) Area parameter penalty item

[0039] Improving the angle and shape parameter penalties for CRIoU can significantly improve model optimization, but a drawback remains: it is insensitive to area changes while maintaining the same CRIoU loss. When area differences are similar, the larger prediction box containing the most information should be selected; however, when area differences are large, the smaller prediction box should be chosen, as excessively large areas cannot be determined to contain only one target and may contain a lot of useless information. Therefore, although different area differences result in the same CRIoU loss, an area parameter penalty term L still needs to be added. area .

[0040] Area penalty item L area Based on the area difference, it should increase slowly when the area difference is small, increase rapidly when the area difference is large, and eventually stabilize. The Sigmoid function not only conforms to the above trend, but also has a value range of [0, 1] and does not have a significant impact on the value of the original loss function, making it very suitable as the basic function for area differences. Since it is necessary to distinguish between positive and negative area differences, the Sigmoid function needs to be appropriately transformed.

[0041] Taking the complete Sigmoid(x-2) curve within the domain [0, 4] as the calculation for positive area differences, then the Sigmoid(-x-2) curve represents the change in negative area differences. Since the trend of change in negative area differences should be greater than that of change in positive area differences—that is, when the area change is small—it should tend towards a larger prediction frame—Sigmoid(-2x-2) can be used to represent the difference in the trends between positive and negative area differences. In summary, Sigmoid(-2x-2) is used for negative area differences, and Sigmoid(x-2) is used for positive area differences.

[0042] In summary, the final area parameter penalty term L area The calculation formula is designed as follows:

[0043] L area =(1-RIoU)×s (8)

[0044]

[0045]

[0046]

[0047] Where 's' is the area difference adjustment coefficient multiplied by (1-RIoU), it retains the original physical meaning and achieves appropriate fine-tuning; it has no effect when the two boxes do not intersect; 'k' represents different coefficients for positive and negative area differences; 'DoA' represents the normalized area difference; and 'area' and 'area' represent the area difference coefficients. gtThese represent the predicted box area and the actual box area, respectively.

[0048] The complete formula for MFROIoU loss is as follows:

[0049]

[0050]

[0051] Beneficial effects:

[0052] Rotated RetinaNet was used as the feature extraction backbone network to verify the effectiveness of the MFRIOU learning strategy of this invention on the DOTAv1 dataset for detecting small rotated targets. Detailed comparison of experimental data (as shown in Table 1) demonstrates that the overlap rate penalty, center point penalty, improved shape parameter penalty, and area penalty terms of the MFRIOU loss are all effective in improving the accuracy of rotating targets.

[0053] Table 1 Comparison of OBB performance of Rotated RetinaNet using different RIOU losses on DOTAv1

[0054] loss function <![CDATA[mAP 50 ]]> <![CDATA[mAP 75 <!-- 4 -->]]> <![CDATA[L L1 ]]> 68.00 43.37 <![CDATA[L RIoU ]]> 68.61 43.41 <![CDATA[L RIoU +L center ]]> 69.47 44.48 <![CDATA[L RIoU +L center +L shape ′]]> 70.05 45.53 <![CDATA[L RIoU +L center +L shape ′+L area ]]> 70.42 46.14

[0055] Currently, rotation box detection algorithms based on regression loss mainly include Smooth IoU Loss (reference 1), PIoU Loss (reference 2), GWD Loss (reference 3), and KLD Loss (reference 4). To analyze the performance of MFRioU loss compared to other rotation box detection algorithms based on regression loss functions, all methods were implemented using the Rotated RetinaNet network architecture. Table 2 shows that the MFRioU loss performs well in terms of mAP... 50 Indicators and mAP 75 All indicators are optimal.

[0056] Table 2 Comparison of OBB performance of Rotated RetinaNet on DOTAv1 using different regression losses.

[0057] loss function <![CDATA[mAP 50 ]]> <![CDATA[mAP 75 ]]> <![CDATA[L Smooth IoU (Method from Reference 1) 68.44 43.76 <![CDATA[L PIoU (Reference 2 Method) 68.66 43.84 <![CDATA[L GWD (Reference 3 Method) 69.87 44.93 <![CDATA[L KLD (Reference 4 Method) 70.12 46.01 <![CDATA[L ARIoU (Method of the present invention) 70.42 46.14

[0058] The RetinaNet backbone is ResNet-50, which has strong universality but lacks outstanding feature extraction capabilities. The Swintransformer Tiny (Swin-T) backbone is used, whose self-attention method has strong feature extraction capabilities. Multi-scale cropping (MS) can be applied to the dataset: images in the DOTA dataset are cropped to (0.5, 1.0, 1.5) times the size of 1024; random transformations (RT) such as rotation, flipping, and cropping are used to enhance the training images. As shown in Table 3, the MFRioU of this invention also exhibits high accuracy.

[0059] Table 3 Performance of MFRIOU loss under different accuracy enhancement methods

[0060] Attached image description:

[0061] Figure 1 Flowchart of this invention. Detailed implementation method:

[0062] The process of target detection for rotating small targets is as follows: Figure 1 As shown, the DOTAv1 dataset is first processed by cropping the image data to 1024×1024. Next, the processed images are input into the detection network, which uses RetinaNet. The backbone is a ResNet-50 for feature extraction; the neck uses an FPN network to fuse and enhance the extracted features, improving the network's multi-scale expressive power; the head consists of multiple convolutions, outputting multiple tensors for classification and regression prediction. After Non-Maximum Suppression (NMS), the classification and regression prediction parameters of the predicted bounding boxes are obtained. Finally, the parameters of the predicted and ground truth bounding boxes are compared using MFRioU regression loss and Focal Loss classification loss to determine the weight update direction. The loss parameters are then backpropagated for weight updates until the best model is trained. Choosing an appropriate loss function can speed up training and improve detection accuracy.

[0063] The calculation of the prediction box regression loss MFRioU in this invention is performed as follows:

[0064] 1) Overlap rate penalty

[0065] Calculate L using formula (13) overlap Align the objective function and evaluation metrics to optimize the area intersection-union ratio between the predicted and actual rotated boxes.

[0066] L overlap =L RIOU =1-RIoU (13)

[0067] 2) Center point distance penalty

[0068] Calculate L using formula (14) center The center point distance penalty term provides the network with the correct optimization direction when the predicted box and the ground truth box do not intersect.

[0069]

[0070] 3) Shape parameter penalty term

[0071] Calculate L using formula (15) shape The shape parameter penalty term accelerates the fitting of the predicted box shape to the true box shape in the later stages of training.

[0072]

[0073] 4) Area parameter penalty item

[0074] Calculate L using formula (16) area This yields the optimization direction of the image prediction box for the area.

[0075]

[0076] The overlap rate, center point distance, shape, and area penalty terms were calculated separately above. These four penalty terms were then added together as a unified loss, and finally backpropagated to update the weights, thereby training a more accurate rotating object detection network model.

Claims

1. A method for detecting rotating small targets in aerial images, characterized in that: The learning strategies of this method include shape parameter penalty terms, area parameter penalty terms, and the MFRioU learning strategy; (1) Shape parameter penalty The main function of the shape parameter penalty term is to accelerate the fitting of the predicted bounding box shape to the ground truth bounding box shape in the later stages of training. The shape parameter penalty term in the early stages is only based on aspect ratio optimization and cannot reflect w and w'. gt h and h gt The true relationship between the long and short sides is not established, and the contradiction that the long and short sides cannot increase or decrease simultaneously during network optimization makes later optimization very difficult. Therefore, a new shape parameter penalty term is proposed, using Euclidean distance to directly optimize the long and short side parameters. The specific formula is as follows: Where β allows the network to have different optimization priorities at different training stages; u represents the bounding box shape parameter penalty factor, which is used to measure the difference of each shape parameter between the predicted box and the ground truth box; a is a constant 1 / 2, used to balance the value of u and prevent excessively large loss values ​​from causing training instability; w gt h gt w and h represent the long and short sides of the ground truth bounding box; w and h represent the long and short sides of the predicted bounding box; w c h c This represents the long and short sides of the smallest bounding rotated rectangle that can simultaneously cover both the predicted and ground truth bounding boxes. (2) Area parameter penalty When only the overlap rate penalty, center point distance penalty, and shape parameter penalty are considered, different area differences will have the same rotation intersection-overlap ratio loss. However, different area differences should include different optimization strategies. When the area differences are similar, the larger prediction box that contains as much information as possible should be selected. When the area differences are large, the smaller prediction box should be selected because if the area is too large, it is impossible to determine whether it contains only one target and may contain a lot of useless information. Propose area penalty term L area The area parameter penalty term is based on the area difference. It grows slowly when the area difference is small, grows rapidly when the area difference is large, and eventually tends to stabilize. The Sigmoid function not only conforms to the above trend, but also has a value range between [0,1] and does not have a significant impact on the value of the original loss function. Therefore, it is chosen as the basic function of the area difference. Since it is necessary to distinguish between positive and negative area differences, the Sigmoid function is then appropriately transformed. Taking the complete Sigmoid(x-2) curve within the domain [0,4] as the calculation for positive area difference, then the Sigmoid(-x-2) curve represents the change in negative area difference. Since the trend of negative area difference should be slightly more volatile than that of positive area difference—that is, when the area change is small—it should tend towards a larger prediction frame, Sigmoid(-2x-2) can be used to represent the difference in the trend between positive and negative area differences. Sigmoid(-2x-2) is used for negative area differences, and Sigmoid(x-2) is used for positive area differences; the area parameter penalty term L... area The specific formula is as follows: L area <(1-RIoU)×s (2) Where 's' is the area difference adjustment coefficient multiplied by (1-RIoU), it retains the original physical meaning and achieves appropriate fine-tuning; it has no effect when the two boxes do not intersect; 'k' represents different coefficients for positive and negative area differences; 'DoA' represents the normalized area difference; and 'area' and 'area' represent the area difference coefficients. gt These represent the predicted bounding box area and the actual bounding box area, respectively. (3) MFRIOU learning strategy The designed multi-factor fusion loss function MFRioU consists of four parts: overlap rate penalty term, center point distance penalty term, shape parameter penalty term, and rotation angle penalty term; the overlap rate penalty term loss term is as follows: L overlap =L RIOU =1-RIoU (6) where RIoU represents the area crossover ratio between the predicted and actual rotated boxes; Center point distance penalty: Where ρ 2 (b,b gt ) represents the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box; c represents the diagonal length of the smallest bounding rectangle that can simultaneously cover both boxes. The complete formula for the final loss function MFROIoU loss is as follows: The parameters of the model are learned using the loss function in formula (8) and the backpropagation algorithm.

Images