A method for rotating object detection in remote sensing images based on anchor-free frames under attention mechanism

Abstract

This invention discloses a method for detecting rotated objects in remote sensing images based on an attention mechanism and anchor-free rotation. This method belongs to the field of remote sensing image object detection technology. The method first divides the acquired remote sensing image set into training and test sets. It then improves the Faster RCNN remote sensing image detection network model, incorporating an attention mechanism to enhance the network's feature extraction capabilities and adding a masked attention module after the neck network to enhance feature representation. It also designs an AFOrpn network based on the FCOS anchor-free detection head network concept to extract high-quality proposals. Next, the model extracts a multi-layer feature map from the input image through the backbone and neck networks. This is then input into the SMAM module, where the attention mechanism generates a fused feature map. The fused feature map is then added to the original feature map through a scaling operation to improve the original feature representation. The fused feature map is then input into the AFOrpn network for proposal screening. The proposals are then passed through the RCNN detection head network to output the final detection results. Model parameters are then updated based on a designed loss function, completing model training. Finally, the trained object detection model is used for object detection in remote sensing images.

Description

Technical Field

[0001] The present invention relates to a method for remote sensing image target detection technology, and in particular to a remote sensing image rotation target detection method based on an anchor-free frame under an attention mechanism. Background Art

[0002] The detection task aims to determine the location and category information of a specified target in an image and surround the target with a detection box. With the development of remote sensing technology in recent years, remote sensing target detection has received widespread attention. By efficiently identifying and locating specific targets on the surface, such as buildings, vehicles, or farmland, remote sensing target detection provides support for a comprehensive understanding of large areas. This not only helps with environmental monitoring and resource management, but also plays a key role in military and emergency rescue. However, due to the special perspective and wide field of view, remote sensing images have diverse target categories, diverse target directions, and large variations in target scales, which greatly affects performance indicators such as target detection accuracy and efficiency.

[0003] Benefiting from the rise of convolutional neural networks (CNNs), which provide powerful tools for object detection, many researchers have proposed many innovative methods and techniques and applied them to remote sensing imagery. However, despite the proposal of a variety of effective methods to deal with issues such as scale variations and arbitrary orientations of remote sensing objects, most existing orientation detection methods are built based on anchor boxes, where thousands of anchor boxes are predefined for subsequent recognition and regression. This leads to problems such as imbalanced positive and negative samples due to the anchor boxes, a large number of additional hyperparameters, and a high computational load of model post-processing operations. Even for small objects, serious under-detection may occur.

[0004] Based on the above problems, the trend of using anchor-free directional target detectors is gradually increasing. However, the direct application of anchor-free directional frameworks may face some challenges when dealing with complex target distributions in remote sensing images. First, directly using key points to locate objects may be affected by complex backgrounds and blurred areas, resulting in the inclusion of non-negligible background interference in the features extracted by the feature extraction network, which ultimately leads to certain deviations in the detection head when predicting results based on the feature map. Secondly, due to the sensitivity of large aspect ratio objects to angles and the feature misalignment problem of the full convolutional network, it is difficult for anchor-free directional target detectors to generate accurate prediction boxes for large aspect ratio objects. In addition, since classification and regression are two branches of the model, they are independently optimized through their own loss functions during training, and there is a problem of insufficient correlation, which leads to inconsistency between classification confidence and positioning regression effects. Summary of the Invention

[0005] Purpose of the invention: In order to solve the above problems, the present invention provides a method for detecting rotated targets in remote sensing images based on an anchor-free frame under an attention mechanism.

[0006] Technical solution: A method for detecting rotated objects in remote sensing images based on an attention mechanism and without anchor boxes, including the following steps:

[0007] S1. Obtain a remote sensing image set and divide it into a training set and a test set;

[0008] S2. Build a remote sensing image detection network model, which is based on the improvement of Faster RCNN and includes a fusion attention mechanism to improve the feature extraction ability of the network. The mask attention module is added after the neck network to enhance the feature expression ability.

[0009] S3. Design the target detection AFOrpn network. The feature map obtained by the remote sensing image detection network model is used as the input of the AFOrpn network. The AFOrpn network is used to detect the category of each pixel in the feature map and perform regression processing on the detection box. The detection box is filtered through classification confidence and non-maximum suppression operations to obtain the proposed box.

[0010] The proposed box is extracted through the RotatedRoiAlign pooling layer and input into the subsequent fully connected layer, and finally classified through the softmax layer and the regression loss layer to refine the proposed box;

[0011] S4. Design loss functions, including AFOrpn network loss, RCNN loss, and supervised mask loss;

[0012] The AFOrpn network loss consists of classification loss and regression loss, and the regression loss of the AFOrpn network is split into distance loss and center offset loss;

[0013] RCNN loss is composed of classification loss and regression loss, and uses the original loss of Faster RCNN remote sensing image detection network;

[0014] S5. Update the parameters of the remote sensing image rotation target detection model to complete the model training, and then use the trained target detection model to perform remote sensing image target detection.

[0015] Furthermore, the remote sensing image detection network model includes a backbone network, a neck network, SMAM, AFOrpn and an RCNN detection head network, wherein the RCNN detection head network includes a RotatedRoiAlign pooling layer, a fully connected layer with a Relu activation function, a Softmax classifier and a regression loss layer;

[0016] The backbone network of the remote sensing image detection network model is a resnet50 structure, and the neck network adopts an FPN structure. The input image is extracted into a multi-layer feature map through the backbone network and the neck network, and input into a fused feature map obtained by the attention mechanism. The fused features are added to the original features through a scaling operation to improve the expression ability of the original features, and are input into AFOrpn for proposal screening. The design of AFOrpn is based on the anchor-free box detection head network implementation of FCOS. Finally, the proposals are output as the final detection results through the RCNN network.

[0017] In step S2 of the above method, the remote sensing image detection network model is implemented based on Faster RCNN with added supervised mask attention and anchor-free frame design;

[0018] The supervised mask attention adopts the idea of ​​channel attention in the feature fusion stage of SMAM to adaptively adjust the feature channels and the weights of deep and shallow features, thereby realizing the comprehensive utilization of several features.

[0019] SMAM consists of three stages: feature fusion, spatial attention, and supervised mask enhancement. The feature fusion unifies feature maps of different resolutions to the same size through upsampling operations. Feature maps from different levels are spliced ​​and the weights of feature channels and deep and shallow features are adaptively adjusted using the idea of ​​channel attention to add and fuse them, so as to comprehensively utilize multi-level feature information. Spatial attention uses the idea of ​​self-attention to adaptively integrate local features and global dependencies to eliminate the aliasing effect caused by fusion. The supervised mask enhancement stage uses a pseudo mask to guide the network to retain image features and filter out noise information.

[0020] The anchor frame design is based on the anchor-based RPN network, which generates anchors of different sizes and aspect ratios for each pixel on the feature map. The anchor-free design directly regresses the detection frame through points on the feature map without presetting anchor frames of multiple sizes and ratios. The detection frame is predicted by regressing the offset of the preset frame. At the same time, the positional relationship between the feature point and the object is used as the criterion for dividing positive and negative samples during the training process. In this invention, the anchor-free method realizes the construction of the RPN network based on the FCOS algorithm.

[0021] Specifically, supervised mask attention involves the following computation:

[0022] (1) Upsampling

[0023] For the input low-resolution image, a sub-pixel convolution operation is performed to obtain a feature image with the same size of the input image and the number of channels as r^2. Then, the r^2 channels of each pixel of the feature image are rearranged into an r×r area, corresponding to an r×r sub-block in the high-resolution image, so that the feature image of size H×W×r^2 is rearranged into a high-resolution image of rH×rW×1. The calculation formula is as follows:

[0024]

[0025] Where r represents the upsampling coefficient; i and j represent the coordinates of the upsampled feature map, C represents the channel dimension of the input feature map X (Cth channel), and c represents the channel dimension of the upsampled feature map Y;

[0026] (2) Channel Attention

[0027] First, the global pooling operation is used to compress the spatial dimension of each channel, and then the fully connected layer is used to learn the importance weight of the channel; finally, the channel features are weighted and summed with the learned weights to obtain a feature representation with enhanced attention to optimize the performance and effect of the model. The following formula exists for calculating the channel attention map:

[0028] M c (F) = σ(MLP(AvgPool(F))

[0029] Where Mc(F) represents the channel attention map; σ represents the sigmoid function; MLP represents the multi-layer perception layer; this method extends it as follows; F∈R WxHxC Represents the output of the feature map, AVG represents the average pooling operation, and F is obtained by the pooling operation on the spatial dimension F'∈R 1x1xC (F' = AvgPoo(F)).

[0030] Then reshape the channel attention map to convert it into M c ∈R 1x1x4xC , and perform a softmax operation on the transformed channel attention map in the hierarchical dimension, calculated as follows:

[0031]

[0032] Where i represents the i+2th feature layer, j represents the jth channel, and the fusion of multi-layer feature maps is achieved by adaptively adjusting the weights of different feature layers of the corresponding channels. c It means that the channel attention map is obtained after the normalization operation (softmax).

[0033] (3) Self-attention

[0034] Self-attention generates a more representative representation by weighting the importance of each element by leveraging the associated information within the input sequence. It also calculates the attention weight between each pair of elements and interacts with each other to consider global context information. By learning the attention weights, the model can adaptively capture the importance of each element in the input sequence, thereby achieving more accurate representation learning. The following calculations exist:

[0035]

[0036] Among them, W Q ,W K ,W V are learnable weight matrices that are used to linearly transform the feature map, and d is the feature dimension after transformation, where X∈R WxHxC For the input feature map ∈R WxHxC CxHxW,F∈R WxHxC is the output feature map enhanced by self-attention.

[0037] (4) Pseudo-mask annotation

[0038] The bounding box of the object is filled with its category, the background is filled with 0, and for the supervised loss, focalloss is used to calculate the loss. The formula is as follows:

[0039] FL(p t )=-α t (1-p t ) γ log(p t )

[0040] Among them, p t Indicates the probability that the model predicts that the sample belongs to the positive sample, α t represents the weight of sample t, and γ represents the adjustment factor of sample difficulty.

[0041] Furthermore, the anchor-free frame design performs the following calculations on the SMAM network:

[0042] (1) Center point offset prediction

[0043] For a detection box OBB(X ctr ,Y ctr ,W,H,θ), where (X ctr ,Y ctr ) represents the center point coordinates, W and H represent the width and height of the detection frame, θ∈[-π / 2,π / 2) represents the acute angle between the detection frame and the x-axis, corresponding to the long side W of the detection frame, and the detection frame is represented as (X ctr ,Y ctr ,w gt,hgt,Δα,Δβ), where w and h represent the width and height of the minimum bounding rectangle of the OBB, and Δα and Δβ are the offset differences relative to the top and right midpoints of the minimum bounding rectangle of the detection frame. For a positive sample point on the feature map, the regression target calculation formula is as follows:

[0044]

[0045] Among them, x, y correspond to the coordinates of the feature point in the original image, l, t, r and b represent the vertical distance from the point to the left, top, right and bottom sides of the circumscribed rectangle of the detection frame, t * α , t * β Both represent the ratio of the center point offset distance to the width and height of the circumscribed rectangular frame;

[0046] (2) Loss function design

[0047] The loss of the RPN network consists of classification loss and regression loss. QualityFocalLoss combines the target quality information in target detection on the basis of FocalLoss, integrates the target positioning quality into the classification loss, and forms a joint representation. The formula is as follows:

[0048] y = RotatedIOU(A,G)

[0049] QFL(σ)=-|y-σ| β ((1-y)log(1-D)+ylog(σ))

[0050] Where A represents the predicted rotation box, G represents the instance box, y represents the IOU value between the predicted box and the actual box, and σ represents the predicted classification confidence; QualityFocalLoss can make full use of the quality information of the target, perform weighted processing on targets of different qualities, and more finely adjust the model's attention to different targets, thereby improving detection performance;

[0051] For regression loss, it is divided into distance loss and center point offset loss as follows:

[0052]

[0053] In this method, CLOU_LOSS is used as the loss function for distance loss.

[0054] For the center point offset angle loss, SmoothL1 Loss is used, and its calculation formula is as follows:

[0055]

[0056] Where x represents the difference between the predicted value and the target value.

[0057] The loss of the RCNN detection head network consists of classification loss and regression loss, and the loss function of the original Faster RCNN is used.

[0058] The classification loss is calculated as follows:

[0059]

[0060] L cls (p t )=(1-p t ) γ log(p t )

[0061] Where p is the probability of a positive sample, label represents the predicted category, and label* represents the corresponding true value label.

[0062] The regression loss is calculated as follows:

[0063]

[0064] Among them, the smoothL1 loss function is as described above, is the coordinate vector of the real frame, t=(t x ,t y ,t w ,t h ,t α ,t β ) is the coordinate vector where the prediction box is located.

[0065] Beneficial effects: Compared with the prior art, the method provided by the present invention integrates the attention mechanism to improve the feature extraction capability of the network, and enhances the feature expression capability by adding a mask attention module after the neck network. Specifically, the present invention designs an AFOrpn based on center point detection, which does not require the use of anchor frames, reduces the hyperparameters introduced by anchor frames, and reduces computational complexity. Then, in response to the inconsistency problem of classification and regression, QualityFocalLoss is used to jointly optimize classification and regression, and the center point offset is used to represent indirect prediction instead of directly predicting the angle to avoid the sensitivity problem of angle prediction. Secondly, the present invention includes the implementation of a supervised mask attention module, which adaptively adjusts the fusion weights of the texture information of shallow features and the deep semantic information by combining channel attention in the fusion process, and eliminates the aliasing effect caused by the fusion by adaptively integrating local features and global dependencies through spatial attention. Finally, background noise is targetedly suppressed through mask supervision, key features are retained, and detection accuracy is significantly improved. Therefore, the present invention improves the detection accuracy of remote sensing image target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a flowchart of a robust remote sensing image target detection method according to an embodiment of the present invention;

[0067] Figure 2 A schematic diagram of the overall network structure provided by the present invention;

[0068] Figure 3 Schematic diagram of the Supervised Mask Attention Module (SMAM) processing process provided by the present invention;

[0069] Figure 4 This is a schematic diagram showing the predicted offset of the center point of the detection frame provided by the present invention;

[0070] Figure 5 This is a visualization result diagram of a robust remote sensing image target detection method of the present invention. DETAILED DESCRIPTION

[0071] To introduce the technical solution disclosed in the present invention in detail, further description is given below with reference to the accompanying drawings.

[0072] The present invention provides a method for detecting rotational targets in remote sensing images based on an anchor-free frame under an attention mechanism. Compared with the existing remote sensing image target detection technology, the present invention can significantly improve the detection accuracy of remote sensing image target detection. Figure 1 The present invention can be implemented according to the following steps.

[0073] S1. Obtain a remote sensing image set, which is divided into a training set and a test set.

[0074] In order to better train remote sensing images, the present invention first preprocesses the remote sensing images: remote sensing images of any size are divided into sub-images of a uniform size of 1024x1024. In order to ensure that all targets can be placed in the sub-images, the sub-images are divided into remote sensing images with an overlap rate of 224 pixels. At the same time, considering the need to ensure that the sub-images can be mapped to the original image, the name of each sub-image is uniformly named according to the rule of the original image name plus the upper left corner coordinate of the cutting position.

[0075] S2. Improve Faster RCNN to build a remote sensing image detection network model.

[0076] Faster RCNN consists of a backbone network, a neck network, SMAM, AFOrpn, and an RCNN detection head network. In this paper, Faster RCNN is improved to create a network model suitable for remote sensing image detection. This improved model includes the following components:

[0077] Backbone network, neck network, SMAM, AFOrpn and RCNN detection head network (RCNN detection head network includes RotatedRoiAlign pooling layer, fully connected layer with Relu activation function, a Softmax classifier and a regression loss layer (Regressor)).

[0078] To further explain, the Faster RCNN model's backbone network uses the ResNet50 architecture, while the neck network uses the FPN architecture. This invention extracts multi-layer feature maps from the backbone and neck networks and inputs them into the SMAM module. This fused feature map is generated through an attention mechanism and then added to the original features through scaling operations, improving the expressiveness of the original features. The fused features are then input into the AFOrpn module for proposal screening. The AFOrpn module's design utilizes the FCOS anchor-free detection head network concept. Finally, the proposals are passed through the RCNN detection head network to output the final detection results.

[0079] The design of SMAM takes into account the need to extract effective target features, screen and detect these features to achieve optimal results. However, due to the complexity and diversity of remote sensing images, existing technologies, which simply extract feature information through a backbone network and enhance it with a feature pyramid, are unable to identify rich objects in remote sensing images. This results in difficulties in detecting dense targets, low localization accuracy of multi-scale targets, and the tendency of small targets to be missed or misdetected, significantly reducing the effectiveness of the detection model. Therefore, further structural optimization and adjustments to the Faster RCNN network are necessary. The attention mechanism is effective in extracting key features and suppressing background noise, and it primarily acts on feature maps. Therefore, by integrating the attention mechanism at appropriate locations in the network, the network's feature extraction capabilities can be effectively improved. In this invention, a masked attention module is added after the neck network to enhance feature representation. The design of AFOrpn takes into account the characteristics of remote sensing targets: densely arranged, arbitrarily oriented, and at multiple scales. While models based on anchor frame design achieve satisfactory precision, this relies on pre-setting a large number of carefully designed anchor frames for prediction, which significantly increases the model's complexity. The anchor-free concept based on FCOS eliminates the need to design a large number of anchor boxes and achieves a lightweight model while ensuring model recognition accuracy. This is explained below.

[0080] Combine Figure 2 It can be seen that the network model designed by the present invention consists of four parts:

[0081] 1) A feature extraction network consisting of a backbone network and a neck network. The backbone network adopts the ResNet50 structure and outputs feature maps {C3, C4, C5}. The neck network adopts the FPN structure and obtains the feature pyramid {P3, P4, P5, P6} through lateral connections and top-down feature fusion and pooling operations on the top feature map C5.

[0082] 2) The feature enhancement module SMAM fuses multi-layer features and performs background suppression through the attention mechanism. It outputs the fused feature O, which is fused and enhanced with the feature pyramid through scaling operations to obtain a new feature pyramid {E3, E4, E5, E6};

[0083] 3) AFOrpn network for multi-scale object detection and screening of high-quality porposals. Due to the inconsistency between the sensitive features of classification and regression, AFOrpn consists of two parallel branches. Each branch has two convolutions (Conv / Bn / Relu) to transform the feature map into the corresponding feature map for classification / regression. The corresponding prediction results are generated through 1x1 convolution.

[0084] 4) Refine the object detection box and perform multi-classification rcnn detection head network, whose output is the corresponding category probability {P1,P2,P3,....Pn}, and the offset relative to the proposal {t x， t y， t w， t h， tθ}, the model decodes the offset on the proposals to obtain the final detection box and outputs the final detection result in combination with the category probability.

[0085] Combine Figure 3 , Figure 3 The feature maps {P3, P4, P5, P6} output by the FPN are upsampled to a uniform size (256x 128x 128) through sub-pixel convolution and concatenated in the channel dimension to obtain concatenated features (1024x128x128). The concatenated features are then input into the channel attention module (CAM) to achieve adaptive fusion of multi-layer features to obtain fused features of size 256x128x128. They are then input into the self-attention module to restore long-range feature dependencies and eliminate aliasing effects caused by fusion, resulting in enhanced features of size 256x128x128. Finally, two parallel branches are input to obtain the mask prediction result M and the feature map O (256x128x128) enhanced by the supervision module. The mask prediction module branch consists of 3x3 conv, relu, and 1x1 convolution, while the supervision attention map A branch consists of 3x3 conv and sigmoid activation function.

[0086] Specifically, first, the feature pyramid {P3, P4, P5, P6} extracted by the backbone network and FPN is used, where p1 and p2 are not enabled in this embodiment. The reasons are:

[0087] 1) Reduce the amount of computation and reduce model complexity: The feature map resolution of p1 and p2 is too high, which will increase the amount of model computation.

[0088] 2) Increase model recognition accuracy.

[0089] The feature map P1 is extracted in the early stage of the backbone network, so it is usually a low-level feature representation of the input image with low semantic information. Such low-level features are not sufficient for accurate detection and localization of complex objects.

[0090] Then, sub-pixel convolution is used to upsample the feature maps to a downsampling factor of 8 and then splice them in the channel dimension. Then, adaptive feature fusion is performed through the channel attention mechanism. Then, the spatial attention machine is used for further enhancement to eliminate the aliasing effect. Finally, through the annotation mask supervision, the final fused feature map is obtained, which is then scaled and added to the original feature map {P3, P4, P5, P6} for enhancement. The specific implementation process includes the following:

[0091] (1) Supervised Masked Attention (SMAM) Module

[0092] SMAM consists of three stages: feature fusion, spatial attention, and supervised mask enhancement. The feature fusion stage uses the concept of channel attention to adaptively adjust the weights of feature channels and deep and shallow features, enabling the comprehensive utilization of multiple features. Spatial attention uses the concept of self-attention to adaptively integrate local features and global dependencies to eliminate aliasing effects caused by fusion. The supervised mask enhancement stage uses a pseudo-mask to guide the network to retain key features and filter out noisy information.

[0093] Feature fusion unifies feature maps of different resolutions to the same size through up and down sampling operations, and comprehensively utilizes multi-level feature information by splicing and weighting feature maps from different levels.

[0094] Upsampling: Traditional image super-resolution algorithms usually use interpolation techniques to enlarge low-resolution images, but this method easily leads to distortion and blurring of feature maps. In contrast, sub-pixel convolution uses convolution operations to increase the details and clarity of the image and better preserve the detailed information of the image. The implementation is as follows: Input a low-resolution image of size HxWxC, and through the convolution operation, obtain a feature image with r^2 channels and the same size as the input image. Then rearrange the r^2 channels of each pixel of the feature image into an r×r area, corresponding to an r×r sub-block in the high-resolution image, so that the feature image of size H×W×Cxr^2 is rearranged into a high-resolution image of rH×rW×C. The formula is as follows:

[0095]

[0096] Where r represents the upsampling coefficient.

[0097] Channel attention: Channel attention improves the model's focus on specific channels by learning the importance of each channel. First, a global pooling operation is used to compress the spatial dimension of each channel, and then a fully connected layer is used to learn the importance weights of the channels. Finally, the channel features are weighted and summed with the learned weights to obtain a feature representation with enhanced attention, thereby optimizing the performance and effectiveness of the model. The formula is as follows:

[0098] M c (F) = σ(MLP(AvgPool(F))

[0099] Where Mc(F) represents the channel attention map; σ represents the sigmoid function; and MLP represents the multi-layer perception layer. However, the traditional channel attention does not involve the compression of the channel dimension, and the essence of feature fusion is the comprehensive utilization of multiple features. Therefore, the present invention expands it. Specifically, the present invention reshapes the attention map to convert it into M c ∈R 1x1x4xC , and perform a softmax operation on the transformed attention map in the layer dimension:

[0100]

[0101] Where i represents the i+2th feature layer and j represents the jth channel. The weights of different feature layers of the corresponding channels are adaptively adjusted to effectively achieve the fusion of multi-layer feature maps.

[0102] Self-attention: Self-attention is designed to leverage the correlation information within the input sequence to weight the importance of each element, thereby generating a more representative representation. It calculates attention weights between each pair of elements, interacting with each element with other elements to take into account global contextual information. By learning attention weights, the model is able to adaptively capture the importance of each element in the input sequence, thereby achieving more accurate representation learning:

[0103]

[0104] Among them, W Q ,W K ,W V are learnable weight matrices that are used to perform linear transformations on feature maps, and d is the feature dimension after the transformation.

[0105] Pseudo-mask supervision design: Existing remote sensing datasets do not have mask annotations, so this paper fills the bounding box of the object with its category and the background with 0. For the supervision loss, this paper chooses to use Focal loss to calculate the loss. The formula is as follows:

[0106]

[0107] FL(p t )=-α t (1-p t ) γ log(p t )

[0108] Among them, p represents the probability that the model predicts that the sample belongs to the positive sample, α t represents the weight of sample t, and γ represents the adjustment factor of sample difficulty.

[0109] S3. Design the target detection AFOrpn module and use the feature map obtained in S2 as the input of the AFOrpn module. The AFOrpn module detects the category of each pixel in the feature map and the regression of the detection box.

[0110] Based on FCOS, this paper designs a new regression method to solve the problem of inaccurate accuracy caused by the sensitivity of remote sensing images to angles due to large aspect ratio objects. The regression method is based on the four vertical distances (l, t, r, b) of the feature point coordinates to the minimum bounding rectangle of the instance box, as well as the ratio of the offset value relative to the top and right midpoint of the bounding rectangle to the width and height of the corresponding ground truth box. α , t β For details, please refer to Figure 4 .

[0111] Based on existing techniques, anchor-based RPN networks generate anchors of varying sizes and aspect ratios for each pixel in the feature map, using IoU as the criterion for distinguishing positive and negative samples during training. While this improves detection accuracy, anchors introduce a large number of difficult-to-adjust hyperparameters, increasing model training complexity. Anchor design requires strong prior information, making it difficult to design anchors for objects with unusual shapes, leading to missed detections. This paper proposes designing an RPN network based on the concept of FCOS.

[0112] S4, extract the topk prediction boxes with the highest scores in each layer of feature map through the classification confidence of S3, and filter them through NMS (non-maximum suppression) to select the top_n boxes with the highest scores as proposals.

[0113] like Figure 4 As shown, in the present invention, a method for predicting the regression frame by center point offset is designed, which is as follows:

[0114] For a detection box OBB(X ctr ,Y ctr ,W,H,θ), where (X ctr ,Y ctr ) represents the center point coordinates, W, H represent the width and height of the detection frame, θ∈[-π / 2,π / 2) represents the acute angle between the detection frame and the x-axis, corresponding to the long side W of the detection frame. In this paper, the detection frame is represented as (X ctr ,Y ctr ,w,h,α,β), where w,h represent the width and height of the minimum bounding rectangle of the OBB, and α,β are the offset values ​​relative to the top and right midpoints of the minimum bounding rectangle of the detection frame. Then for a sample point on the feature map, the encoding calculation formula for its prediction value is as follows:

[0115]

[0116] Among them, x, y correspond to the coordinates of the feature point in the original image, l, t, r and b represent the vertical distance from the point to the left, top, right and bottom sides of the circumscribed rectangle of the detection frame. α , t β Indicates the ratio of the center point offset distance to the width and height of the bounding rectangle.

[0117] S4. According to the proposal, the RotatedRoiAlign pooling layer is used to extract features and input them into the subsequent fully connected layer. Finally, the softmax layer is used for classification and the regression loss layer is used to refine the proposal.

[0118] S5. Design loss functions, calculate the supervised mask loss function, calculate the AFOrpn loss function: classification loss function, regression loss function, including distance loss function and center point offset loss function. Calculate the RCNN network loss function: classification loss function, regression loss function.

[0119] (1) Loss function design

[0120] The loss of the RPN network consists of classification loss and regression loss.

[0121] The currently commonly used classification loss Focal Loss effectively reduces the impact of easily classified samples on the training process by adjusting the weight of the loss function, making the model pay more attention to difficult-to-classify samples, thereby improving the performance of the model in the case of class imbalance. However, this ignores the inconsistency problem between classification and regression. QualityFocalLoss combines the target quality information in target detection on the basis of Focal Loss, and directly integrates the target positioning quality (such as the overlap measure between the bounding box and the real object, such as the IOU score) into the classification loss to form a joint representation. The formula is as follows:

[0122] y = RotatedIOU(A,G)

[0123] QFL(σ)=-|y-σ| β ((1-y)log(1-σ)+ylog(σ))

[0124] Where A represents the predicted rotated bounding box, G represents the instance bounding box, y represents the IoU between the predicted and actual bounding boxes, and σ represents the predicted classification confidence. QualityFocalLoss fully utilizes the quality information of the target, weighting targets of different qualities to more finely adjust the model's focus on different targets, thereby improving detection performance.

[0125] For regression loss, the present invention divides it into distance loss and center point offset loss as follows:

[0126]

[0127] In the present invention, CLOU_LOSS is used as the loss function for distance loss:

[0128]

[0129] Where IOU represents the intersection-over-union ratio of the minimum bounding rectangle of the predicted rotation box to the minimum bounding rectangle of the actual box, w * , h *The width and height of the minimum bounding rectangle of the actual box correspond to w and h, respectively. The width and height of the minimum bounding rectangle of the predicted rotated box are w and h. b and b* represent the midpoints of the predicted box and the ground-truth box, respectively. c represents the diagonal length of the minimum bounding rectangle that contains the minimum bounding rectangle of the predicted box and the minimum bounding rectangle of the ground-truth box. Compared to the IOU loss used by FCOS, the CIOU loss can better handle scale mismatches by introducing a penalty for the distance between the center points of the bounding box. Furthermore, the CIOU loss takes into account factors such as the distance between the center points of the bounding box and the aspect ratio, allowing the loss function to more comprehensively evaluate the degree of match between the predicted box and the ground-truth box, making the model more accurate and robust.

[0130] For the center point offset angle loss, the present invention adopts SmoothL1 Loss. The formula is as follows:

[0131]

[0132] Where x represents the difference between the predicted value and the target value.

[0133] The loss of the RCNN detection head network consists of classification loss and regression loss, and the loss function of the original Faster RCNN is used.

[0134] The classification loss is calculated as follows:

[0135]

[0136] L cls (p t )=(1-p t ) γ log(p t )

[0137] Where p is the probability of a positive sample, label represents the predicted category, and label* represents the corresponding true value label.

[0138] The regression loss is calculated as follows:

[0139]

[0140] Among them, the smoothL1 loss function is as described above, is the coordinate vector of the real frame, t=(t x ,t y ,t w ,t h ,t α ,t β ) is the coordinate vector of the predicted box:

[0141]

[0142] Where x and y are the center coordinates of the prediction box, w and h are the width and height of the minimum bounding rectangle of the prediction box, and α and β are the offset values ​​relative to the top and right midpoints of the minimum bounding rectangle of the prediction box. p and y p is the center coordinate of the proposal output by the AFOrpn network, w p and h a are the width and height of the minimum bounding rectangle of the corresponding proposal, α p , β p It is expressed as the offset value relative to the top and right midpoint of the proposal's minimum bounding rectangle, where x* and y* are the center coordinates of the real box, and w* and h* are the width and height α of the minimum bounding rectangle of the real box, respectively. * , β * Expressed as the offset value relative to the top and right midpoints of the minimum bounding rectangle of the real box.

[0143] S6, update the parameters of the remote sensing image rotation target detection model, complete the model training, and use the trained target detection model to perform remote sensing image target detection. Figure 5 The visualization results are shown.

Claims

1. A method for detecting rotated objects in remote sensing images based on an attention mechanism and without anchor boxes, characterized by: The steps include: S1. Obtain a remote sensing image set and divide it into a training set and a test set; S2. Build a remote sensing image detection network model based on the improved Faster RCNN, including the fusion of attention mechanism to improve the network's feature extraction ability, and enhance the feature expression ability by adding a mask attention module after the neck network; The remote sensing image detection network model is based on Faster RCNN with increased supervision mask attention and anchor-free frame design; SMAM consists of three stages: feature fusion, spatial attention, and supervised mask enhancement: Feature fusion unifies feature maps of different resolutions to the same size through upsampling operations. It then concatenates feature maps from different layers and uses the idea of ​​channel attention to adaptively adjust the feature channels and the weights of deep and shallow features for additive fusion, thereby comprehensively utilizing multi-level feature information. Spatial attention uses the idea of ​​self-attention to adaptively integrate local features and global dependencies to eliminate the aliasing effect caused by fusion; In the supervised mask enhancement stage, a pseudo mask is used to guide the network to retain image features and filter out noise information; The anchor-free frame design is performed for the AFOrpn network as follows: (1) Center point offset prediction For a detection box OBB(X ctr ,Y ctr ,W,H,θ), where (X ctr ,Y ctr ) represents the center point coordinates, W and H represent the width and height of the detection frame, θ∈[-π / 2,π / 2) represents the acute angle between the detection frame and the x-axis, corresponding to the long side W of the detection frame, and the detection frame is represented as (X ctr ,Y ctr， w, h, α, β), where w and h represent the width and height of the minimum bounding rectangle of the OBB, and α and β are the offset values ​​relative to the top and right midpoints of the minimum bounding rectangle of the detection box. For a sample point on the feature map, the encoding calculation formula of its AFOrpn predicted regression value is as follows: Among them, x, y correspond to the coordinates of the feature point in the original image, l, t, r and b represent the vertical distance from the point to the left, top, right and bottom sides of the circumscribed rectangle of the detection frame, t α t β Both represent the ratio of the center point offset distance to the width and height of the circumscribed rectangular frame; (2) Loss function design The loss of the AFOrpn network consists of classification loss and regression loss; The classification loss is calculated using QualityFocalLoss, which combines the target quality information in target detection with Focal Loss and integrates the target positioning quality into the classification loss to form a joint representation. The formula is as follows: y = RotatedIOU(A,G) QFL(σ)=-|y-σ| β ((1-y)log(1-σ)+ylog(σ)) Where A represents the predicted rotation box, G represents the instance box, y represents the IOU value between the predicted box and the actual box, and σ represents the predicted classification confidence; QualityFocalLoss can make full use of the quality information of the target, perform weighted processing on targets of different qualities, and more finely adjust the model's attention to different targets, thereby improving detection performance; For regression loss, it is divided into distance loss and center point offset loss as follows: In this method, CLOU_LOSS is used as the loss function for distance loss; For the center point offset angle loss, SmoothL1 Loss is used, and its calculation formula is as follows: Where x represents the difference between the predicted value and the target value; The loss of the RCNN detection head network consists of classification loss and regression loss, and the loss function of the original Faster RCNN is used; The classification loss is calculated as follows: L cls (p t )=(1-p t ) γ log(p t ) Where p is the probability of a positive sample, label represents the predicted category, and label* represents the corresponding true value label; The regression loss is calculated as follows: Among them, the smoothL1 loss function is as described above, is the coordinate vector of the real frame, t=(t x ,t y ,t w ,t h ,t α ,t β ) is the coordinate vector of the predicted box; S3. Design the target detection AFOrpn network. The feature map obtained by the remote sensing image detection network model is used as the input of the AFOrpn network. The AFOrpn network is used to detect the category of the pixel points in the feature map and the regression processing of the detection box. The detection box is screened and the proposal box is obtained through the classification confidence and non-maximum suppression operation. The proposed box is extracted through the RotatedRoiAlign pooling layer and input into the subsequent fully connected layer. Finally, it is classified through the softmax layer and the regression loss layer is used to refine the proposed box. S4. Design loss functions, including AFOrpn network loss, RCNN loss, and supervised mask loss; The AFOrpn network loss consists of classification loss and regression loss, and the regression loss of the AFOrpn network is split into distance loss and center offset loss; RCNN loss is composed of classification loss and regression loss, and the original loss of Faster RCNN detection network is used. S5. Update the parameters of the remote sensing image rotation target detection model to complete the model training, and then use the trained target detection model to perform remote sensing image target detection.

2. The method for detecting rotated objects in remote sensing images based on an anchor-free frame under an attention mechanism according to claim 1, wherein: The remote sensing image detection network model includes a backbone network, a neck network, a SMAM, an AFOrpn and an RCNN detection head network; The RCNN detection head network includes a RotatedRoiAlign pooling layer, a fully connected layer with a Relu activation function, a Softmax classifier, and a regression loss layer; The backbone network of the remote sensing image detection network model is a resnet50 structure, and the neck network adopts an FPN structure. The input image is extracted into a multi-layer feature map through the backbone network and the neck network, and input into a fused feature map obtained by the attention mechanism. The fused features are added to the original features through a scaling operation to improve the expression ability of the original features, and are input into AFOrpn for proposal screening. The design of AFOrpn is based on the anchor-free box detection head network implementation of FCOS. Finally, the proposals are output as the final detection results through the RCNN detection head network.

3. The method for detecting rotated objects in remote sensing images based on an anchor-free frame under an attention mechanism according to claim 1, wherein: The anchor-free frame design is based on the RPN network of the prediction frame to directly generate prediction frames of different sizes and aspect ratios for pixels on the feature map. In this method, the anchor-free frame design realizes the construction of the RPN network based on the FCOS algorithm.

4. The method for detecting rotated objects in remote sensing images based on an anchor-free frame under an attention mechanism according to claim 1, wherein: Supervised mask attention includes the following calculations: (1) Upsampling For the input low-resolution image, a sub-pixel convolution operation is performed to obtain a feature image with the same size of the input image and the number of channels as r^2. Then, the r^2 channels of each pixel of the feature image are rearranged into an r×r area, corresponding to an r×r sub-block in the high-resolution image, so that the feature image of size H×W×r^2 is rearranged into a high-resolution image of rH×rW×1. The calculation formula is as follows: Where r represents the upsampling coefficient, i j represents the coordinates of the upsampled feature map, C represents the channel dimension of the input feature map X, and c represents the channel dimension of the upsampled feature map Y; (2) Channel Attention First, the global pooling operation is used to compress the spatial dimension of each channel, and then the fully connected layer is used to learn the importance weight of the channel; finally, the channel features are weighted and summed with the learned weights to obtain a feature representation with enhanced attention to optimize the performance and effect of the model. The following formula exists for calculating the channel attention map: M c (F)=σ(MLP(AvgPool(F)) Where Mc(F) represents the channel attention map; σ represents the sigmoid function; MLP represents the multi-layer perception layer; this method expands it as follows, F represents the output feature map, and AVG represents the average pooling operation; Then reshape the channel attention map to convert it into M c ∈R 1x1x4xC , and perform a softmax operation on the transformed channel attention map in the hierarchical dimension, that is, the normalized calculation exists: Where i represents the i+2th feature layer, j represents the jth channel, and the fusion of multi-layer feature maps is achieved by adaptively adjusting the weights of different feature layers of the corresponding channels. Mc represents the channel attention map obtained after normalization operation; (3) Self-attention Self-attention generates a more representative representation by weighting the importance of each element by leveraging the associated information within the input sequence. It also calculates the attention weight between each pair of elements and interacts with each other to consider global context information. By learning the attention weights, the model can adaptively capture the importance of each element in the input sequence, thereby achieving more accurate representation learning. The following calculations exist: Among them, W Q ,W K ,W V are learnable weight matrices that are used to perform linear transformations on feature maps. X is the input of the feature map, and d k Represents the feature dimension after transformation, F represents the output of the feature map, which is the output feature map enhanced by self-attention; (4) Pseudo-mask supervision design The bounding box of the object is filled with its category, the background is filled with 0, and for the supervised loss, the Focal loss is used to calculate the loss. The formula is as follows: FL(p t )=-a t (1-p t ) γ log(p t ) Among them, p represents the probability that the model predicts that the sample belongs to the positive sample, α t represents the weight of sample p, and γ represents the adjustment factor of sample difficulty.

Images