Remote sensing image target detection method based on feature pyramid and boundary-aware vector

Abstract

This invention discloses a remote sensing image target detection method based on feature pyramids and boundary-aware vectors. The method includes the following steps: S1. Feeding the input image into the feature pyramid backbone to extract multi-scale features; S2. Using a dynamic selection mechanism, adaptively selecting a suitable kernel for different targets based on the extracted multi-scale features; S3. Inputting the comprehensive features into a decoder to obtain predicted coordinates and target categories. This invention combines feature pyramids with learned bounding box boundary-aware vectors to capture the rotated bounding boxes of targets, significantly reducing computational load and achieving higher accuracy, meeting the application needs of the aerospace field for remote sensing image rotation target detection under resource constraints.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, and in particular to a method for detecting rotating targets in optical remote sensing images based on feature pyramids and boundary-aware vectors. Background Technology

[0002] Optical remote sensing image rotation target detection is a technique for detecting rotating targets in remote sensing images. Remote sensing images are image data of the Earth's surface acquired through platforms such as satellites and drones. They are characterized by wide coverage, high resolution, and rich geographic information. Because targets in remote sensing images (such as aircraft, ships, and vehicles) can appear at any angle, they have significant application value in various fields such as military reconnaissance, intelligent transportation, environmental monitoring, and land use planning.

[0003] With the increasing application of Convolutional Neural Networks (CNNs) in computer vision, target detection in aerial imagery has rapidly developed. Related models typically use Horizontal Bounding Boxes (HBBs) to locate targets. However, most targets in remotely sensed images are characterized by dense distribution, occlusion, and high aspect ratios; therefore, HBB-based models may result in severe misalignment and overlap between the detected bounding boxes and the targets, and the size and aspect ratio may not reflect the true shape of the targets. To address this issue, rotated bounding boxes are used to process these targets. These contain fewer background pixels, making target-background classification easier, increasing the accuracy of target capture, and significantly reducing the overlap between adjacent targets compared to CNNs. Therefore, rotated bounding boxes are one of the effective solutions for capturing targets in aerial images.

[0004] Target detection in aerial imagery presents unique challenges due to bird's-eye view, complex backgrounds, and varying target appearances. This often results in high confidence scores for target classification but inaccurate localization. Remote sensing target detection methods frequently employ detectors based on two-level anchors. For example, using the DOTA dataset, the Faster R-CNN algorithm based on two-level anchors achieved a mean average precision (mAP) of 0.85 for detecting aircraft targets, while the YOLOv3 algorithm with a single-level anchor only achieved an mAP of 0.72. This is primarily because the two-level anchor method generates high-quality region proposals through RPN in the first stage, enabling more accurate approximate target location and providing a solid foundation for subsequent fine-grained detection. In the second stage, ROI feature merging and refinement further improve detection accuracy. Replacing anchor boxes with center points allows for adaptive target localization using spatial information and semantic information of key regions, resulting in improved detection accuracy. Typically, in the first stage, these detectors densely expand anchor boxes on the feature map and then regress the offset between the target box and anchor box parameters to provide region suggestions. In the second stage, the region of interest (ROI) features are merged to refine the bounding box parameters and classify the target categories. They use center, width, height, and angle as descriptions of the rotated bounding boxes.

[0005] In recent years, keypoint-based object detectors have been developed, addressing the slow inference speed of anchor-based methods in horizontal object detection due to their computational efficiency and fast processing capabilities. These methods detect the corners of bounding boxes and group them by comparing the embedding distance or center distance of the points. While these strategies show improved performance, they are computationally expensive due to the grouping process. To address this issue, CenterNet detects the center of the object and directly regresses the width (w) and height (h) of the bounding box, achieving faster detection speeds with the same accuracy. Intuitively, by learning additional angles θ, as well as w and h, CenterNet can be extended to rotated object detection. However, since w and h are measured in different rotation coordinate systems for each arbitrary rotated object, the joint learning of these parameters poses a significant challenge to the model.

[0006] Traditional horizontal target detection methods often struggle to accurately identify and locate rotating targets, necessitating specialized rotating target detection techniques to address this issue. Detection of rotating targets in remote sensing images includes densely packed targets, partially occluded targets, and targets with high aspect ratios. Traditional horizontal bounding box detection methods fail to accurately describe the true shape of these targets, leading to significant alignment errors between the detection box and the target, and excessive background pixels, thus reducing target detection accuracy. Compared to horizontal bounding boxes, oriented bounding boxes (OBBs) better adapt to the target's rotation angle, reduce background pixel interference, and improve target localization accuracy. However, rotating target detection faces the following technical challenges: the parameters of the OBB (such as width, height, and angle) are typically defined in different rotation coordinate systems, posing a significant challenge to the model's joint learning of these parameters. Existing detection methods often struggle to simultaneously balance detection accuracy and computational efficiency when handling targets at multiple scales and in complex backgrounds, failing to meet application requirements. Summary of the Invention

[0007] To address the problems of low experimental accuracy, slow inference speed, and high model parameters and computational load in existing technologies, the present invention aims to provide a rotating target detection method for optical remote sensing images based on feature pyramids and boundary-aware vectors. This method combines feature pyramids with learned bounding box boundary-aware vectors to capture the rotating bounding box of the target, significantly reducing computational load and achieving higher accuracy.

[0008] To achieve the above objectives, this invention provides a remote sensing image target detection method based on feature pyramids and boundary-aware vectors, the method comprising the following steps:

[0009] S1. Feed the input image into the feature pyramid backbone to extract multi-scale features;

[0010] S2. A dynamic selection mechanism is adopted to adaptively select appropriate kernels for different targets based on the extracted multi-scale features;

[0011] S3. Input the integrated features into the decoder to obtain the predicted coordinates and target category, thereby obtaining the target's rotated bounding box.

[0012] Furthermore, the backbone network of the method includes a backbone network, a neck module, an LSK module, and a detection head (here, the detection head refers to the decoder or prediction head of the four vectors below); the backbone network is implemented based on the ResNet architecture, and the ResNet101 backbone processes the RGB input image; the backbone network extracts hierarchical features by performing a series of convolutions, batch normalization, and ReLU activation layers on the input image.

[0013] Furthermore, step S1 includes:

[0014] S1-1. First, through the backbone network, four sets of feature maps of different scales are obtained through layer-by-layer convolution;

[0015] S1-2. To address the issue of different feature maps, a feature pyramid structure is used to construct bottom-up and top-down feature fusion paths, fusing feature maps of different scales to generate a feature pyramid with multi-scale information.

[0016] S1-3. The feature pyramid after FPN is stitched together to obtain a feature map of shape 152×152×256, where 152×152 is the image resolution and 256 is the number of feature maps.

[0017] Furthermore, step S2 includes:

[0018] S2-1. The input feature map is processed by the large kernel selection module, and its receptive field is expanded by 5×5 and 7×7 convolutions to obtain information about the surrounding area of ​​the target.

[0019] S2-2. A spatial selection mechanism is adopted to weight and fuse these features according to the input features, thereby generating a receptive field that expands multiple features.

[0020] Furthermore, at the top of the backbone network, a Feature Pyramid Network (FPN) is used as the neck module, through... The convolutional layer refines the upsampled feature map.

[0021] Furthermore, the core of the feature pyramid network is to effectively integrate multi-scale features. Through feature pyramids of different scales, the network can capture both fine-grained and coarse-grained information. During the upsampling process, skip connections are used to combine deep and shallow features.

[0022] Furthermore, the feature pyramid network first upsamples the deep feature maps to the size of the shallow feature maps using bilinear interpolation, and then uses... The convolutional layer refines the upsampled mapping; after concatenation with the shallow feature map, Convolutional layers refine channel features, and batch normalization and ReLU activation are applied in the latent layers.

[0023] Furthermore, the LSK module processes the input feature map and enhances its representation capabilities through a series of convolution and attention mechanisms.

[0024] Furthermore, the LSK module converts the feature map into four branches: heatmap, offset, bounding box parameters, and orientation map.

[0025] Furthermore, the LSK module uses two [unclear] when transforming feature maps. The kernel and a 256-channel convolutional layer are implemented.

[0026] Furthermore, the LSK module includes two sub-blocks: a large kernel selection sub-block and a feedforward network sub-block. In the large kernel selection sub-block of the LSK module, large kernel convolutions are first constructed by decomposing large kernel convolutions into a series of depthwise convolutions, which have increased kernel size and expansion rate. This process generates multiple features with different receptive fields. Then, a spatial selection mechanism is used to weight and fuse these features according to the input features.

[0027] The beneficial effects of this invention are as follows:

[0028] This invention uses BBAVectors as the baseline method, focusing on regression of four vectors in the Cartesian coordinate system. Comparison of this method with BBAVectors verifies the advantages of utilizing feature pyramids and boundary vectors. Experimental evaluations on the DOTA and HRSC2016 datasets demonstrate the superiority of this invention's method over existing techniques, which has the following four advantages:

[0029] a) Higher detection accuracy: This invention, through a novel architecture that integrates a feature pyramid network and boundary-aware vectors, can more accurately locate and identify rotating targets. The boundary-aware vectors, by predicting the vector information of the target's bounding box, avoid the complex joint learning of width, height, and angle found in traditional methods, thus significantly improving detection accuracy.

[0030] b) Lower computational cost: This invention employs a single-stage, anchor-free detection method, which avoids the complex region proposal generation and refined detection steps in traditional two-stage detection methods, such as Faster R-CNN, thus significantly reducing computational complexity.

[0031] c) Enhanced multi-scale adaptability: The feature pyramid network efficiently integrates multi-scale features, enabling the model to perform exceptionally well when handling small, large, and high aspect ratio targets. This multi-scale feature extraction capability makes the invention more adaptable to complex scenes. Furthermore, the invention employs a dynamic selection mechanism to adaptively select appropriate feature kernels based on the target's scale and orientation, further enhancing its ability to detect multi-scale targets.

[0032] d) Enhanced real-time performance: This invention prioritizes computational efficiency and real-time performance in its design. By optimizing the network architecture and reducing the number of model parameters, it significantly improves detection speed. Experiments show that this invention significantly outperforms existing technologies in detection speed on standard datasets (DOTA and HRSC2016), meeting the needs of real-time application scenarios.

[0033] Overall, the method of this invention can effectively balance detection accuracy and speed, meeting the application needs of the aerospace field for detecting rotating targets in remote sensing images under resource constraints. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the remote sensing image target detection method based on feature pyramids and boundary-aware vectors of the present invention.

[0035] Figure 2 This is a schematic diagram of the detection results of FLBA-R on the DOTA V1.0 dataset;

[0036] Figure 3 This is a schematic diagram of the detection results of FLBA-R on the HRSC 2016 dataset;

[0037] Figure 4 These are schematic diagrams comparing detection results, including (a) comparison of missed detections; (b) comparison of erroneous detections; (c) comparison of redundant detections; (d) comparison of angle corrections; and (e) comparison of increasing confidence levels.

[0038] Figure 5 This is a detailed schematic diagram of the detection results in the DOTA dataset;

[0039] Figure 6 This is a visual comparison diagram of feature maps before and after feature enhancement, where (a): visualization of detection results on the DOTA V1.0 test dataset; (b) and (c): in the method of this invention, the effectiveness of feature map visualization before and after processing by the LSK module is further verified.

[0040] Figure 7 This is a diagram illustrating the predicted results. Detailed Implementation

[0041] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0043] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0044] The following combination Figures 1-7 Specific embodiments of the present invention will be described in detail below. It should be understood that the specific embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the present invention.

[0045] This invention first achieves rotating target detection in optical remote sensing images using a feature pyramid structure; then, a feature integration module integrates the features extracted from the feature pyramid to better separate target and background information; finally, boundary-aware vectors are used for post-processing to obtain the final target. Compared to the original boundary-aware vector (BBAVectors) method, this method has higher detection accuracy, lower computational cost, and faster detection speed. This invention optimizes the network architecture, reduces the number of model parameters, and improves the model's scalability and adaptability, enabling it to run efficiently in resource-constrained environments.

[0046] The difference between existing models and this invention lies in their model architecture. Existing models, i.e., the baseline models, use the U-Net network for feature extraction. The U-Net structure first downsamples the image and then upsamples it. The upsampling part has some parameter redundancy and poor performance. Therefore, a feature pyramid is used to directly fuse the downsampled features from different stages, which can improve the feature extraction effect while reducing the number of parameters. Then, the LSK module is used to improve the overall performance of the network with a limited increase in the number of parameters and computation. Finally, boundary-aware vectors are used to process the extracted features to obtain the final rotated bounding box result. This invention removes the upsampling of the baseline model, uses a feature pyramid to obtain multi-scale features, and uses the feature selection module LSK to further optimize the features. Finally, the boundary-aware module of the baseline model is used to obtain better results.

[0047] In remote sensing image rotation target detection tasks, using spatial pyramids can better capture multi-scale targets, and predicting vectors in the four quadrants of the Cartesian coordinate system is superior to directly predicting the spatial parameters of the bounding box. For example... Figure 1 As shown, the remote sensing image target detection method based on feature pyramids and boundary-aware vectors of the present invention includes the following steps:

[0048] S1. Initially, the input image is fed into the feature pyramid backbone to extract multi-scale features. Multi-scale features refer to the feature maps obtained after upsampling and stacking through the feature pyramid after passing through the backbone network (ResNet101). The feature map size is 152×152×256, containing feature information of targets at different scales. Compared to the existing U-Net structure, one improvement of this invention is that the upsampling operation of U-Net is replaced by a feature pyramid. This improvement saves a significant amount of parameters and effectively improves the computation speed without reducing detection accuracy. Specific parameters are shown in Tables 1 and 2 (#P). Both this method and U-Net have upsampling operations, but they differ somewhat. The upsampling in this method uses nearest neighbor interpolation to upsample the feature maps of various sizes extracted from the backbone. This operation is simple and requires no parameter learning. In contrast, the upsampling in the U-Net structure is transposed convolution or interpolation (such as bilinear interpolation) + convolution.

[0049] These include:

[0050] S1-1. First, the backbone network, ResNet-101, is used to obtain four sets of feature maps at different scales through layer-by-layer convolution.

[0051] S1-2. To address the issue of different feature maps, a feature pyramid structure is used to construct bottom-up and top-down feature fusion paths, fusing feature maps of different scales to generate a feature pyramid with rich multi-scale information.

[0052] S1-3. The feature pyramid after FPN is stitched together to obtain a feature map of shape 152×152×256, where 152×152 is the image resolution and 256 is the number of feature maps.

[0053] S2. Subsequently, a dynamic selection mechanism is adopted to adaptively select appropriate kernels for different targets based on the extracted multi-scale features, including:

[0054] S2-1. The input feature map is processed by the large kernel selection module, which expands its receptive field through 5×5 and 7×7 convolutions to obtain information about the surrounding area of ​​the target.

[0055] S2-2. Then, a spatial selection mechanism is used to weight and fuse the results obtained from different large kernel selection modules. This process expands the receptive field of various features.

[0056] The features here are those processed by 5×5 and 7×7 kernels. The features after these two large kernel convolutions are weighted and fused.

[0057] S2-3 fuses the features obtained through the spatial selection mechanism with the original features obtained through FPN to obtain the final comprehensive features.

[0058] S3. Finally, the comprehensive features are input into the decoder to obtain the predicted coordinates and target category. The decoder, or prediction head, decodes the feature map output by LSK, obtaining four branches: heatmap, offset, bounding box parameters, and orientation map. Specifically, the tensor shape obtained from the heatmap is 15×152×152, the tensor shape obtained from the offset is 2×152×152, the tensor shape obtained from the bounding box parameters is 10×152×152, and the tensor shape obtained from the orientation map is 1×152×152. This invention combines feature pyramids with boundary-aware vectors to capture the rotated bounding boxes of targets, significantly reducing computational cost and achieving higher accuracy. Furthermore, large-kernel convolutions are used to enhance intermediate features, further improving the final accuracy.

[0059] like Figure 1 As shown, the FLBANet proposed in this invention is built on the Feature Pyramid Network (FPN) architecture and is specifically designed for object detection and related tasks. It includes a backbone network, a neck module, an LSK module, and a head branch. The backbone network (also known as the main network) is responsible for feature extraction; in this invention, the backbone network uses ResNet101. The Feature Pyramid Network (FPN) helps the model better handle targets of different sizes by constructing multi-scale feature representations. Object detection is divided into four parts: the backbone network, the neck network, and the head. The backbone network uses ResNet101, the neck network uses FPN, and the head uses the BBAVector head. The model in this invention improves upon the feature extraction and target perception parts of the baseline model. The FPN module enhances the feature extraction capability for targets of different sizes, and the LSK module perceives information around the target to improve overall accuracy.

[0060] In step S1, the backbone network is implemented based on the ResNet architecture. Specifically, the ResNet101 backbone processes the RGB input image. ,in W and W represent the height and width of the input image, respectively. The backbone network extracts hierarchical features by performing a series of convolutions, batch normalization, and ReLU activation layers on the input image. Each residual block in the backbone network includes a convolutional layer with skip connections, which enables the network to learn residual mappings and facilitates the flow of information between layers.

[0061] At the top of the backbone network, a Feature Pyramid Network (FPN) is used as the neck module, through... The convolutional layer refines the upsampled feature maps. The neck module's role is to fuse the hierarchical features extracted from the backbone.

[0062] Feature Pyramid Networks (FPNs) are a key component in many computer vision tasks, particularly in object detection and semantic segmentation. In this invention, the core of FPN is the efficient integration of multi-scale features. By using feature pyramids of different scales, the network can capture both fine-grained and coarse-grained information. The original input image has a resolution of 608×608. After downsampling by the backbone, the resolution of the feature maps gradually decreases, resulting in features at different scales with resolutions of 152×152, 76×76, 38×38, and 19×19. These features at different scales typically contain more specific, low-level semantic information, such as texture and edges, while low-resolution feature maps contain more abstract, high-level semantic information, such as object category and scene category. By combining features at different scales, the feature pyramid upsamples the feature maps of different resolutions to 152×152, thus fusing features of different scales while maintaining consistent resolution. Coarse-grained features, such as the smallest feature map in the image above with a resolution of 19×19, contain features of a large region at each point and are therefore typically interpreted as high-level semantic information. Fine-grained information, on the other hand, corresponds to the largest feature map in the image above with a resolution of 152×152, and contains texture information of objects, such as edges, which are well represented.

[0063] In the upsampling process of FPN, a stacking operation is used to combine deep and shallow features. This allows the network to utilize high-level semantic and low-level texture information. The process first upsamples the deep feature map to the size of the shallow feature map using bilinear interpolation, and then... The convolutional layer refines the upsampled mapping. After concatenation with the shallow feature map, Convolutional layers refine channel features, and batch normalization and ReLU activation are applied in the latent layers to achieve better training stability and representational power. Deep features and shallow features are relative terms. Shallow features typically refer to features extracted in the first few layers of the neural network (closer to the input layer), while deep features refer to features extracted in the later layers (closer to the output layer). This invention achieves output resolutions of 152×152, 76×76, 38×38, and 19×19 from shallow to deep features. Shallow features provide fine-grained information because they focus on local details and specific visual elements in the image. Deep features provide coarse-grained information because they focus on the overall structure and semantics of the image, rather than specific details.

[0064] FPN typically has a top-down path and lateral connections. The top-down path upsamples coarse-scale features, and the lateral connections combine these features with bottom-up features of the same scale. Mathematically, , and These represent bottom-up, top-down upsampling, and output feature maps, respectively. (It's a pyramid level), the process is as follows:

[0065] ;

[0066] in It is an upsampling operation based on bilinear interpolation, namely:

[0067] ;

[0068] in, It is a learnable weight matrix (which can be an identity or a simple convolutional layer) used to adjust the bottom-up feature contributions. Output feature map. ,in and (s is the scaling factor, Make the output image 4 times smaller than the input image. Indicates the pyramid level as Top-down upsampling feature map, This represents the output feature map at pyramid level l. This represents the spatial dimension of the output feature map, where C is the number of channels. and These are the height and width of the feature map, respectively, after being scaled by the scaling factor s.

[0069] The role of FPN is to fuse feature maps of different sizes output by the backbone. For the part that implements lateral connectivity and bottom-up feature merging, upsampling and convolution are used to enlarge the smaller feature maps, and then they are stitched together to obtain feature maps of size 152×152.

[0070] In step S2, the LSK module processes the input feature map and enhances its representational power through a series of convolutions and attention mechanisms. Output feature map. (in this article) It was then converted into four branches: heatmap ( ), offset ( ), box parameters ( ) and radiation pattern ( Here, K represents the number of categories in the dataset, and s=4 represents the scaling factor. This transformation uses two [variables / factors]. It is implemented with a kernel and a 256-channel convolutional layer.

[0071] LSK takes a feature map as input and outputs a feature map, without changing the size of the feature map. It only modifies the numerical values ​​of the feature map, aiming to increase the response of values ​​in the target feature map and suppress relevant values ​​in the background feature map. After processing, decoding is required. The decoding head consists of a heatmap, offset, bounding box parameters, and orientation map.

[0072] The LSK module primarily adjusts the multi-scale features extracted by the FPN to select appropriate kernels for different targets. Subsequently, the integrated features are fed into the decoder to obtain predicted coordinates and target categories. The role of the LSK part is to integrate features, which are then processed by a detection head and divided into four parts: heatmap, offset, etc. Finally, the decoder decodes these four parts to obtain the final bounding box and predicted category. The LSK module consists of two sub-blocks: a Large Kernel Selection (LK selection) sub-block and a Feedforward Network (FFN) sub-block. In the LK selection sub-block of the LSK module, a large kernel convolution is first constructed by decomposing a large kernel convolution into a series of depthwise convolutions, which have increased kernel size and expansion rate. This process generates multiple features with different receptive fields (here, multiple features refer to two types of features obtained through 5×5 and 7×7 different receptive fields). Then, a spatial selection mechanism is employed to weight and fuse these features based on the input features (the previously obtained multiple features were derived from the feature pyramid, fused through the features from different stages of the backbone. The multiple features obtained in the LSK part are inputs from the feature pyramid feature maps of the previous stage. The multiple features from different receptive fields mentioned here refer to targets of different sizes, such as cars and boats, whose scales differ greatly. LSK adds surrounding information to these targets, taking into account the features of the surrounding water and land, hence the description of multiple features from different receptive fields). In large kernel convolution, the kernel size... Expansion rate and feel the wild The expansion formula is as follows:

[0073] ;

[0074] ;

[0075] Let's illustrate this with a concrete example. A large kernel convolution can be divided into several smaller kernel convolutions to achieve the same receptive field. For instance, using a large kernel convolution with k=23 and d=1 results in a receptive field RF=23. However, using two smaller kernel convolutions... but To obtain the same sense of field.

[0076] in, Let be the size of the i-th convolutional kernel, which represents the width and height of the convolutional kernel. Let be the dilation rate of the i-th convolution. The dilation rate controls the size of the receptive field of the convolution kernel, expanding the receptive field by inserting holes between kernel elements. Let be the receptive field of the i-th convolution. The receptive field refers to the range of the input feature map that the convolution kernel can cover, representing the size of the input image region that the convolution kernel can perceive.

[0077] This indicates that the size of the i-th convolutional kernel is not less than the size of the previous convolutional kernel. This shows that the size of the convolutional kernel is increasing, thus gradually expanding the receptive field.

[0078] This indicates that the dilation rate of the first convolution is 1, meaning there are no holes, and the receptive field is determined solely by the size of the convolution kernel. This means that the dilation rate of subsequent convolutions increases, but will not exceed the receptive field size of the previous convolution. This ensures that the receptive field expands gradually while avoiding receptive field overlap caused by excessive dilation.

[0079] This means that the receptive field of the i-th convolution is determined by the current kernel size, dilation rate, and the receptive field of the previous convolution. This indicates the additional receptive field expansion caused by the dilation of the current convolution kernel. This indicates a further expansion based on the receptive field of the previous convolution.

[0080] This design enables the LSK module to better simulate the detection of objects at different scales in remote sensing scenes and achieves state-of-the-art performance on multiple remote sensing tasks. Furthermore, this design allows the method of this invention to achieve improved mAP with a limited increase in the number of parameters.

[0081] Tables 1 and 2 compare the number of parameters with and without LSK. Taking the DOTA dataset as an example, FBA-R and FLBA-R represent the comparison of using and without LSK, respectively. With ResNet-101 as the backbone, the computational cost increases by 131.39 - 128.68 = 2.71 GB, and the number of parameters increases by 45.96 - 45.84 = 0.12 million. The performance improvement is 75.84 - 75.72 = 0.12%.

[0082] For the four branches—heatmap, offset, box parameters, and orientation pattern—the details are as follows:

[0083] Heat map ( The heatmap branch is used to predict targets at different locations in an image. The output of this branch is... ,in This refers to the number of categories in the dataset. Heatmaps are typically used to locate specific key points in an input image. This invention applies them to center point detection of targets in arbitrary orientations within aerial images. For the ground truth, given the center point of the rotated target bounding box... The invention is achieved by placing a 2D Gaussian around the center point. Generate ground truth heatmap ,in The frame size is adaptive.

[0084] The loss function formula is:

[0085] ;

[0086] in, and These are the actual values ​​and the predicted heatmap values. Index pixel position, It is the target number. and These are hyperparameters selected empirically. In a specific embodiment, the preferred hyperparameters are... =2, =4, where one pixel is actually the target, and the predicted value is close to the background, taking 0.01, that is: =0.01, =1, then the loss of this pixel is Finally, all pixel losses need to be accumulated and multiplied by (-1 / 15), where N is 15 categories. Figure 7 It is the result of the prediction. Figure 7 The left side of the image shows the input image, and the right side shows the output heatmap result. It can be seen that the heatmap tends to be 1 for the center of the target, while the heatmap predicts 0 for the background.

[0087] K is the number of categories in the dataset. middle, These are the coordinates of the center point of the rotated target box. These are the pixel coordinates on the heatmap. Let be the standard deviation of the Gaussian distribution. In the loss function, This represents the loss value for each objective. Used to adjust the weights of positive samples in the loss function. Used to adjust the weights of negative samples in the loss function.

[0088] Offset ( The Offset branch predicts the offset value for each target location. Its output is... In the inference stage of heatmap-based aerial image target detection, the predicted offset map is... Compensate for the difference between the floating-point center point and the integer center point.

[0089] For the true center point On the input image, the offset o between the scaling floating-point center point and the quantization center point is calculated as follows:

[0090] ;

[0091] Use smoothing Loss function optimization offset:

[0092] ;

[0093] Smoothing L1 loss is the most commonly used loss function in machine learning. For example, if two values ​​for one class are 0.3 and 0.5, then... Then, the offsets of all pixels in that class are summed, and the average is taken over all classes to obtain the loss function of that branch.

[0094] in, represents the coordinates of the ground truth center point on the input image. The offset 'o' is the offset between the scaled floating-point center point and the quantized center point. This is a floor operation, representing the largest integer less than or equal to the given value. The number of targets. This represents the actual offset of the k-th target. This is the predicted offset for the k-th target. The loss function for the offset branch.

[0095] Box parameters ( This branch predicts the bounding box parameters of the target bounding box. The output is... This invention uses box parameter mapping. Capture boundary-aware vectors (BBAVectors) representing the rotated bounding boxes (OBBs) of objects. Each object's bounding box is represented by five 2D vectors, for a total of ten channels. Given the center point of an object... and direction angle In the In a Cartesian coordinate system centered at , It is the distance from the center point to the bounding box, a vector. Calculation as follows

[0096] ;

[0097] The ground truth value of the bounding box parameter mapping is generated from the ground truth value of the bounding box, and its training loss is a smoothing loss.

[0098] ;

[0099] The purpose of setting the training loss function is primarily to make the final predicted value closer to the true situation. For example, if a box is predicted to be in the top left corner, while the true value is in the bottom right corner, a simple L1 loss is used to gradually shift the predicted value closer to the bottom right corner. The closer the predicted value is to the true value, the smaller the loss becomes, until the predicted value and the true value overlap. At this point, the loss reaches its minimum of 0.

[0100] The coordinates of the center point of the target. Let be the i-th boundary sensing vector. represents the true bounding box parameters for the k-th target. The parameters are the predicted bounding box parameters for the k-th target. The loss function is for the box parameter branch. The direction angle of the target. This is the distance from the center point c to the bounding box.

[0101] Direction chart ( ): The pattern branch predicts the direction of the target. Its output is This invention utilizes a radiation pattern Bounding boxes are divided into horizontal bounding boxes (HBB) and rotated bounding boxes (RBB). An HBB is defined as having a certain threshold... Orientation angle inside While RBB has an outer threshold. Applying the sigmoid function maps the predicted direction value to a range. The ground truth values ​​for the radiation pattern are set to HBB = 0 and RBB = 1. The training loss for the radiation pattern is the binary cross-entropy loss.

[0102] ;

[0103] If a rotated box is determined to be a horizontal box, the value is set to 0; otherwise, the value is set to 1.

[0104] The total loss of the network is the weighted sum of the losses of all output graphs, given by the following formula:

[0105] ;

[0106] in , and These are the weights for offset loss, box parameter loss, and pattern loss, respectively, and these weights can be adjusted empirically.

[0107] Let be the predicted direction value for the i-th pixel position. This represents the true orientation value at the i-th pixel position. If it's a horizontal bounding box (HBB), then... =0. If it is a rotated bounding box (RBB), then =1. This is the loss function for the heatmap branch. The loss function for the offset branch, The loss function is for the box parameter branch. This is the loss function for the pattern branch. , , These are weighting coefficients used to balance the losses of different branches.

[0108] The loss function is calculated and backpropagated using the GPU. The ultimate goal of this invention is to make the overall loss function approach 0, indicating that the network has fit the data.

[0109] This invention develops a method for rotating target detection in optical remote sensing images by fusing the FPN backbone and the BBAVectors head. The FPN backbone effectively solves the problem of multi-scale target detection and achieves robust feature extraction at different levels. Simultaneously, the Feature Integration Module (LSK) effectively separates the background and the target, and finally, the BBAVectors head accurately determines the rotating bounding box of the target.

[0110] The decoder is essentially a post-processor of the output from the four parts mentioned above.

[0111] The tensor shape obtained from the heatmap is 15×152×152, the tensor shape obtained from the offset is 2×152×152, the tensor shape obtained from the bounding box parameters is 10×152×152, and the tensor shape obtained from the orientation map is 1×152×152. The post-processing workflow is as follows: First, the heatmap tensors represent 15 categories, each a 152×152 resolution center point heatmap. The offset is used because the heatmap resolution is reduced by a factor of 4 (608 to 152), so a floating-point number, the offset, represents the offset of the center point, correcting for the target center point. The bounding box parameters have a shape of 10×152×152. The first 8 parameters are the x and y distances from the center point to the four boundaries, and the last two parameters are the width and height of the rotated bounding box. The orientation map has only one parameter; if the horizontal angle is less than 5°, it is treated as a horizontal bounding box; otherwise, a rotated bounding box is used for post-processing.

[0112] This invention was validated on the public datasets HRSC 2016 and DOTA V1.0.

[0113] HRSC 2016: Released in 2016, HRSC 2016 contains 1680 images, of which 1061 are effectively annotated. The dataset contains 2976 ship targets, divided into 436 training images, 181 validation images, and 444 test images. The spatial resolution of the images ranges from 0.4 to 2 meters, clearly capturing ship details. Image sizes range from 300×300 pixels to 1500×900 pixels. It uses a Rotated Bounding Box (OBB) annotation format, with rotated box annotations used for accurate ship orientation and attitude.

[0114] The DOTA V1.0 dataset comprises 2806 aerial images from different sensors and platforms, resulting in a rich variety of image features. Targets in these images exhibit a wide range of scales, orientations, and shapes, making detection algorithms extremely challenging. Image resolutions range from 800×800 pixels to 4000×4000 pixels. There are a total of 188,282 fully annotated instances belonging to 15 different categories: aircraft (PL), baseball field (BD), bridge (BR), ground athletics field (GTF), small vehicles (SV), large vehicles (LV), ships (SH), tennis court (TC), basketball court (BC), storage tank (ST), football field (SBF), roundabout (RA), port (HA), swimming pool (SP), and helicopter (HC). The dataset is divided into a training set of 1411 images, a validation set of 458 images, and a test set of 937 images.

[0115] Throughout the training process, this invention trains the network on two NVIDIA RTX 3090 TI GPUs with a batch size of 12. For the DOTA dataset, the network is trained for approximately 200 iterations to ensure sufficient convergence and learning of complex patterns in the data. Given the relatively small size and unique features of the HRSC2016 dataset, the network is trained for 100 iterations, achieving superior performance. Furthermore, this invention filters out images without targets, promoting better network convergence. The speed of the proposed network was measured on a single NVIDIA 3090 TI GPU using the HRSC 2016 and DOTA V1.0 datasets, demonstrating the computational efficiency of the method in real-world scenarios.

[0116] Quantitative Results: To verify the effectiveness of each component in the framework proposed in this invention, Tables 1 and 2 summarize the results of ablation experiments conducted on the DOTA V1.0 and HRSC2016 datasets. Specifically, FBA-R represents results without using the LSK module, FLBA-R represents results with the LSK module, and FLBA-D represents results using a lightweight backbone network.

[0117] Table 1: Experimental Results of DOTA V1.0

[0118] ;

[0119] Table 1 summarizes the results, in which this invention introduces ablation studies of different components of the DOTA dataset. The evaluation methods include the baseline method (BBAVectors), the FBA-R method of this invention (using ResNet+FPN), the FLBA-R method (using ResNet+FPN+LSK), and the FLBA-D method (using a decoupled network with FPN and LSK).

[0120] The baseline method using ResNet-101 achieved a mAP of 75.36%, a FLOP of 176.61 G, 53.43 million parameters, and an inference speed of 39.41 FPS. By integrating the FPN structure with the ResNet backbone, the FBA-R method of this invention exhibits competitive performance while reducing computational complexity and improving inference speed. For example, using ResNet-101, FBA-R achieved a mAP of 75.72%, reduced the FLOP to 128.68 G, 45.84 million parameters, and an inference speed of 43.14 FPS. This demonstrates that FPN effectively improves feature extraction efficiency without significantly reducing performance. Further integrating the LSK module into the FBA-R method yields the FLBA-R method, which achieves higher accuracy with a slight increase in computational cost. For example, using ResNet-101, FLBA-R achieved an mAP of 75.84%, a FLOP of 131.39 G, 45.96 million parameters, and an inference speed of 43.43 FPS. This demonstrates that the LSK module effectively improves the model's sensitivity to local features, thereby enhancing detection performance. The FLBA-D method, with a decoupled network as its backbone, achieves a balance between accuracy and computational efficiency. Using DecoupleNet D2, FLBA-D achieves 73.59% mAP, 81.11 G FLOPs, 9.49 million parameters, and an inference speed of 43.46 FPS. This highlights the effectiveness of decoupled networks in maintaining competitive performance while reducing model complexity.

[0121] The proposed methods FBA-R and FLBA-R consistently outperform baseline methods (BBAVectors) on different ResNet backbones. The integration of the FPN and LSK modules significantly improves detection accuracy while reducing computational complexity and increasing inference speed. Furthermore, the FLBA-D method, utilizing a decoupled network, achieves a good trade-off between accuracy and computational efficiency, making it suitable for resource-constrained environments. Overall, these results validate the effectiveness of the proposed methods in improving object detection on the DOTA dataset and highlight their potential for practical applications.

[0122] Table 2: HRSC2016 Experimental Results

[0123] ;

[0124] As shown in Table 2, ablation experiments on the HRSC2016 dataset demonstrate the effectiveness of the proposed method. Compared to the baseline method (ResNet-101 backbone, mAP 88.22%, FLOP 176.53 G, 53.43M parameters, 49.87 FPS), the FBA-R method (ResNet+FPN) of this invention achieves comparable or higher mAP values ​​while reducing computational complexity and improving inference speed. For example, FBA-R using ResNet-101 achieves 89.68% mAP, 128.59 G FLOP, 45.83M parameters, and 54.14 FPS. The FLBA-R method (ResNet+FPN+LSK) further improves performance, achieving 89.73% mAP using ResNet-101 while maintaining reasonable computational efficiency (FLOP 131.31 G, 45.95M parameters, 49.48 FPS). The FLBA-D method using DecoupleNet D2 achieves 86.05% mAP with only 9.49M parameters and 53.85 FPS, highlighting its efficiency in resource-constrained applications. Overall, the proposed FLBA-R and FLBA-D methods demonstrate significant improvements in accuracy, computational efficiency, and inference speed compared to baselines.

[0125] Qualitative Results: To demonstrate the advantages of the feature pyramid and feature integration architecture, this invention was compared with baseline methods. The proposed method uses the same decoder and post-processing steps as the baseline method, and the training process is identical, without data augmentation. As shown in Tables 1 and 2, the proposed method outperforms the baseline method by 1.91% and 0.48% on the HRSC 2016 and DOTA datasets, respectively. Compared to the U-Net structure used in the baseline method, the feature pyramid structure is more suitable for rotating object detection, offering not only faster inference speed and fewer parameters but also higher accuracy.

[0126] Figure 4 The qualitative results of the method proposed in this invention are compared in detail with those of baseline methods in various detection scenarios. Figure 4 Figure (a) illustrates that baseline methods cannot handle targets with low contrast or partial blurriness. In contrast, the method of the present invention successfully identifies these targets, demonstrating its robustness in capturing a wider range of target appearances. Figure 4 Figure (b) illustrates the error detection of the baseline method, which incorrectly detects a small car as a large car. This problem may stem from the loss of detail during feature extraction, limiting the method's ability to distinguish between different targets. However, the method of this invention can accurately distinguish between targets and non-targets, thus achieving more precise and reliable detection. Figure 4 Figure (c) illustrates that the baseline method produces redundant detections, where multiple bounding boxes are applied to the same target and background is mistakenly detected as different targets. This redundancy leads to confusion and inefficiency in subsequent analysis stages. The method of this invention effectively alleviates this problem, thereby improving the clarity and usability of the detection results. Figure 4 Figure (d) illustrates that in a rotating target detection scenario in aerial imagery, the baseline method randomly predicts the angle of targets with no orientation. The method of this invention tends to unify the prediction of such random angles for targets. Figure 4 Figure (e) illustrates that baseline methods typically assign low confidence scores to correct detections, which may lead to the rejection of valid detections during post-processing. The method of this invention assigns higher confidence scores to correct targets, thereby reducing the likelihood of false alarms and improving overall detection performance. In summary, the method of this invention demonstrates significant advantages over baseline methods in these scenarios. These improvements are crucial for enhancing the reliability and effectiveness of remote sensing applications.

[0127] This invention further provides a qualitative comparison of the visualization results on the DOTA V1.0 dataset. Figure 5Detailed information about the image is presented. The input image contains targets of different categories and scales (including ordinary targets, small targets, and targets with high aspect ratios). For small targets such as cars, medium-sized targets such as swimming pools, and targets with high aspect ratios such as harbors, the method can accurately predict the rotated bounding boxes aligned with the targets. This demonstrates the effectiveness of the method of the present invention in rotating target detection in optical remote sensing images.

[0128] exist Figure 6 The visualization of the feature maps is shown in the figure. This invention demonstrates the comparison between the image detection results and the feature maps of the same channel before and after processing by the LSK module. After processing by the LSK module, the feature maps generated at each stage are closer to the semantic meaning of the corresponding prediction parameters. Before processing by the LSK module, the features with center position information x and y are more scattered. After processing by the LSK module, the features become more obvious and more concentrated in the target center, ignoring background information. This phenomenon may be the result of the stepwise guidance of different feature kernels in the LSK module, further confirming the effectiveness of the proposed architecture method.

[0129] Any process or method described in the flowcharts of this invention or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, which can be implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device. The computer-readable medium can be any medium containing a program for storage, communication, propagation, or transmission for use by the execution system, apparatus, or device, including read-only memory, magnetic disks, or optical disks.

[0130] In the description of this specification, references to terms such as "embodiment," "example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, those skilled in the art can combine or combine the different embodiments or examples described in this specification and the features therein without causing contradiction.

[0131] While embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and alterations to the above embodiments within the scope of the present invention.

Claims

1. A remote sensing image target detection method based on feature pyramids and boundary-aware vectors, characterized in that, The method includes the following steps: S1. Feed the input image into the feature pyramid backbone to extract multi-scale features; S2. A dynamic selection mechanism is adopted to adaptively select appropriate kernels for different targets based on the extracted multi-scale features; S3. Input the comprehensive features into the decoder to obtain the predicted coordinates and target category, thereby obtaining the target's rotated bounding box; The backbone network of the method includes a backbone network, a neck module, an LSK module, and a detection head; the backbone network is implemented based on the ResNet architecture, with the ResNet101 backbone processing the RGB input image; the backbone network extracts hierarchical features by performing a series of convolutions, batch normalization, and ReLU activation layers on the input image; Step S2 includes: S2-1. The input feature map is processed by the large kernel selection module, and its receptive field is expanded by 5×5 and 7×7 convolutions to obtain information about the surrounding area of ​​the target. S2-2. A spatial selection mechanism is adopted to weight and fuse these features according to the input features, thereby generating a receptive field that expands multiple features; S2-3. The features obtained through the spatial selection mechanism are fused with the original features obtained through FPN to obtain the final comprehensive features; Mapping using box parameters Capture the boundary-aware vector representing the target's rotated bounding box, where, These are the height and width of the feature map, respectively. For box parameters.

2. The remote sensing image target detection method based on feature pyramids and boundary-aware vectors according to claim 1, characterized in that, Step S1 includes: S1-1. First, through the backbone network, four sets of feature maps of different scales are obtained through layer-by-layer convolution; S1-2. To address the issue of different feature maps, a feature pyramid structure is used to construct bottom-up and top-down feature fusion paths, fusing feature maps of different scales to generate a feature pyramid with multi-scale information. S1-3. The feature pyramid after FPN is stitched together to obtain a feature map of shape 152×152×256, where 152×152 is the image resolution and 256 is the number of feature maps.

3. The remote sensing image target detection method based on feature pyramids and boundary-aware vectors according to claim 1, characterized in that, The core of the feature pyramid network is to effectively integrate multi-scale features. Through feature pyramids of different scales, the network can capture both fine-grained and coarse-grained information. During the upsampling process, skip connections are used to combine deep and shallow features.

4. The remote sensing image target detection method based on feature pyramids and boundary-aware vectors according to claim 3, characterized in that, The feature pyramid network first upsamples the deep feature maps to the size of the shallow feature maps using bilinear interpolation, and then uses... Convolutional layers refine the upsampling mapping; After concatenation with shallow feature maps Convolutional layers refine channel features, and batch normalization and ReLU activation are applied in the latent layers.

5. The remote sensing image target detection method based on feature pyramids and boundary-aware vectors according to claim 3, characterized in that, The LSK module processes the input feature map and enhances its representation capabilities through a series of convolution and attention mechanisms.

6. The remote sensing image target detection method based on feature pyramids and boundary-aware vectors according to claim 5, characterized in that, The LSK module converts the feature map into four branches: heatmap, offset, bounding box parameters, and orientation map.

7. The remote sensing image target detection method based on feature pyramids and boundary-aware vectors according to claim 6, characterized in that, When the LSK module transforms the feature map, it uses two [functions / methods] with [specific characteristics]. The kernel and a 256-channel convolutional layer are implemented.

8. The remote sensing image target detection method based on feature pyramids and boundary-aware vectors according to claim 5, characterized in that, The LSK module consists of two sub-blocks: a large kernel selection sub-block and a feedforward network sub-block. In the large kernel selection sub-block of the LSK module, large kernel convolutions are first constructed by decomposing large kernel convolutions into a series of depthwise convolutions, which have increased kernel size and expansion rate. This process produces a variety of features with different receptive fields. Then, a spatial selection mechanism is used to weight and fuse these features based on the input features.

Images