Small target fine granularity detection method and system for remote sensing image

Abstract

The invention discloses a small-target fine-granularity detection method and system for a remote sensing image, belongs to the technical fields of computer vision, remote sensing image processing, deep learning and the like, and solves the problems that in the prior art, only the overall contour feature of a target is focused, the internal key discrimination area is neglected, and the detection accuracy is poor. And fine modeling of target structure details and discrimination regions is lacked, and fine-grained recognition is difficult to support. The method comprises the following steps: preprocessing an FAIR1M data set, and constructing an MAE pre-training data set and a YOLOv5 training data set; the improved YOLOv5 network is trained on the basis of the YOLOv5 training data set; after an MAE encoder is pre-trained in an unsupervised manner by adopting an MAE pre-training data set, a backbone network in the trained and improved YOLOv5 network is replaced, and then a YOLOv5 training data set is adopted for training after replacement, so that the final trained and improved YOLOv5 network is obtained to perform small-target fine-grained detection on a remote sensing image.

Description

Technical Field

[0001] A method and system for fine-grained detection of small targets in remote sensing images is disclosed, which belongs to the technical fields of computer vision, remote sensing image processing, and deep learning. Background Technology

[0002] With the development of deep learning, object detection methods based on convolutional neural networks (such as Faster R-CNN and YOLO series) have made significant progress in natural image scenes. Their success largely depends on large-scale labeled data (such as COCO and ImageNet) and rich local texture features. However, when these methods are directly applied to remote sensing images, the detection performance often drops significantly due to factors such as high image resolution, small target size, complex background, and large pose variations.

[0003] Ultimately, the unique characteristics of remote sensing scenes bring new challenges: ground targets (such as vehicles and ships) viewed from a height of thousands of meters usually occupy only tens of pixels and are easily disturbed by background textures, becoming submerged in complex backgrounds, causing their features to gradually degrade in deep networks, thus reducing the detection rate; small targets in the remote sensing field usually appear densely, making it difficult for traditional horizontal boxes to detect them effectively; their effective features gradually degrade during multiple downsampling in traditional convolutional neural networks, leading to a surge in false negatives.

[0004] Building upon this, the diverse types of small targets in remote sensing images, with minimal differences between categories, further highlight the challenge of fine-grained detection of small targets. Traditional target detection algorithms perform poorly under such data conditions, failing to meet the task's requirements for detection accuracy and fine-grained recognition capabilities.

[0005] In previous research, some works have attempted to introduce deep learning models to improve detection performance for small targets. Common methods include CNN-based detection frameworks and the introduction of feature pyramids. These methods, starting from multi-scale perception, rotation box regression, and attention mechanisms, have alleviated the problems of variable poses and strong background interference of small targets in remote sensing images to some extent, and have a certain effect on improving detection accuracy and adaptability.

[0006] However, these methods still have the following technical problems in fine-grained detection of small targets in remote sensing images:

[0007] 1. Focusing only on the overall outline features of the target while ignoring its key internal discrimination regions, it lacks detailed modeling of the target's structural details and discrimination regions, making it difficult to support fine-grained recognition;

[0008] 2. The model is not robust enough in complex backgrounds and occlusion interference. In scenarios with small target occlusion or complex background interference, the stability and accuracy of detection drop significantly.

[0009] 3. Due to its inherent local receptive field design, CNN-based detection frameworks have limitations in modeling long-distance dependencies and global contextual information, resulting in insufficient perception of small-sized target features and ultimately a significant decrease in the recall rate of small targets. Summary of the Invention

[0010] To address the problems mentioned above, the present invention aims to provide a method and system for fine-grained detection of small targets in remote sensing images. This solves the problem that existing technologies only focus on the overall contour features of the target, neglecting its key internal discrimination regions, lacking detailed modeling of the target's structural details and discrimination regions, and thus failing to support fine-grained recognition.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] A method for fine-grained detection of small targets in remote sensing images includes the following steps:

[0013] Step 1: Preprocess the FAIR1M dataset to construct the MAE pre-training dataset and the YOLOv5 training dataset;

[0014] Step 2: Train the improved YOLOv5 network based on the YOLOv5 training dataset. The improved YOLOv5 network uses a multi-task total loss function.

[0015] Step 3: After unsupervised pre-training the MAE encoder using the MAE pre-training dataset, replace the backbone network in the trained improved YOLOv5 network. After replacement, train the YOLOv5 training dataset again to obtain the final trained improved YOLOv5 network.

[0016] Step 4: Perform fine-grained detection of small targets in remote sensing images based on the trained and improved YOLOv5 network.

[0017] Furthermore, the specific steps of step 1 are as follows:

[0018] Step 1.1: Obtain the FAIR1M dataset, which includes TIFF image files and XML bounding box annotation files. The TIFF image files include TIFF images in the range of 1000-10000 pixels, and the XML bounding box annotation files include bounding boxes that annotate the targets in the TIFF images with non-horizontal bounding boxes.

[0019] Step 1.2: Use the ElementTree library in Python's XML library to parse the bounding boxes in the XML bounding box annotation file and obtain the coordinate values ​​of the five corner points. and target category labels;

[0020] Step 1.3: Determine the category label of each target. If the target category label is vehicle, the target is considered a small target. Convert the coordinates of the five corner points of the target into txt format and create a txt file. At the same time, store the corresponding TIFF image in the small target dataset. Otherwise, skip the label and do not convert it. The small target dataset is obtained.

[0021] Step 1.4: Crop the images in the small target dataset to 448×448 pixels to construct the MAE pre-training dataset. Crop the images in the small target dataset to 1024×1024 pixels, and simultaneously map the target boxes in the corresponding images before cropping to the cropped images to construct the YOLOv5 training dataset.

[0022] Furthermore, the improved YOLOv5 network in step 2 is based on the original YOLOv5 network. KFM modules are inserted between the second C3 layer in the backbone network and the second Conct layer in the Neck network, between the third C3 layer in the backbone network and the first Conct layer in the Neck network, and between the SPPF layer in the backbone network and the first C3 layer in the Neck network. A decoupled super-resolution distillation network is added to extract mid- and high-level semantic information from the multi-scale feature maps output by the second and fourth C3 layers in the backbone network. The YOLOv5 network includes the YOLOv5X network and the YOLOv5n network.

[0023] The specific implementation of the KFM module is as follows:

[0024] Multi-scale feature maps Divide the data into k sub-features evenly along the width W dimension. The sub-features are represented as follows:

[0025]

[0026] For each sub-feature Through 1×1 convolution mapping, followed by global average pooling and softmax normalization, we obtain the... Channel dimension importance factor of individual features The formula is:

[0027]

[0028] Based on the importance factor of the maximum channel dimension Corresponding index Select the most discriminative sub-feature block ;

[0029] Repeat the above operation on the sub-feature block X along the height H dimension to obtain the region importance factor. The maximum value b is taken as the final regional importance factor;

[0030] Adaptive weighted enhancement is applied to the original feature map Y based on the regional importance factor b to obtain the focused features. The formula is:

[0031]

[0032] in, The learnable scaling factor is represented by ⊙, which indicates element-wise multiplication.

[0033] The decoupled super-resolution distillation network includes two first 1×1 Conv layers that receive the multi-scale feature maps output by the second and fourth C3 layers in the backbone network and perform dimensionality reduction operations on them in sequence, two upsample layers that perform upsampling, a Conct layer that processes the output results of the two upsample layers in sequence, a decoder, and a second 1×1 Conv layer.

[0034] Furthermore, the multi-task total loss function in step 2 is based on the classification loss of the original YOLOv5 network. Regression loss And the improved YOLOL1 reconstruction loss The construction is performed using the following formula:

[0035]

[0036] The formula is:

[0037]

[0038] Where C is the number of image channels of the improved YOLOv5 network input, and H and W are the height and width of the input image. This is the reconstructed image vector output by the decoupled super-resolution distillation network. The input image is a normalized image vector. , , They are respectively , and Hyperparameters.

[0039] Furthermore, in step 3, after unsupervised pre-training the MAE encoder using the MAE pre-training dataset, the backbone network in the trained improved YOLOv5 network is replaced. This means replacing the backbone network with the trained MAE encoder. After the replacement, the first VIT layer in the MAE encoder is connected to the KFM module connected to the second Conct layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; the fifth VIT layer in the MAE encoder is connected to the KFM module connected to the first Conct layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; the SPPF layer in the MAE encoder is connected to the KFM module connected to the first C3 layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; and the second and sixth VIT layers in the MAE encoder are connected to the decoupled super-resolution distillation network.

[0040] A fine-grained detection system for small targets in remote sensing images includes:

[0041] Training set acquisition module: preprocesses the FAIR1M dataset to construct the MAE pre-training dataset and the YOLOv5 training dataset;

[0042] Improvement module: The improved YOLOv5 network is trained based on the YOLOv5 training dataset, where the improved YOLOv5 network adopts a multi-task total loss function;

[0043] After unsupervised pre-training of the MAE encoder using the MAE pre-training dataset, the backbone network in the trained improved YOLOv5 network is replaced. After replacement, the YOLOv5 training dataset is used for training again to obtain the final trained improved YOLOv5 network.

[0044] Detection module: Performs fine-grained detection of small targets in remote sensing images based on the trained and improved YOLOv5 network.

[0045] Furthermore, the specific implementation steps of the training set acquisition module are as follows:

[0046] Step 1.1: Obtain the FAIR1M dataset, which includes TIFF image files and XML bounding box annotation files. The TIFF image files include TIFF images in the range of 1000-10000 pixels, and the XML bounding box annotation files include bounding boxes that annotate the targets in the TIFF images with non-horizontal bounding boxes.

[0047] Step 1.2: Use the ElementTree library in Python's XML library to parse the bounding boxes in the XML bounding box annotation file and obtain the coordinate values ​​of the five corner points. and target category labels;

[0048] Step 1.3: Determine the category label of each target. If the target category label is vehicle, the target is considered a small target. Convert the coordinates of the five corner points of the target into txt format and create a txt file. At the same time, store the corresponding TIFF image in the small target dataset. Otherwise, skip the label and do not convert it. The small target dataset is obtained.

[0049] Step 1.4: Crop the images in the small target dataset to 448×448 pixels to construct the MAE pre-training dataset. Crop the images in the small target dataset to 1024×1024 pixels, and simultaneously map the target boxes in the corresponding images before cropping to the cropped images to construct the YOLOv5 training dataset.

[0050] Furthermore, the improved YOLOv5 network in the improved module is based on the original YOLOv5 network. KFM modules are inserted between the second C3 layer of the backbone network and the second Conct layer of the Neck network, between the third C3 layer of the backbone network and the first Conct layer of the Neck network, and between the SPPF layer of the backbone network and the first C3 layer of the Neck network. A decoupled super-resolution distillation network is added to extract mid- and high-level semantic information from the multi-scale feature maps output by the second and fourth C3 layers of the backbone network. The YOLOv5 network includes the YOLOv5X network and the YOLOv5n network.

[0051] The specific implementation of the KFM module is as follows:

[0052] Multi-scale feature maps Divide the data into k sub-features evenly along the width W dimension. The sub-features are represented as follows:

[0053]

[0054] For each sub-feature Through 1×1 convolution mapping, followed by global average pooling and softmax normalization, we obtain the... Channel dimension importance factor of individual features The formula is:

[0055]

[0056] Based on the importance factor of the maximum channel dimension Corresponding index Select the most discriminative sub-feature block ;

[0057] Repeat the above operation on the sub-feature block X along the height H dimension to obtain the region importance factor. The maximum value b is taken as the final regional importance factor;

[0058] Adaptive weighted enhancement is applied to the original feature map Y based on the regional importance factor b to obtain the focused features. The formula is:

[0059]

[0060] in, The learnable scaling factor is represented by ⊙, which indicates element-wise multiplication.

[0061] The decoupled super-resolution distillation network includes two first 1×1 Conv layers that receive the multi-scale feature maps output by the second and fourth C3 layers in the backbone network and perform dimensionality reduction operations on them in sequence, two upsample layers that perform upsampling, a Conct layer that processes the output results of the two upsample layers in sequence, a decoder, and a second 1×1 Conv layer.

[0062] Furthermore, the multi-task total loss function in the improved module is based on the classification loss of the original YOLOv5 network. Regression loss And the improved YOLOL1 reconstruction loss The construction is performed using the following formula:

[0063]

[0064] The formula is:

[0065]

[0066] Where C is the number of image channels of the improved YOLOv5 network input, and H and W are the height and width of the input image. This is the reconstructed image vector output by the decoupled super-resolution distillation network. The input image is a normalized image vector. , , They are respectively , and Hyperparameters.

[0067] Furthermore, in the improved module, after unsupervised pre-training the MAE encoder using the MAE pre-training dataset, the backbone network in the trained improved YOLOv5 network is replaced. Specifically, the backbone network is replaced with the trained MAE encoder. After the replacement, the first VIT layer in the MAE encoder is connected to the KFM module connected to the second Conct layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; the fifth VIT layer in the MAE encoder is connected to the KFM module connected to the first Conct layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; the SPPF layer in the MAE encoder is connected to the KFM module connected to the first C3 layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; and the second and sixth VIT layers in the MAE encoder are connected to the decoupled super-resolution distillation network.

[0068] Compared with the prior art, the beneficial effects of this invention are as follows:

[0069] This invention effectively utilizes the rich global information in remote sensing images to enhance the perception of key target areas and maintains classification accuracy and generalization ability under complex conditions, thus better adapting to the application needs of remote sensing images in complex environments. Specifically, it is manifested in the following ways:

[0070] I. This invention introduces an unsupervised pre-training strategy based on mask autoencoder (MAE) (i.e., a trained MAE encoder) into the trained improved YOLOv5 network. By applying a high proportion of masking to remote sensing images and performing image reconstruction tasks, the final improved YOLOv5 network learns global semantic and fine-grained structural features under strongly sparse input. This solves the problems of unclear representation of small remote sensing targets in complex backgrounds and weak model generalization ability due to limited training samples. It significantly enhances the feature abstraction ability of the detection backbone and cross-scene transfer performance.

[0071] Second, this invention proposes a decoupled super-resolution distillation module (DSRD), which utilizes multi-scale feature fusion and pixel-level reconstruction loss to transfer high-frequency details in high-resolution remote sensing images to the detection backbone. This effectively alleviates the problem of small target size and severe feature degradation after downsampling in remote sensing images. At the same time, by separating the lightweight structure (upsampling and 1×1 convolution) from the inference stage module, the detection accuracy and inference efficiency of the final improved YOLOv5 network are balanced.

[0072] Third, this invention constructs a Key Region Focusing Module (KFM Module). The KFM Module adopts a two-layer spatial block, attention evaluation and region enhancement mechanism to guide the final improved YOLOv5 network to dynamically focus on the target discrimination region, effectively suppressing background interference and mutual occlusion between dense targets, improving the accuracy and stability of small target detection, and is particularly suitable for complex scenes in remote sensing images with occlusion and inter-class interference.

[0073] Fourth, the improved YOLOv5 network in this invention introduces a joint optimization mechanism of reconstruction distillation loss, target detection loss and attention regularization term in the multi-task loss design, which solves the problem of limited gradient propagation under a single supervision signal, improves the synergy and convergence speed of model training, and is particularly suitable for the high-efficiency training needs under data-scarce conditions.

[0074] Fifth, this invention achieves seamless integration with the detection backbone of the trained improved YOLOv5 network by weight transfer and structural adaptation of the trained MAE encoder (that is, replacing the backbone network in the trained improved YOLOv5 network with the trained MAE encoder), and introduces parameter momentum update pairs to enhance the temporal stability of the feature space and the generalization ability of the model in the testing phase.

[0075] VI. This invention adopts a modular design approach, integrating the MAE encoder, DSRD reconstruction branch, and KFM module into the backbone detection network step by step as independent sub-modules (that is, it realizes the improvement of the original YOLOv5 network), which solves the problems of poor portability and difficult engineering deployment of existing methods, achieves flexible compatibility with mainstream detection frameworks and on-demand tailoring, and has good system scalability and practical engineering feasibility.

[0076] VII. This invention systematically solves the core problems in remote sensing small target detection, such as small size, weak texture, few annotations, and similar categories, by jointly introducing semantic modeling (MAE encoder), detail compensation (DSRD), and attention guidance (KFM module). It significantly improves the robustness and accuracy of fine-grained recognition of small targets and has significant technological innovation and industrial promotion value. Attached Figure Description

[0077] Figure 1 This is the final improved YOLOv5 network framework diagram in this invention. The backbone network is replaced with a pre-trained MAE encoder, which mainly consists of multiple VIT layers, each of which is a single-layer Transformer encoder. The DSRD module inputs are layers 2 and 6. The KFM module is inserted into the Neck part, with inputs to layers 1, 5, and SPPF of the backbone, respectively.

[0078] Figure 2 This is a framework diagram of the improved YOLOv5 network in this invention; the DSRD module is a model-aided task, mainly including two upsampling layers and a Concat and 1×1 convolutional channel layer, with the input being the 3rd and 7th layers of the backbone; the KFM module is inserted into the Neck part;

[0079] Figure 3 for Figure 1 and Figure 2 The diagram shows the structure of BottleNeck2 in the C3 layer, SPPF layer, and C3 layer. ConvBNSiLU represents the combination of convolutional layer (Conv), batch normalization layer (BN), and activation function (SiLU).

[0080] Figure 4 This is a schematic diagram of the Key Region Focusing Module (KFM module) in this invention, illustrating the spatial segmentation process of the feature map in both width and height directions, the calculation of importance weights, and the process of saliency enhancement.

[0081] Figure 5 This is a comparison curve showing the performance of the KFM module and DSRD module combined with YOLOv5x and the performance of YOLOv5x alone in this invention. In the figure, "AP50" is the average accuracy index when the IoU threshold is 0.5, "Training Progress" is the training progress, and "Model Performance (Last 50%)" is the model performance. The figure shows the comparison of the last 50% of the effect. The AP50 of YOLOv5x alone is 71.4%, while the AP50 of YOLOv5x combined with the KFM module in this invention is 72.3%, and the AP50 of YOLOv5x combined with the DSRD module is 72.5%.

[0082] Figure 6 The graphs show the performance of the KFM module and DSRD module combined with yolov5n and the performance of yolov5n alone. The AP50 of yolov5n alone is 34%. In this invention, the AP50 of yolov5n combined with the KFM module is 36.7%, and the AP50 of yolov5n combined with the DSRD module is 36%. Detailed Implementation

[0083] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments.

[0084] A method for fine-grained detection of small targets in remote sensing images includes the following steps:

[0085] Step 1: Preprocess the FAIR1M dataset to construct the MAE pre-training dataset and the YOLOv5 training dataset;

[0086] The specific steps are as follows:

[0087] Step 1.1: Obtain the FAIR1M dataset, which includes TIFF image files and XML bounding box annotation files. The TIFF image files include TIFF images in the range of 1000-10000 pixels, and the XML bounding box annotation files include bounding boxes that annotate the targets in the TIFF images with non-horizontal bounding boxes.

[0088] Step 1.2: Use the ElementTree library in Python's XML library to parse the bounding boxes in the XML bounding box annotation file and obtain the coordinate values ​​of the five corner points. and target category labels;

[0089] Step 1.3: Determine the category label of each target. If the target category label is vehicle, the target is considered a small target. Convert the coordinates of the five corner points of the target into txt format and create a txt file. At the same time, store the corresponding TIFF image in the small target dataset. Otherwise, skip the label and do not convert it. The small target dataset is obtained.

[0090] Step 1.4: Crop the images in the small target dataset to 448×448 pixels to construct the MAE pre-training dataset. Crop the images in the small target dataset to 1024×1024 pixels, and simultaneously map the target boxes in the corresponding images before cropping to the cropped images to construct the YOLOv5 training dataset.

[0091] Specifically, the data used in this invention is the FAIR1M remote sensing image target dataset V1.0 (Fine-grained Aerial Image Recognition). FAIR1M is a large-scale dataset for fine-grained target detection and recognition in remote sensing images. To meet the needs of practical applications, the images in the FAIR1M dataset were collected from different sensors and platforms, with a spatial resolution ranging from 0.3m to 0.8m.

[0092] This dataset is widely used for target detection and fine-grained recognition tasks in remote sensing images, containing over one million instances and over fifteen thousand images. With its high resolution, complex scenes, and multi-scale characteristics, it is a typical benchmark dataset for small target detection tasks.

[0093] All objects in the FAIR1M dataset are annotated with oriented bounding boxes for 5 categories and 37 subcategories. Each image ranges in size from 1000×1000 to 10,000×10,000 pixels and contains objects displaying various scales, orientations, and shapes. The dataset also features finely annotated, multi-angle distributed targets, covering hundreds of typical cities and towns worldwide, as well as common airports and ports. Over 80% of the data comes from my country's proprietary Gaofen series satellites. This invention utilizes the vehicle subset within the dataset.

[0094] In the dataset, each object is labeled with an oriented bounding box (OBB) containing a class label and coordinates. Image annotations are stored in XML files with the same filename, in the following format:

[0095] <possibleresult>

[0096] <name> Cargo Truck< / name>

[0097] < / possibleresult>

[0098] <points>

[0099] <point> 107.000031,935.000000< / point>

[0100] <point> 93.000031,945.000000< / point>

[0101] <point> 88.000031,938.000000< / point>

[0102] <point> 102.000031,928.000000< / point>

[0103] <point> 107.000031,935.000000< / point>

[0104] < / points>

[0105] Based on the original format annotation, this invention converts the target bounding box coordinates and category label into the following format: 107.000031 935.000000 93.000031 945.000000 88.000031 938.000000 102.000031928.000000 Cargo Truck 0; This annotation format integrates the rotated bounding box coordinates and category in the same line.

[0106] The dataset is cropped to sizes of 448 and 1024. For targets located on the boundary, their IOU (Intersection over Union) in the current cropped image is determined. If the IOU is greater than the threshold of 0.5, the target is assigned to the current cropped image and its bounding box is re-labeled. The missing part is padded with a pixel value of 127.

[0107] This invention employs YOLOv5 as the backbone structure for object detection and makes key improvements for rotating object detection tasks. By introducing a rotation bounding box parameter modeling mechanism (which refers to mapping coordinates to a self-made YOLOv5 training dataset when cropping images) and a rotation detection head (corresponding to the Head in the image), YOLOv5 can directly predict the rotation bounding box, thereby achieving efficient and accurate end-to-end rotating object detection. This improved YOLOv5 network can aggregate and locate multi-scale image features and has strong pose adaptability and structural stability. The overall implementation of YOLOv5 has good modular extensibility, providing a flexible implementation foundation for the multi-task object detection method proposed in this invention.

[0108] Step 2: Train the improved YOLOv5 network based on the YOLOv5 training dataset. The improved YOLOv5 network uses a multi-task total loss function.

[0109] The improved YOLOv5 network is based on the original YOLOv5 network. KFM modules are inserted between the second C3 layer of the backbone network and the second Conct layer of the Neck network, between the third C3 layer of the backbone network and the first Conct layer of the Neck network, and between the SPPF layer of the backbone network and the first C3 layer of the Neck network. A decoupled super-resolution distillation network is added to extract mid- and high-level semantic information from the multi-scale feature maps output by the second and fourth C3 layers of the backbone network. The YOLOv5 network includes the YOLOv5X network and the YOLOv5n network.

[0110] The specific implementation of the KFM module is as follows:

[0111] Multi-scale feature maps Divide the data into k sub-features evenly along the width W dimension. The sub-features are represented as follows:

[0112]

[0113] For each sub-feature Through 1×1 convolution mapping, followed by global average pooling and softmax normalization, we obtain the... Channel dimension importance factor of individual features The formula is:

[0114]

[0115] Based on the importance factor of the maximum channel dimension Corresponding index Select the most discriminative sub-feature block ;

[0116] Repeat the above operation on the sub-feature block X along the height H dimension to obtain the region importance factor. The maximum value b is taken as the final regional importance factor;

[0117] The original feature map Y is adaptively weighted and enhanced based on the regional importance factor b to obtain the focused features, as shown in the formula:

[0118]

[0119] in, The learnable scaling factor is represented by ⊙, which indicates element-wise multiplication.

[0120] Focused features Inputting the subsequent multi-scale fusion and detection head enables dynamic focusing and background suppression of key areas;

[0121] The decoupled super-resolution distillation network (DSRD module) includes two first 1×1 Conv layers that receive the multi-scale feature maps output by the second and fourth C3 layers in the backbone network and perform dimensionality reduction operations on them in sequence, two upsample layers that perform upsampling, a Conct layer that processes the output results of the two upsample layers in sequence, a decoder, and a second 1×1 Conv layer.

[0122] The size of the output feature map of the second C3 layer is The feature map size output by the fourth C3 layer is , , .

[0123] in, , The number of input vector channels, , The height of the input vector. , The width of the input vector.

[0124] In the decoupled super-resolution distillation network, the number of channels is reduced to 64 using 1×1 convolutions.

[0125]

[0126] In order to fuse features at the same resolution, and The high-level tensors in the model are upsampled four times to medium resolution using bilinear interpolation, and finally concatenated along the channel dimension to obtain the fused features. The decoder generates a detail compensation image magnified 4 times:

[0127]

[0128] In EDSR mode, the decoder utilizes residual connections and convolutions to achieve super-resolution reconstruction; in deconvolution mode, three transposed convolutions progressively increase the resolution to the original image size, and finally, a 1×1 convolution is applied. The super-resolution feature map output from the DSRD module branch is then compared with the original input image. Calculate the L1 reconstruction loss at the pixel level.

[0129] The multi-task total loss function is based on the classification loss of the original YOLOv5 network. Regression loss And the improved YOLOL1 reconstruction loss The construction is performed using the following formula:

[0130]

[0131] The formula is:

[0132]

[0133] Where C is the number of image channels of the improved YOLOv5 network input, and H and W are the height and width of the input image. This is the reconstructed image vector output by the decoupled super-resolution distillation network. The input image is a normalized image vector. , , They are respectively , and Hyperparameters.

[0134] Step 3: After unsupervised pre-training the MAE encoder using the MAE pre-training dataset, replace the backbone network in the trained improved YOLOv5 network. After replacement, train the YOLOv5 training dataset again to obtain the final trained improved YOLOv5 network.

[0135] After unsupervised pre-training the MAE encoder using the MAE pre-training dataset, the backbone network in the trained improved YOLOv5 network is replaced. Specifically, the backbone network is replaced with the trained MAE encoder. After the replacement, the first VIT layer in the MAE encoder is connected to the KFM module connected to the second Conct layer in the Neck network via a 1×1 convolutional layer or a linear mapping layer; the fifth VIT layer in the MAE encoder is connected to the KFM module connected to the first Conct layer in the Neck network via a 1×1 convolutional layer or a linear mapping layer; the SPPF layer in the MAE encoder is connected to the KFM module connected to the first C3 layer in the Neck network via a 1×1 convolutional layer or a linear mapping layer; and the second and sixth VIT layers in the MAE encoder are connected to a decoupled super-resolution distillation network.

[0136] The Backbone network in the improved YOLOv5 network after training is completely replaced with the pre-trained MAE encoder to utilize the global semantic and fine-grained structural features learned during the unsupervised pre-training phase. Simultaneously, the pre-trained weights of the MAE encoder are directly transferred to the corresponding layers of the improved YOLOv5 network after training. The MAE encoder pre-trained weights are in pth format, which are mapped to pt format while maintaining parameter consistency across network layers. The feature maps output by the pre-trained MAE encoder are dimensional and channel-mapped, and then converted to an input format acceptable to the Neck network using 1×1 convolutional layers or linear mapping layers. That is, during the forward propagation of the MAE encoder, for each input batch... (Where B is the batch size, H×W is the flattened spatial dimension, and C is the number of hidden channels) Generate serialized token features. To adapt to the Neck network, these need to be reshaped and projected into standard convolutional feature maps:

[0137] .

[0138] First, a 1×1 convolutional layer or an equivalent linear mapping layer is used to map the number of channels C to the expected number of channels C′. Then, the flattened spatial dimensions are reshaped. Format.

[0139] After replacing it with a pre-trained MAE encoder, the output feature vectors can be directly obtained from its multiple stages. Since the features used in different layers of the DSRD module are already unified in spatial resolution, no additional upsampling is required. Only channel mapping and reshaping of the selected vectors are needed, followed by concatenation along the channel dimensions and then super-resolution reconstruction.

[0140] The mapped features can be directly input into the improved YOLOv5 network's Neck network and interconnected with the original convolutional blocks to achieve multi-scale feature fusion and decoupled super-resolution learning. This allows for full utilization of the global semantic and fine-grained structural information learned by MAE while maintaining the original YOLOv5 detection process.

[0141] A differentiated learning rate strategy was adopted, using a lower learning rate for fine-tuning the encoder layer during transfer learning, and a higher learning rate for training the original YOLOv5 network and the added modules, in order to balance the preservation of pre-training information and adaptability to downstream tasks.

[0142] During the post-transfer training process, a momentum update mechanism is introduced, which uses a sliding coefficient α to dynamically update the network parameters in order to smooth parameter changes and enhance feature stability.

[0143] After each training round, new model parameters are generated based on the current training results. The prototype is then updated using an exponential moving average formula:

[0144]

[0145] in, This represents the parameter in round t. This represents the model parameters from the previous round. These are the updated parameters obtained from this round of training. It is the smoothing coefficient.

[0146] The model built on the MAE backbone and integrating the KFM and DSRD modules (i.e., the final improved YOLOv5 network) was validated. During inference, the super-resolution module inference was decoupled—in the testing phase, the super-resolution branch in the DSRD module was removed, and only statically reparameterized lightweight convolutional kernels were used for fast upsampling and detail restoration of multi-scale feature maps to ensure inference efficiency. The final class scoring metric Map was defined as follows:

[0147]

[0148] Where C is the total number of categories in the test set. The average precision of class c on the test set is calculated with an IoU threshold of 0.5:0.95 (step size 0.05).

[0149] Step 4: Perform fine-grained detection of small targets in remote sensing images based on the trained and improved YOLOv5 network.

[0150] To verify the effectiveness of the method of this invention, model training and algorithm testing were both completed in the following hardware and software environment:

[0151] To verify the effectiveness of the method of this invention, model training and algorithm testing were both completed in the following hardware and software environment:

[0152] Hardware environment: CPU is AMD Ryzen Threadripper PRO 5975WX, 32 cores, 3.60GHz; GPU is NVIDIA GeForce RTX 4090 × 5;

[0153] Operating system: Linux;

[0154] Numpy: 1.23.0;

[0155] Python environment: Python 2.9.0;

[0156] Deep learning frameworks:

[0157] torch: 1.10.1+cu113

[0158] torchvision: 0.11.2+cu113;

[0159] opencv-python: 4.10.0.84;

[0160] In summary, this invention systematically solves key technical challenges in remote sensing images, such as small target size, weak detail, complex background, and minimal category differences, by jointly introducing semantic modeling (global pre-training based on the MAE encoder), detail enhancement (super-resolution distillation based on the decoupled super-resolution distillation network (DSRD module)), and region awareness (key region focusing based on the KFM model). This innovative method integrates an improved YOLOv5 network with the MAE encoder across architectures for small target detection. The DSRD module guides distillation-based reconstruction of mid-to-high-level features, and the KFM module further enhances region saliency, resulting in a remote sensing small target detection system with end-to-end trainability and semantic-detail co-optimization capabilities. This method effectively improves the structural representation and region awareness capabilities of small targets while balancing model training efficiency and inference performance, significantly enhancing the accuracy, stability, and cross-scene adaptability of remote sensing small target detection. It possesses good engineering practicality and scalability, demonstrating significant technological advancement and industrial transformation potential.

[0161] The above are merely representative embodiments among the many specific applications of this invention, and do not constitute any limitation on the scope of protection of this invention. All technical solutions formed by transformation or equivalent substitution fall within the scope of protection of this invention.

Claims

1. A small target fine-grained detection method for remote sensing images, characterized in that, Includes the following steps: Step 1: Preprocess the FAIR1M dataset to construct the MAE pre-training dataset and the YOLOv5 training dataset; Step 2: Train the improved YOLOv5 network based on the YOLOv5 training dataset. The improved YOLOv5 network uses a multi-task total loss function. Step 3: After unsupervised pre-training the MAE encoder using the MAE pre-training dataset, replace the backbone network in the trained improved YOLOv5 network. After replacement, train the YOLOv5 training dataset again to obtain the final trained improved YOLOv5 network. Step 4: Perform fine-grained detection of small targets in remote sensing images based on the trained and improved YOLOv5 network.

2. The method for small target fine-grained detection of remote sensing images according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1: Obtain the FAIR1M dataset, which includes TIFF image files and XML bounding box annotation files. The TIFF image files include TIFF images in the range of 1000-10000 pixels, and the XML bounding box annotation files include bounding boxes that annotate the targets in the TIFF images with non-horizontal bounding boxes. Step 1.2: Use the ElementTree library in Python's XML library to parse the bounding boxes in the XML bounding box annotation file and obtain the coordinate values ​​of the five corner points. and target category labels; Step 1.3: Determine the category label of each target. If the target category label is vehicle, the target is considered a small target. Convert the coordinates of the five corner points of the target into txt format and create a txt file. At the same time, store the corresponding TIFF image in the small target dataset. Otherwise, skip the label and do not convert it. The small target dataset is obtained. Step 1.4: Crop the images in the small target dataset to 448×448 pixels to construct the MAE pre-training dataset. Crop the images in the small target dataset to 1024×1024 pixels, and simultaneously map the target boxes in the corresponding images before cropping to the cropped images to construct the YOLOv5 training dataset.

3. The method for fine-grained detection of small targets in remote sensing images according to claim 1, characterized in that, The improved YOLOv5 network in step 2 is based on the original YOLOv5 network. KFM modules are inserted between the second C3 layer of the backbone network and the second Conct layer of the Neck network, between the third C3 layer of the backbone network and the first Conct layer of the Neck network, and between the SPPF layer of the backbone network and the first C3 layer of the Neck network. A decoupled super-resolution distillation network is added to extract mid- and high-level semantic information from the multi-scale feature maps output by the second and fourth C3 layers of the backbone network. The YOLOv5 network includes the YOLOv5X network and the YOLOv5n network. The specific implementation of the KFM module is as follows: Multi-scale feature maps Divide the data into k sub-features evenly along the width W dimension. The sub-features are represented as follows: For each sub-feature Through 1×1 convolution mapping, followed by global average pooling and softmax normalization, we obtain the... Channel dimension importance factor of individual features The formula is: Based on the importance factor of the maximum channel dimension Corresponding index Select the most discriminative sub-feature block ; Repeat the above operation on the sub-feature block X along the height H dimension to obtain the region importance factor. The maximum value b is taken as the final regional importance factor; Adaptive weighted enhancement is applied to the original feature map Y based on the regional importance factor b to obtain the focused features. The formula is: in, The learnable scaling factor is represented by ⊙, which indicates element-wise multiplication. The decoupled super-resolution distillation network includes two first 1×1 Conv layers that receive the multi-scale feature maps output by the second and fourth C3 layers in the backbone network and perform dimensionality reduction operations on them in sequence, two upsample layers that perform upsampling, a Conct layer that processes the output results of the two upsample layers in sequence, a decoder, and a second 1×1 Conv layer.

4. The method for fine-grained detection of small targets in remote sensing images according to claim 3, characterized in that: The multi-task total loss function in step 2 is based on the classification loss of the original YOLOv5 network. Regression loss And the improved YOLOL1 reconstruction loss The construction is performed using the following formula: The formula is: Where C is the number of image channels of the improved YOLOv5 network input, and H and W are the height and width of the input image. This is the reconstructed image vector output by the decoupled super-resolution distillation network. The input image is a normalized image vector. , , They are respectively , and Hyperparameters.

5. A method for fine-grained detection of small targets in remote sensing images according to claim 3, characterized in that, In step 3, after unsupervised pre-training the MAE encoder using the MAE pre-training dataset, the backbone network in the trained improved YOLOv5 network is replaced. This means replacing the backbone network with the trained MAE encoder. After the replacement, the first VIT layer in the MAE encoder is connected to the KFM module connected to the second Conct layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; the fifth VIT layer in the MAE encoder is connected to the KFM module connected to the first Conct layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; the SPPF layer in the MAE encoder is connected to the KFM module connected to the first C3 layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; and the second and sixth VIT layers in the MAE encoder are connected to the decoupled super-resolution distillation network.

6. A fine-grained detection system for small targets in remote sensing images, characterized in that, include: Training set acquisition module: preprocesses the FAIR1M dataset to construct the MAE pre-training dataset and the YOLOv5 training dataset; Improvement module: The improved YOLOv5 network is trained based on the YOLOv5 training dataset, where the improved YOLOv5 network adopts a multi-task total loss function; After unsupervised pre-training of the MAE encoder using the MAE pre-training dataset, the backbone network in the trained improved YOLOv5 network is replaced. After replacement, the YOLOv5 training dataset is used for training again to obtain the final trained improved YOLOv5 network. Detection module: Performs fine-grained detection of small targets in remote sensing images based on the trained and improved YOLOv5 network.

7. A fine-grained detection system for small targets in remote sensing images according to claim 6, characterized in that, The specific implementation steps of the training set acquisition module are as follows: Step 1.1: Obtain the FAIR1M dataset, which includes TIFF image files and XML bounding box annotation files. The TIFF image files include TIFF images in the range of 1000-10000 pixels, and the XML bounding box annotation files include bounding boxes that annotate the targets in the TIFF images with non-horizontal bounding boxes. Step 1.2: Use the ElementTree library in Python's XML library to parse the bounding boxes in the XML bounding box annotation file and obtain the coordinate values ​​of the five corner points. and target category labels; Step 1.3: Determine the category label of each target. If the target category label is vehicle, the target is considered a small target. Convert the coordinates of the five corner points of the target into txt format and create a txt file. At the same time, store the corresponding TIFF image in the small target dataset. Otherwise, skip the label and do not convert it. The small target dataset is obtained. Step 1.4: Crop the images in the small target dataset to 448×448 pixels to construct the MAE pre-training dataset. Crop the images in the small target dataset to 1024×1024 pixels, and simultaneously map the target boxes in the corresponding images before cropping to the cropped images to construct the YOLOv5 training dataset.

8. A fine-grained detection system for small targets in remote sensing images according to claim 7, characterized in that, The improved YOLOv5 network in the improved module is based on the original YOLOv5 network. KFM modules are inserted between the second C3 layer of the backbone network and the second Conct layer of the Neck network, between the third C3 layer of the backbone network and the first Conct layer of the Neck network, and between the SPPF layer of the backbone network and the first C3 layer of the Neck network. A decoupled super-resolution distillation network is added to extract mid- and high-level semantic information from the multi-scale feature maps output by the second and fourth C3 layers of the backbone network. The YOLOv5 network includes the YOLOv5X network and the YOLOv5n network. The specific implementation of the KFM module is as follows: Multi-scale feature maps Divide the data into k sub-features evenly along the width W dimension. The sub-features are represented as follows: For each sub-feature Through 1×1 convolution mapping, followed by global average pooling and softmax normalization, we obtain the... Channel dimension importance factor of individual features The formula is: Based on the importance factor of the maximum channel dimension Corresponding index Select the most discriminative sub-feature block ; Repeat the above operation on the sub-feature block X along the height H dimension to obtain the region importance factor. The maximum value b is taken as the final regional importance factor; Adaptive weighted enhancement is applied to the original feature map Y based on the regional importance factor b to obtain the focused features. The formula is: in, The learnable scaling factor is represented by ⊙, which indicates element-wise multiplication. The decoupled super-resolution distillation network includes two first 1×1 Conv layers that receive the multi-scale feature maps output by the second and fourth C3 layers in the backbone network and perform dimensionality reduction operations on them in sequence, two upsample layers that perform upsampling, a Conct layer that processes the output results of the two upsample layers in sequence, a decoder, and a second 1×1 Conv layer.

9. A fine-grained detection system for small targets in remote sensing images according to claim 8, characterized in that: The multi-task total loss function in the improved module is based on the classification loss of the original YOLOv5 network. Regression loss And the improved YOLOL1 reconstruction loss The construction is performed using the following formula: The formula is: Where C is the number of image channels of the improved YOLOv5 network input, and H and W are the height and width of the input image. This is the reconstructed image vector output by the decoupled super-resolution distillation network. The input image is a normalized image vector. , , They are respectively , and Hyperparameters.

10. A fine-grained detection system for small targets in remote sensing images according to claim 8, characterized in that, The improved module uses the MAE pre-trained dataset to unsupervised pre-train the MAE encoder, then replaces the backbone network in the trained improved YOLOv5 network with the trained MAE encoder. After the replacement, the first VIT layer in the MAE encoder is connected to the KFM module connected to the second Conct layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; the fifth VIT layer in the MAE encoder is connected to the KFM module connected to the first Conct layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; the SPPF layer in the MAE encoder is connected to the KFM module connected to the first C3 layer in the Neck network through a 1×1 convolutional layer or a linear mapping layer; and the second and sixth VIT layers in the MAE encoder are connected to the decoupled super-resolution distillation network.

Images