A fine-grained target detection method based on structural prior and multi-scale feature fusion

Abstract

This invention proposes a fine-grained target detection method based on structural prior and multi-scale feature fusion. The method includes: Step 1, acquiring a fine-grained remote sensing image dataset; Step 2, building an improved Oriented R-CNN network model, using a ResNet50 backbone network and a Feature Pyramid Network (FPN) to obtain multi-scale features of the image, and using the Canny edge detection algorithm to perform edge detection on the input image to obtain edge detection results. The edge detection results are fused with the multi-scale features to obtain fused features, which are used as input to the Oriented RPN. Based on the localization results output by the Oriented RPN, a feature extraction algorithm is used to extract features from the fused features, and the feature extraction results are input into the head classification network and the head localization network; Step 3, training the target detection model built in Step 2 using the remote sensing dataset; Step 4, using the trained remote sensing image target detection model to detect remote sensing images to obtain the localization and classification of targets in the remote sensing images.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a fine-grained target detection method based on structural prior and multi-scale feature fusion. Background Technology

[0002] With the development and expansion of deep learning, deep learning models have made progress in various fields, including remote sensing. The rapid development of remote sensing technology has greatly improved the quantity and quality of remote sensing images. These images can be used to describe various objects on the Earth's surface, such as airports, airplanes, and cars. This naturally creates a demand for intelligent Earth observation through the automatic analysis and understanding of satellite or aerial images.

[0003] Object detection is a fundamental task in computer vision, primarily involving locating regions in an image where specific objects appear and determining the object's category. It can generally be divided into two main categories: two-stage object detectors and single-stage object detectors. These two types of detectors can be distinguished by whether they contain a region of interest (ROI) module. Two-stage object detectors offer higher accuracy in localization and object recognition, while single-stage object detectors achieve higher inference speed.

[0004] Despite significant successes with natural images, it is difficult to directly transfer deep learning-based object detection methods to optical remote sensing images. The differences between remote sensing images and natural scene images are substantial. Remote sensing images typically capture rooftop information of geospatial objects, while natural scene images typically capture the outline information of objects. Therefore, object detectors learned from natural scene images are not easily transferred to remote sensing images. The challenges of remote sensing object detection are as follows:

[0005] (1) Large aspect ratio of the target: The aspect ratio of the target to be detected in the remote sensing image is very large. For such targets, the offset of the center point of the prediction box or the deviation of the prediction angle will greatly affect the accuracy of the positioning.

[0006] (2) Densely arranged targets: The distribution of targets in remote sensing images is relatively dense, which causes the network to be interfered with by other targets when extracting the features of a certain target.

[0007] (3) Inconsistency between localization and classification: Dense labeling strategy can cause the detector to output multiple bounding boxes for the same target in an image. Therefore, after the detector outputs the results, NMS post-processing is required to remove the redundant bounding boxes. However, due to the inconsistency between localization and classification, the classification results are not completely positively correlated with the accuracy of localization. The NMS operation may leave low-quality bounding boxes. Summary of the Invention

[0008] Purpose of the Invention: The purpose of this invention is to overcome the shortcomings of the prior art and propose a fine-grained target detection method based on structural prior and multi-scale feature fusion. This invention offers solutions to problems such as inconsistencies between loss functions and standard metrics, dense target distribution in remote sensing images, and inconsistencies between localization and classification.

[0009] Inconsistent Loss Metrics: Intersection over Union (IoU) is a standard metric for horizontally rotating object detection; however, there is inconsistency between this metric and regression loss (e.g., ln-norms), meaning the loss value fails to reflect accuracy deviations. This inconsistency is further amplified in rotating object detection due to the introduction of angle parameters.

[0010] The feature enhancement module based on structural priors addresses the issue that the dense distribution of targets in remote sensing images can cause interference from other targets when the network extracts features from a particular target. Figure 1 As shown, this invention proposes a feature enhancement module based on structural priors. It uses Gaussian filtering to extract target edges, downsamples the extracted edge features, and concatenates them with depth features to achieve feature enhancement, thereby guiding the network's prediction results.

[0011] Inconsistency between localization and classification: Due to the inconsistency between localization and classification, the NMS operation may filter out high-quality detection boxes while retaining low-quality ones, and the two-stage object detector ignores the aspect ratio information of the target when extracting candidate box features. The classification loss function proposed in this invention solves the above problems by adding prior aspect ratio information.

[0012] Technical Solution: To address the aforementioned technical problems, this invention proposes a fine-grained target detection method based on structural priors and multi-scale feature fusion. This method includes the following steps:

[0013] Step 1: Obtain the fine-grained remote sensing image dataset Fair1m;

[0014] Step 2: Construct an improved Oriented R-CNN network model. Multi-scale features of the image are obtained using the backbone network ResNet50 and the Feature Pyramid Network (FPN). The Canny edge detection algorithm is then used to perform edge detection on the input image. The edge detection results are fused with the multi-scale features to obtain fused features, which serve as input to the Oriented RPN. Based on the localization results output by the Oriented RPN, the RoI Align Rotated feature extraction algorithm is used to extract features from the fused features. The extracted features are then input into the head network, where a VisionTransformer-based multi-scale feature fusion classification network replaces the classification branch in the head network. Finally, the head network obtains the localization and classification detection results.

[0015] Step 3: Use the remote sensing dataset processed in Step 1 to train the target detection model built in Step 2;

[0016] Step 4: Use the trained remote sensing image target detection model to detect targets in the remote sensing image to obtain the location and classification of targets in the remote sensing image.

[0017] Furthermore, the method for step 1 is as follows:

[0018] Step 1-1: Obtain the fine-grained remote sensing image dataset Fair1m. The annotation information of each image in the dataset includes the coordinates of the four vertices of the target to be detected and the category name. The image size of this dataset is from 1000*1000 pixels to 10000*10000 pixels. The Fair1m dataset is cut into multiple images of a fixed size of 1024*1024, and the cut images are set to have a 50% overlap area.

[0019] Steps 1-2 involve modifying the dataset annotation information files accordingly and deleting any annotation files that do not meet the requirements. Unacceptable cases include: empty annotation targets and annotation targets exceeding the boundaries.

[0020] Steps 1-3: Divide the processed Fair1m dataset into three files: train, val, and test. The training set, validation set, and test set are randomly divided in a 7:1:2 ratio.

[0021] Furthermore, the specific process of Canny edge detection described in step 2 is as follows:

[0022] (3.1) Convert the three-channel RGB image processed in step 1 into a single-channel grayscale image G;

[0023] (3.2) The processed grayscale image G is smoothed by the Gaussian function to obtain the processed image g, where σ in the Gaussian filter function is set to 1.5 and the Gaussian kernel size is set to 3*3;

[0024] (3.3) Use the first-order partial derivative finite difference to differentiate the gray value of image g, and calculate the gradient of each pixel of image g to obtain the gradient image g'.

[0025] (3.4) Non-maximum suppression is applied to the gradient image g' to obtain the result g′. nms ;

[0026] (3.5) Use the high and low thresholds in the Canny edge detection algorithm to evaluate the gradient image g′ nms Detection is performed, and edges are connected to obtain image edges, resulting in a binary image feature B with 1 channel.

[0027] Furthermore, the method for setting the high and low thresholds in step (3.5) is as follows:

[0028] (4.1) Set the high threshold h to 150 and the low threshold l to 50;

[0029] (4.2) Input the remote sensing images obtained from the preprocessing in step 1 into the backbone network ResNet50 and the feature pyramid FPN to obtain multi-scale features f1-f5. The sizes of the multi-scale features are 256*256*256, 256*128*128, 256*64*64, 256*32*32, and 256*16*16, respectively.

[0030] (4.3) The multi-scale features f1-f5 are downsampled by 4, 8, 16, 32, and 64 times respectively, and then summed to obtain the total feature activation f of the feature pyramid. sum ∈R C*W*H Channel C = 256, width W = 1024, length H = 1024;

[0031] (4.4) Along feature f sum Given the channel dimension C, the feature f∈R is obtained by summing the values ​​of feature points with the same index. W*H ;

[0032] (4.5) Perform SoftMax calculation on feature f and multiply the result by W*H to obtain feature f′∈R. W*H ;

[0033] (4.6) Based on the eigenvalues ​​of f′, the adjustment ratios for the high and low thresholds at each point are obtained. The calculation formula is as follows:

[0034]

[0035] In Formula 1, f′ij This represents the value of the element in the i-th row and j-th column of f′. ij *h is set as the high threshold for that point, and F ij *l is set as the low threshold for that point.

[0036] Furthermore, the feature fusion process described in step 2 is as follows:

[0037] (5.1) The binary image features B obtained by edge detection are downsampled by 4, 8, 16, 32 and 64 times respectively to obtain multi-scale binary image features b1-b5, with feature map sizes of 1*256*256, 1*128*128, 1*64*64, 1*32*32 and 1*16*16 respectively.

[0038] (5.2) Concat the outputs f1-f5 of the feature pyramid with the features b1-b5 of the multi-scale binary image respectively to complete feature fusion and obtain features F1-F5. The feature map sizes are 257*256*256, 257*128*128, 257*64*64, 257*32*32, and 257*16*16 respectively. Replace f1-f5 with F1-F5 as the input of the Oriented RPN module and the feature extraction algorithm RoI Align Rotated.

[0039] Furthermore, in step 2, the structure of the multi-scale feature fusion classification network based on Vision Transformer is as follows: The multi-scale feature fusion classification network based on Vision Transformer replaces the Embedding module in the Vision Transformer model with a multi-scale feature concatenation module, and adjusts the number of Transformer Encoders in the Transformer Encoder module to 3.

[0040] Furthermore, the stitching process of the multi-scale feature stitching module is as follows:

[0041] (7.1) Target multi-scale feature extraction:

[0042] Inputting the fused features F1-F5 into the Oriented RPN yields the localization result LOC∈R. N*5 N is the number of localization results generated by Oriented RPN, based on the fact that for each localization result loc∈R 5The feature extraction algorithm RoI Align Rotated is used to extract features from the fused features F1-F3. The RoI Align Rotated algorithm divides the candidate regions of the fused features F1-F3 into 7*7, 5*5, and 3*3 cells respectively to obtain RoI features R1-R3 with feature sizes of 7*7*256, 5*5*256, and 3*3*256. The first two dimensions of the three-dimensional features R1-R3 are flattened to obtain the target multi-scale features R1'-R3' with feature sizes of 49*256, 25*256, and 9*256 respectively.

[0043] (7.2) Position embedding calculation:

[0044] Based on each location result loc∈(x,y,w,h,θ), where (x,y) are the center point coordinates, w is the width of the rotated bounding box, h is the length of the rotated bounding box, and θ is the angle between the rotated bounding box and the positive x-axis, calculate the location information loc corresponding to each cell of the RoI feature. i The calculation formula is as follows:

[0045]

[0046] In Formula 2, g*g represents the number of cells, and i represents the i-th term of the first dimension of the target multi-scale feature;

[0047] The location information Loc1∈R in each dimension of R1'-R3' is obtained using Formula 2. 49*5 Loc2∈R 25*5 Loc3∈R 9*5 By setting two fully connected layers fc1 and fc2, the positioning information is encoded. fc1 has an input dimension of 5 and an output dimension of 256, while fc2 has an input dimension of 256 and an output dimension of 256. The encoded position information is Loc′1∈R. 49*256 Loc′2∈R 25*256 Loc′3∈R 9*256 The calculation formula is as follows:

[0048] Loc′ i∈{1，2，3} =fc2(fc1(Loc i (3)

[0049] (7.3) Feature splicing:

[0050] Adding the location information Locl′-Loc3′ to the target multi-scale feature R1′-R3′ yields V1∈R 49*256 V2∈R 25 *256 V3∈R 9*256 The calculation formula is as follows:

[0051] V i∈{1，2，3} =R′ i +Loc′ i (4)

[0052] Concatenate V1, V2, V3 with the class token introduced in Vision Transformer, where classtoken∈R 1*256 Finally, the input V of the Transformer Encoder module is obtained. Cat ∈R 84*256 .

[0053] Furthermore, the formula for the head network classification loss function used during training in step 3 is as follows:

[0054]

[0055]

[0056]

[0057] In formula (5), G is the Gaussian distribution function, the parameter μc is the mean aspect ratio of the c-th target, and σ c Let μ be the standard deviation of the aspect ratio of the c-th type of target. c With σ c All values ​​are derived from the annotation information of the training and validation sets in the Fairlm remote sensing target detection dataset. n For the nth target aspect ratio output by Oriented RPN, when the target aspect ratio is r n hour, Fit the probability of the target to the c-th class, and compare it with the predicted value of the c-th class of the n-th target x. n，c Multiplying them together yields the optimized category prediction result. In formula (6), z n，c Let c be the true value of the c-th class of the nth target, where c is the total number of classes. In formula (7), N is the total number of targets used to calculate the loss function. Formula (7) is used to calculate the average cross-entropy loss function of N targets.

[0058] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0059] First, the aspect ratio of the targets to be detected in remote sensing images is very large. For such targets, the offset of the predicted bounding box center point or the deviation of the predicted angle will greatly affect the localization accuracy. However, this huge deviation in accuracy cannot be reflected in existing localization loss functions. This project proposes a reweighted target detection localization loss function based on the IOU index, which can effectively solve the problem of inconsistency between the loss and the standard metric.

[0060] Second, the feature enhancement module based on structural prior introduced in this invention can solve the problem that the network is interfered with by other targets when extracting features of a certain target due to the dense distribution of targets in remote sensing images, and can guide the network's result prediction.

[0061] Third, the classification loss function proposed in this invention introduces localization-related information into the classification score by adding prior aspect ratio information, which alleviates the problem of inconsistency between localization and classification, and also solves the problem of ignoring the aspect ratio information of the target when the two-stage target detector extracts candidate box features. Attached Figure Description

[0062] Figure 1 This is a flowchart of the method of the present invention.

[0063] Figure 2 This is the network structure diagram of the present invention. Detailed Implementation

[0064] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0065] This invention proposes a fine-grained target detection method based on structural prior and multi-scale feature fusion, which includes the following steps:

[0066] Step 1: Obtain the fine-grained remote sensing image dataset Fair1m;

[0067] Step 2: Build an improved Oriented R-CNN network model. The backbone network ResNet50 and the Feature Pyramid Network FPN obtain multi-scale features of the image. The Canny edge detection algorithm is used to perform edge detection on the input image to obtain edge detection results. The edge detection results are fused with the multi-scale features to obtain fused features, which are used as input to the Oriented RPN. Based on the localization results output by the Oriented RPN, the feature extraction algorithm RoI Align Rotated is used to extract features from the fused features. The feature extraction results are input into the head network. The classification branch in the head network is replaced by a multi-scale feature fusion classification network based on VisionTransformer. Finally, the head network obtains the localization and classification detection results.

[0068] Step 3: Use the remote sensing dataset processed in Step 1 to train the target detection model built in Step 2;

[0069] Step 4: Use the trained remote sensing image target detection model to detect targets in the remote sensing image to obtain the location and classification of targets in the remote sensing image.

[0070] Furthermore, the method for step 1 is as follows:

[0071] Step 1-1: Obtain the fine-grained remote sensing image dataset Fair1m. The annotation information of each image in the dataset includes the coordinates of the four vertices of the target to be detected and the category name. The image size of this dataset is from 1000*1000 pixels to 10000*10000 pixels. The Fair1m dataset is cut into multiple images of a fixed size of 1024*1024, and the cut images are set to have a 50% overlap area.

[0072] Steps 1-2 involve modifying the dataset annotation information files accordingly and deleting any annotation files that do not meet the requirements. Unacceptable cases include: empty annotation targets and annotation targets exceeding the boundaries.

[0073] Steps 1-3: Divide the processed Fairlm dataset into three files: train, val, and test. The training set, validation set, and test set are randomly divided in a 7:1:2 ratio.

[0074] Furthermore, the specific process of Canny edge detection described in step 2 is as follows:

[0075] (3.1) Convert the three-channel RGB image processed in step 1 into a single-channel grayscale image G;

[0076] (3.2) The processed grayscale image G is smoothed by the Gaussian function to obtain the processed image g, where σ in the Gaussian filter function is set to 1.5 and the Gaussian kernel size is set to 3*3;

[0077] (3.3) Use the first-order partial derivative finite difference to differentiate the gray value of image g, and calculate the gradient of each pixel of image g to obtain the gradient image g′;

[0078] (3.4) Non-maximum suppression is applied to the gradient image g' to obtain the result g′. nms ;

[0079] (3.5) Use the high and low thresholds in the Canny edge detection algorithm to evaluate the gradient image g′ nms Detection is performed, and edges are connected to obtain image edges, resulting in a binary image feature B with 1 channel.

[0080] Furthermore, the method for setting the high and low thresholds in step (3.5) is as follows:

[0081] (4.1) Set the high threshold h to 150 and the low threshold l to 50;

[0082] (4.2) Input the remote sensing images obtained from the preprocessing in step 1 into the backbone network ResNet50 and the feature pyramid FPN to obtain multi-scale features f1-f5. The sizes of the multi-scale features are 256*256*256, 256*128*128, 256*64*64, 256*32*32, and 256*16*16, respectively.

[0083] (4.3) The multi-scale features f1-f5 are downsampled by 4, 8, 16, 32, and 64 times respectively, and then summed to obtain the total feature activation f of the feature pyramid. sum ∈R C*W*H Channel C = 256, width W = 1024, length H = 1024;

[0084] (4.4) Along feature f sum Given the channel dimension C, the feature f∈R is obtained by summing the values ​​of feature points with the same index. W*H ;

[0085] (4.5) Perform SoftMax calculation on feature f and multiply the result by W. * H obtains the feature f′∈R W*H ;

[0086] (4.6) Based on the eigenvalues ​​of f′, the adjustment ratios for the high and low thresholds at each point are obtained. The calculation formula is as follows:

[0087]

[0088] In Formula 1, f′ ij This represents the value of the element in the i-th row and j-th column of f′. ij *h is set as the high threshold for that point, and F ij *l is set as the low threshold for that point.

[0089] Furthermore, the feature fusion process described in step 2 is as follows:

[0090] (5.1) The binary image features B obtained by edge detection are downsampled by 4, 8, 16, 32 and 64 times respectively to obtain multi-scale binary image features b1-b5, with feature map sizes of 1*256*256, 1*128*128, 1*64*64, 1*32*32 and 1*16*16 respectively.

[0091] (5.2) Concat the outputs f1-f5 of the feature pyramid with the features b1-b5 of the multi-scale binary image respectively to complete feature fusion and obtain features F1-F5. The feature map sizes are 257*256*256, 257*128*128, 257*64*64, 257*32*32, and 257*16*16 respectively. Replace f1-f5 with F1-F5 as the input of the Oriented RPN module and the feature extraction algorithm Rol Align Rotated.

[0092] Furthermore, in step 2, the structure of the multi-scale feature fusion classification network based on Vision Transformer is as follows: The multi-scale feature fusion classification network based on Vision Transformer replaces the Embedding module in the Vision Transformer model with a multi-scale feature concatenation module, and adjusts the number of Transformer Encoders in the Transformer Encoder module to 3.

[0093] Furthermore, the stitching process of the multi-scale feature stitching module is as follows:

[0094] (7.1) Target multi-scale feature extraction:

[0095] Inputting the fused features F1-F5 into the Oriented RPN yields the localization result LOC∈R. N*5 N is the number of localization results generated by Oriented RPN, based on the fact that for each localization result loc∈R 5The feature extraction algorithm Rol Align Rotated is used to extract features from the fused features F1-F3. The Rol Align Rotated algorithm divides the candidate regions of the fused features F1-F3 into 7*7, 5*5, and 3*3 cells respectively to obtain Rol features R1-R3 with feature sizes of 7*7*256, 5*5*256, and 3*3*256. The first two dimensions of the three-dimensional features R1-R3 are flattened to obtain the target multi-scale features R1′-R3′ with feature sizes of 49*256, 25*256, and 9*256 respectively.

[0096] (7.2) Position embedding calculation:

[0097] Based on each location result loc∈(x,y,w,h,θ), where (x,y) are the coordinates of the center point, w is the width of the rotated bounding box, h is the length of the rotated bounding box, and θ is the angle between the rotated bounding box and the positive x-axis, calculate the location information loc corresponding to each cell of the Rol feature. i The calculation formula is as follows:

[0098]

[0099] In Formula 2, g*g represents the number of cells, and i represents the i-th term of the first dimension of the target multi-scale feature;

[0100] The location information Loc1∈R in each dimension of R1′-R3′ is obtained using Formula 2. 49*5 Loc2∈R 25*5 Loc3∈R 9*5 By setting two fully connected layers fc1 and fc2, the positioning information is encoded. fc1 has an input dimension of 5 and an output dimension of 256, while fc2 has an input dimension of 256 and an output dimension of 256. The encoded position information is Loc′1∈R. 49*256 Loc′2∈R 25*256 Loc′3∈R 9*256 The calculation formula is as follows:

[0101] Loc′ i∈{1，2，3} =fc2(fc1(Loc i (3)

[0102] (7.3) Feature splicing:

[0103] Adding the location information Loc1′-Loc3′ to the target multi-scale features R1′-R3′ yields V1∈R 49*256 V2∈R 25 *256 V3∈R 9*256 The calculation formula is as follows:

[0104] V i∈{1，2，3} =R′ i +Loc′ i (4)

[0105] Concatenate V1, V2, V3 with the class token introduced in Vision Transformer, where classtoken∈R 1*256 Finally, the input V of the Transformer Encoder module is obtained. Cat ∈R 84*256 .

[0106] Furthermore, the formula for the head network classification loss function used during training in step 3 is as follows:

[0107]

[0108]

[0109]

[0110] In formula (5), G is the Gaussian distribution function, the parameter μc is the mean aspect ratio of the c-th target, and σ c Let μ be the standard deviation of the aspect ratio of the c-th type of target. c With σ c All results were derived from statistical analysis of the annotation information of the training and validation sets in the Fairlm remote sensing target detection dataset. n For the nth target aspect ratio output by Oriented RPN, when the target aspect ratio is r n hour, Fit the probability of the target to the c-th class, and compare it with the predicted value of the c-th class of the n-th target x. n，c Multiplying them together yields the optimized category prediction result. In formula (6), z n，c Let be the true value of the c-th class of the nth target, where C is the total number of classes. In formula (7), N is the total number of targets used to calculate the loss function. Formula (7) is used to calculate the average cross-entropy loss function of N targets.

[0111] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A fine-grained object detection method based on structure prior and multi-scale feature fusion, characterized in that, The method comprises the following steps: Step 1, obtaining a fine-grained remote sensing image dataset Fair1m, and pre-processing data in the dataset; Step 2, building a network model of the improved Oriented R-CNN, obtaining multi-scale features of a picture from a backbone network ResNet50 and a feature pyramid network FPN, and performing edge detection on the input picture by using a Canny edge detection algorithm to obtain an edge detection result, fusing the edge detection result and the multi-scale features to obtain fused features as input of the Oriented RPN, extracting features from the fused features by using a feature extraction algorithm RoI Align Rotated according to a positioning result output by the Oriented RPN, and inputting the feature extraction result into a head network, wherein a multi-scale feature fusion classification network based on VisionTransformer is used to replace a classification branch in the head network, and a positioning and classification detection result is obtained from the head network; Step 3, training the target detection model built in step 2 by using the remote sensing dataset pre-processed in step 1; Step 4, detecting the remote sensing image by using the trained remote sensing image target detection model to obtain positioning and classification of targets in the remote sensing image; The feature fusion process in step 2 is as follows: (5.1) respectively down-sampling the binary image features B obtained by edge detection by 4, 8, 16, 32 and 64 times to obtain multi-scale binary image features b1-b5, and the feature map sizes are 1*256*256, 1*128*128, 1*64*64, 1*32*32 and 1*16*16 respectively; (5.2) respectively performing Concat splicing on the output f1-f5 of the feature pyramid and the multi-scale binary image features b1-b5 to obtain features F1-F5, and the feature map sizes are 257*256*256, 257*128*128, 257*64*64, 257*32*32 and 257*16*16 respectively, and F1-F5 are used to replace f1-f5 as input of the Oriented RPN module and the feature extraction algorithm RoI Align Rotated; In step 2, the structure of the multi-scale feature fusion classification network based on Vision Transformer is as follows: the multi-scale feature fusion classification network based on Vision Transformer replaces the Embedding module in the Vision Transformer model with a multi-scale feature splicing module, and adjusts the number of Transformer Encoder modules in the Transformer Encoder module to 3; The splicing process of the multi-scale feature splicing module is as follows: (7.1) target multi-scale feature extraction: The fused features F1-F5 are input into the Oriented RPN to obtain a positioning result LOC∈R N*5 , N is the number of positioning results generated by the Oriented RPN, and according to each positioning result loc∈R 5 , a feature extraction algorithm RoI Align Rotated is used to perform feature extraction from the fused features F1-F3, the feature extraction algorithm RoI Align Rotated divides the candidate regions of the fused features F1-F3 into 7*7, 5*5 and 3*3 cells respectively, to obtain RoI features R1-R3, and the feature sizes are 7*7*256, 5*5*256 and 3*3*256 respectively; the first two dimensions of the three-dimensional features R1-R3 are flattened to obtain target multi-scale features R1'-R3', and the feature sizes are 49*256, 25*256 and 9*256 respectively; (7.2) position embedding calculation: According to each positioning result loc∈(x,y,w,h,θ), (x,y) is the center point coordinate, w is the width of the rotated frame, h is the length of the rotated frame, and θ is the included angle between the rotated frame and the positive direction of the x axis. The positioning information Loc corresponding to each cell of the RoI feature is calculated i The calculation formula is as follows: In formula 2, g*g is the number of unit cells, and i is the i-th item of the first dimension of the target multi-scale feature; The location information Loc1∈R in each dimension of R1'-R3' is obtained through formula (2). 49*5 Loc2∈R 25*5 Loc3∈R 9*5 By setting two fully connected layers fc1 and fc2, the positioning information is encoded. fc1 has an input dimension of 5 and an output dimension of 256, while fc2 has an input dimension of 256 and an output dimension of 256. The encoded position information is Loc′1∈R. 49*256 Loc′2∈R 25 *256 Loc′3∈R 9*256 The calculation formula is as follows: Loc′ i∈{1,2,3} = fc2(fc1(Loc i )) (3) (7.3) feature splicing: Adding the position information Loc1'-Loc3' to the target multi-scale features R1'-R3' obtains V1∈R 49*256 , V2∈R 25*256 , V3∈R 9*256 , and the calculation formula is as follows: V i∈{1,2,3} = R' i + Loc' i (4) V1, V2, V3 are spliced with the class token introduced in the Vision Transformer, wherein the class token e R 1*256 Finally, the input V of the Transformer Encoder module is obtained Cat e R 84*256 . 2.The fine-grained object detection method based on structural prior and multi-scale feature fusion according to claim 1, characterized in that, The method of step 1 is as follows: Step 1-1, obtain a fine-grained remote sensing image dataset Fair1m, the annotation information of each picture in the dataset includes four vertex coordinates of the target to be detected and a category name, the picture size of the dataset is 1000*1000 pixels to 10000*10000 pixels, cut the Fair1m dataset, cut each picture in the dataset into multiple 1024*1024 fixed size pictures, and set an overlapping area of 50% between the cut pictures; Step 1-2, modify the dataset annotation information file accordingly, and delete the annotation files that do not meet the requirements, which include: empty annotation target, and annotation target out of bounds; Step 1-3, divide the processed Fair1m dataset into train, val and test files, the training set, the validation set and the test set are randomly divided according to the ratio of 7:1:

2. 3.The fine-grained object detection method based on structural prior and multi-scale feature fusion of claim 1, characterized in that, The specific process of Canny edge detection in step 2 is as follows: (3.1) convert the three-channel RGB picture processed in step 1 into a single-channel gray picture G1; (3.2) smooth the processed gray picture G1 by using a Gaussian function to obtain a processed picture g, wherein the sigma of the Gaussian filter function is set to 1.5, and the Gaussian kernel size is set to 3*3; (3.3) use a first-order partial derivative finite difference to derive the gray value of the picture g, and calculate the gradient of each pixel point of the picture g to obtain a gradient image g'; (3.4) Non-maximum suppression on the gradient image g' to get the result g' nms ; (3.5) Using the high and low thresholds in the Canny edge detection algorithm on the gradient image g' nms The detection is performed and the edges are connected to obtain the image edges, resulting in a binary image feature B with a channel number of 1.

4. The fine-grained object detection method based on structural prior and multi-scale feature fusion according to claim 3, characterized in that, The high and low threshold setting method in step (3.5) is as follows: (4.1) set the initial high threshold h to 150 and the low threshold l to 50; (4.2) input the remote sensing picture obtained by pre-processing in step 1 into the backbone network ResNet50 and the feature pyramid FPN to obtain multi-scale features f1-f5, the sizes of the multi-scale features are 256*256*256, 256*128*128, 256*64*64, 256*32*32 and 256*16*16 respectively; (4.3) Downsample the multi-scale features f1-f5 by 4, 8, 16, 32, 64 times respectively and add them to get the feature pyramid feature activation total f sum ∈R C*W*H , channel C = 256, width W = 1024, length H = 1024; (4.4) Along feature fs sum Given the channel dimension C, the feature f∈R is obtained by summing the values ​​of feature points with the same index. W*H ; (4.5) SoftMax is performed on the feature f and the result is multiplied by W*H to obtain the feature f' e R W*H ; (4.6) according to the characteristic value of f', the rate of adjustment of the high and low thresholds of each point is obtained, and the calculation formula is as follows: In Formula 1, f′ ij This represents the value of the element in the i-th row and j-th column of f′. ij *h is set as the high threshold for that point, and F ij *l is set as the low threshold for that point.

5. The fine-grained object detection method based on structural prior and multi-scale feature fusion according to claim 1, characterized in that, The head network classification loss function formula used in step 3 is as follows: In formula (5), G is a Gaussian distribution function, μ c is the average aspect ratio of the cth target, and σ c is the standard deviation of the aspect ratio of the cth target, μ c and σ c are obtained by statistical information of the training set and the validation set in the remote sensing target detection data set Fair1m, and r n is the aspect ratio of the nth target output by the Oriented RPN, when the aspect ratio of the target is r n , the probability of fitting the target as the cth class is multiplied by the cth class prediction value x n,c of the nth target to obtain the optimized class prediction result, in formula (6), z n,c is the true value of the cth class of the nth target, and C is the total number of classes, in formula (7), N is the total number of targets used to calculate the loss function, and formula (7) is used to calculate the average cross-entropy loss function of N targets.

Images