A remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling

Abstract

This invention proposes a remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling mask generation, comprising: Step 1, acquiring a remote sensing image instance segmentation dataset; Step 2, constructing an improved Mask R-CNN network model, obtaining initial multi-scale feature maps using a backbone network (swin transformer) and a feature pyramid (FPN), and fusing them using a two-dimensional weighting module to obtain a fused multi-scale feature map. The fused feature map is then processed by a Region Proposal Network (RPN) to obtain the coarse location of each instance. The RoIAlign algorithm at multiple resolutions is used to extract feature maps of interest at the target locations in both the initial and fused feature maps. The fused feature maps of interest are used for classification and bounding box detection. Cross-attention is calculated between the feature maps of interest before and after fusion to predict the segmentation result; Step 3, training the instance segmentation model constructed in Step 2 using the remote sensing dataset; Step 4, using the trained model to detect remote sensing images.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling. Background Technology

[0002] With the rapid development of deep learning technology, it is being applied to an increasing number of fields. In the field of remote sensing image processing, there is a significant demand for image processing, therefore, deep learning-based computer vision technology is widely used in various tasks within this field. In the interpretation of remote sensing images, some tasks involve detecting and labeling specific objects in the image, such as airplanes, vehicles, and buildings, to determine their locations; other tasks involve determining the shape of the target, i.e., segmentation. For projects that require completing both parts simultaneously, instance segmentation techniques are needed to process the image.

[0003] Instance segmentation is a fundamental task in computer vision. Its main objective is to scan an image for a given category of objects, locate all objects, and label the location, category, and shape of each object. Instance segmentation models are primarily divided into two frameworks: two-stage and single-stage. Two-stage frameworks generally offer higher accuracy; however, they often require more GPU memory to run. Single-stage frameworks, on the other hand, while slightly lower in segmentation accuracy, are more computationally efficient, requiring less memory and offering faster inference speeds.

[0004] In everyday image processing, existing instance segmentation frameworks are relatively well-established. However, applying these frameworks to the remote sensing field still presents some challenges. This is because the distribution of targets in remote sensing images differs from that in everyday scenes. These differences can impact model performance, leading to decreased accuracy. The characteristics of remote sensing target distribution are as follows:

[0005] (1) Dense distribution of targets: In remote sensing images, there are often many objects of the same type distributed in a single image, such as vehicles in a parking lot or buildings in a city. The large number of similar targets places certain demands on the performance of the model. The model needs to be able to detect and represent these targets and should not abandon the detection results due to limitations on the number of targets.

[0006] (2) Significant differences in target scale: In the task of remote sensing image interpretation, different categories of targets may have significant scale differences. For example, vehicles, aircraft, and buildings appear to have vastly different sizes in the image. In addition, for some categories, the size differences between different targets of the same category may also be large. For example, different buildings may have vastly different floor areas, but they all need to be identified as one building. Therefore, their sizes appearing in the image will also be very different.

[0007] (3) The shapes of the targets are relatively complex: In remote sensing images, the shapes of different categories of objects may vary greatly. For example, various vehicles such as airplanes and cars have their own corresponding shapes and rich details that need to be detected. In addition, various buildings may also have complex and different shapes. Therefore, the model needs to be able to detect and represent these complex shapes. Summary of the Invention

[0008] Summary of the Invention: The purpose of this invention is to overcome the aforementioned problems and supplement and improve upon the shortcomings of existing models. This paper proposes a remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling mask generation. This invention addresses the problem of weak target feature representation ability of feature pyramids in existing models and the rich detail of targets in remote sensing images, proposing corresponding solutions.

[0009] Technical Solution: To address the aforementioned problems, this invention proposes a remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling. This method includes the following steps:

[0010] Step 1: Obtain the remote sensing image instance segmentation dataset iSAID and preprocess the data in the dataset;

[0011] Step 2: Build the Mask R-CNN network model. The initial multi-scale feature map of the image is obtained by the backbone network, the Swing Transformer, and the Feature Pyramid Network (FPN). The feature map is weighted using a two-dimensional weighting module to obtain a fused multi-scale feature map. The fused multi-scale feature map is fed into the Region Proposal Network (RPN) to generate a coarse bounding box for initial target location determination. The RoIAlign network at different resolutions is used to extract the feature map of interest at the target location in the initial feature map and the fused feature map. The feature of interest from the fused feature map is used for target classification and bounding box prediction. Cross-attention is calculated between the feature maps of interest before and after fusion to obtain the final feature map for predicting the segmentation result.

[0012] Step 3: Use the remote sensing image instance segmentation data obtained in Step 1 to train the model built in Step 2;

[0013] Step 4: Use the trained remote sensing image instance segmentation model to segment the remote sensing image and determine the location, size, and shape of the targets within it.

[0014] Furthermore, the method for step 1 is as follows:

[0015] Step 1-1: Obtain the remote sensing image instance segmentation dataset iSAID. This dataset contains 2806 remote sensing images, with 15 categories and 655451 instance objects.

[0016] Steps 1-2: Crop the image, cutting the original image into 800*800 images with 200 pixels of overlap between adjacent images; modify the instance information in the dataset to fit the image size.

[0017] Steps 1-3 involve re-dividing the cut images and labels to obtain training and validation sets for model training.

[0018] Furthermore, in step 2, the specific process of the two-dimensional weighting module is as follows:

[0019] (3.1) Sort the initial multi-scale feature maps generated in step 2 from largest to smallest according to their length or width;

[0020] (3.2) The first four feature maps in the sorting results are designated as feature maps 1, 2, 3 and 4 respectively. The initial feature maps 1 and 2, 2 and 3, and 3 and 4 are grouped into a pair of feature map groups and processed one pair at a time. For each feature map group, the first feature map is designated as feat1 and the second as feat2. Based on the size of each dimension of feat2, they are designated as b, c, h and w respectively. Here, b is the batch size of the feature map, c is the number of channels of the feature map, and h and w are the height and width of the feature map. The batch size dimension is called the first dimension, the channel number dimension is called the second dimension, and the height and width dimensions are called the third and fourth dimensions respectively. In each dimension of feat1, the batch size and number of channels are the same as feat2, and the width and height are twice that of feat2. The size of each dimension is b, c, 2*h, 2*w.

[0021] (3.3) For feat1, global max pooling is used for feature extraction, and the width and height are reduced to 1 / 4 of the original, while the batch size and number of channels remain unchanged to obtain a new feature map feat1. pooled The dimensions are b, c, h / 2, w / 2;

[0022] (3.4) For feat2, convolution is performed using kernels with strides of (1,2) and (2,1) respectively, i.e., horizontal compression and vertical compression, to obtain the new feature map feat2. x and feat2 y Their dimensions are b, c, h, w / 2 and b, c, h / 2, w, respectively;

[0023] (3.5) For feat2, horizontal average pooling and vertical average pooling are then used to obtain the average features feat2 in both directions. xavg and feat2 yavg The dimensions are b, c, h, 1 and b, c, 1, w, respectively;

[0024] (3.6) Use the horizontal dimension to expand the weighted module, and transfer feat1 pooled and feat2 xavg Weighted summation is applied, channel dimension size is halved, height is preserved, and feature2 is summed. xavg Same width and feature1 pooled Keeping them the same, we obtain a new feature map feat2. x_weight The dimensions are b, c / 2, h, w / 2; using the vertical dimension expansion weighting module, feat1 pooled and feat2 yavg Weighted summation is applied, channel dimension size is halved, width is maintained, and feature2 is summed. yavg Same height and feature1 pooled Keeping them the same, we obtain a new feature map feat2. y_weight The dimensions are b, c / 2, h / 2, w;

[0025] (3.7) Using a multilayer perceptron to transfer feat2 x The number of channels is mapped to c / 2, and feat2 is transferred. x_weight with feat2 x Performing a dot product yields a new feature map feat2 x_fused ; using a multilayer perceptron to transfer feat2 y The number of channels is mapped to c / 2, and feat2 is transferred. y_weight with feat2 y Dot product yields feat2 y_fused ;

[0026] (3.8) Using the weighted module, with feat2 x_fused and feat2 y_fused We weighted feat2 to obtain the final output.

[0027] Furthermore, in step (3.6), the specific process of the horizontal dimension expansion weighting module is as follows:

[0028] (4.1) Reading feature map feat1 pooled At this point, the sizes of each dimension are denoted as b, c, h / 2, w / 2; the feature map feat2 is read. xavg At this point, the dimensions of each dimension are b, c, h, and 1.

[0029] (4.2) Using a multilayer perceptron to process the input feat2 xavg Perform mapping to obtain feat2 xmap Its dimensions are b, c / 2, h, 1;

[0030] (4.3) Using convolution pairs feat1 pooled Mapping yields a new feat1 map Its dimensions are b, c / 2, h / 2, w / 2;

[0031] (4.4) feat2 xmap Copy and expand along the fourth dimension, making the size of each dimension b, c / 2, h, w / 2;

[0032] (4.5) For feat1 map Perform interpolation along the third dimension to make the sizes of each dimension b, c / 2, h, w / 2;

[0033] (4.6) Add feat1 map and feat2 xmap Perform a dot product to obtain the output feature feat2 x_weight The dimensions are b, c / 2, h, w / 2.

[0034] Furthermore, in step (3.6), the specific process of the vertical dimension expansion weighting module is as follows:

[0035] (5.1) Reading feature map feat1 pooled At this point, the sizes of each dimension are denoted as b, c, h / 2, w / 2; the feature map feat2 is read. yavg At this point, the dimensions of each dimension are b, c, 1, and w.

[0036] (5.2) Using a multilayer perceptron to process the input feat2 yavg Mapping yields feat2 ymap Its dimensions are b, c / 2, 1, w;

[0037] (5.3) Using convolution pairs feat1 pooled Mapping yields a new feat1 map2 Its dimensions are b, c / 2, h / 2, w / 2;

[0038] (5.4) feat2 ymap Copy and expand along the third dimension, making the size of each dimension b, c / 2, h / 2, w;

[0039] (5.5) For feat1 map2 Perform interpolation along the fourth dimension to make the sizes of each dimension b, c / 2, h / 2, w;

[0040] (5.6) Move feat1 map2 and feat2 ymap Perform a dot product to obtain the output feature feat2 y_weight The dimensions are b, c / 2, h / 2, and w.

[0041] Furthermore, in step (3.8), the specific steps of the weighting module are as follows:

[0042] (6.1) Use interpolation methods for feat2 x_fused After horizontal interpolation, the dimensions are b, c / 2, h, and w.

[0043] (6.2) Use interpolation methods for feat2 y_fused Perform vertical interpolation, and the resulting dimensions are b, c / 2, h, and w.

[0044] (6.3) Use convolution operations to apply them to feat2 respectively. x_fused and feat2 y_fused Perform channel mapping to obtain results with dimensions b, c / 4, h, and w.

[0045] (6.4) feat2 x_fused and feat2 y_fused Adding them together gives feat2 fused ;

[0046] (6.5) Using convolution operations on feat2 fused After channel mapping, the dimensions become b, 1, h, and w;

[0047] (6.6) For feat2 fused The weighted output is obtained by multiplying the product of the two products (feat2 and feature2), and the dimensions are then processed to become b, c, h, and w.

[0048] Furthermore, in step 2, the specific process of RoIAlign extracting feature maps of interest at different resolutions is as follows:

[0049] (7.1) Using a RoIAlign sampler with a resolution of 7*7, sampling is performed on the fused feature map based on the RPN detection results. The sampling results are denoted as feat. small ;

[0050] (7.2) Using the RoIAlign sampler with a resolution of 14*14, sampling is performed on the feature map before fusion based on the RPN detection results. The sampling results are denoted as feat. medium ;

[0051] (7.3) Using the RoIAlign sampler with a resolution of 28*28, sampling is performed on the feature map before fusion based on the RPN detection results. The sampling results are denoted as feat. large .

[0052] Furthermore, in step 2, the specific process of the cross-attention fusion method at different resolutions is as follows:

[0053] (8.1) The feat with a width and height of 28*28 large The feature map is divided into a series of 2*2 small feature maps, which are fed into the transformer module to calculate self-attention. The results are then convolved to obtain the score of each small feature map.

[0054] (8.2) Sort the small feature maps according to the scores, select the top 1 / 4 of the small feature maps, and calculate their score in the original feature map. large The position in the middle;

[0055] (8.3) In the feature with a width and height of 14*14 medium In the feature map, the selected features large Small feature maps in the image are mapped to weighted features using a multilayer perceptron. medium On the corresponding pixel;

[0056] (8.4) The feat with a width and height of 14*14 medium The feature map is divided into a series of 2*2 small feature maps, which are fed into the transformer module to calculate self-attention. The results are then convolved to obtain the score of each small feature map.

[0057] (8.5) Sort the small feature maps according to the scores, select the top 1 / 4 of the small feature maps, and calculate their score in the original feature map. medium The position in the middle;

[0058] (8.5) In a feature with a width and height of 7*7 small In the feature map, the selected features medium Small feature maps in the image are mapped to weighted features using a multilayer perceptron.small On the corresponding pixel;

[0059] (8.6) Weighted features small Using bilinear interpolation, its size was expanded to 14x14, with feat medium Perform addition and then apply convolution mapping;

[0060] (8.7) Weighted features medium Using bilinear interpolation, its size was expanded to 28x28, and featured... large The two sides are added together, and the output is obtained by convolution mapping.

[0061] Furthermore, in step (8.1), the calculation process of the transformer module is as follows:

[0062] (9.1) Expand the dimensions of the feature map x into 4*1;

[0063] (9.2) Calculate self-attention for x using the formula shown below:

[0064] q = W q x (1)

[0065] k = W k x (2)

[0066] v = W v x (3)

[0067]

[0068] Attn normed = Norm(Attn) + x (5)

[0069] output = Norm(MLP(Attn) normed ))+Attn normed (6)

[0070] Among them, W q W k W v These are weight parameters, automatically determined by the network model during training. q, k, and v are variables in the calculation process. d is the size of the channel dimension of q. Norm represents the regularization function. MLP is a multilayer perceptron. output is the final output result.

[0071] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0072] First, remote sensing images often contain a large number of similar objects distributed within a single image, such as vehicles in a parking lot or buildings in a city. The presence of these numerous similar targets places certain demands on the model's performance; the model needs to be able to detect and represent these targets. Existing feature pyramid models have limited effectiveness in enhancing the information of feature maps. This invention proposes a two-dimensional weighted module that enhances features in both the width and height directions of the feature map. By employing different methods, it improves the fusion degree between feature maps, maximizing the feature extraction capability.

[0073] Second, this invention proposes a multi-resolution feature interaction segmentation head module. This module extracts features from the target region of the feature map, generating feature maps with resolutions of 7*7, 14*14, and 28*28. Then, an attention mechanism is used to interact between the two feature maps to generate a mask for the object. This method is compatible with the size of targets at different scales, producing relatively fine masks for both large and small-scale targets, and also meeting the requirements for generating complex mask details. Attached Figure Description

[0074] Figure 1 This is a flowchart of the process of this invention;

[0075] Figure 2 This is an overall structural diagram of the network of the present invention;

[0076] Figure 3 This is a schematic diagram of the network structure of the two-dimensional weighted module;

[0077] Figure 4 This is a schematic diagram of the network structure of the multi-resolution feature interactive segmentation head. Detailed Implementation

[0078] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

[0079] like Figure 1 As shown, this invention proposes a remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling, which includes the following steps:

[0080] Step 1: Obtain the remote sensing image instance segmentation dataset iSAID and preprocess the data in the dataset;

[0081] Step 2: Build the Mask R-CNN network model. The initial multi-scale feature map of the image is obtained by the backbone network, the Swing Transformer, and the Feature Pyramid Network (FPN). The feature map is weighted using a two-dimensional weighting module to obtain a fused multi-scale feature map. The fused multi-scale feature map is fed into the Region Proposal Network (RPN) to generate a coarse bounding box for initial target location determination. The RoIAlign network at different resolutions is used to extract the feature map of interest at the target location in the initial feature map and the fused feature map. The feature of interest from the fused feature map is used for target classification and bounding box prediction. Cross-attention is calculated between the feature maps of interest before and after fusion to obtain the final feature map for predicting the segmentation result.

[0082] Step 3: Use the remote sensing image instance segmentation data obtained in Step 1 to train the model built in Step 2;

[0083] Step 4: Use the trained remote sensing image instance segmentation model to segment the remote sensing image and determine the location, size, and shape of the targets within it.

[0084] Furthermore, the method for step 1 is as follows:

[0085] Step 1-1: Obtain the remote sensing image instance segmentation dataset iSAID. This dataset contains 2806 remote sensing images, with 15 categories and 655451 instance objects.

[0086] Steps 1-2: Crop the image, cutting the original image into 800*800 images with 200 pixels of overlap between adjacent images; modify the instance information in the dataset to fit the image size.

[0087] Steps 1-3 involve re-dividing the cut images and labels to obtain training and validation sets for model training.

[0088] Furthermore, in step 2, the specific process of the two-dimensional weighting module is as follows:

[0089] (3.1) Sort the initial multi-scale feature maps generated in step 2 from largest to smallest according to their length or width;

[0090] (3.2) The first four feature maps in the sorting results are designated as feature maps 1, 2, 3 and 4 respectively. The initial feature maps 1 and 2, 2 and 3, and 3 and 4 are grouped into a pair of feature map groups and processed one pair at a time. For each feature map group, the first feature map is designated as feat1 and the second as feat2. Based on the size of each dimension of feat2, they are designated as b, c, h and w respectively. Here, b is the batch size of the feature map, c is the number of channels of the feature map, and h and w are the height and width of the feature map. The batch size dimension is called the first dimension, the channel number dimension is called the second dimension, and the height and width dimensions are called the third and fourth dimensions respectively. In each dimension of feat1, the batch size and number of channels are the same as feat2, and the width and height are twice that of feat2. The size of each dimension is b, c, 2*h, 2*w.

[0091] (3.3) For feat1, global max pooling is used for feature extraction, and the width and height are reduced to 1 / 4 of the original, while the batch size and number of channels remain unchanged to obtain a new feature map feat1. pooled The dimensions are b, c, h / 2, w / 2;

[0092] (3.4) For feat2, convolution is performed using kernels with strides of (1,2) and (2,1) respectively, i.e., horizontal compression and vertical compression, to obtain the new feature map feat2. x and feat2 y Their dimensions are b, c, h, w / 2 and b, c, h / 2, w, respectively;

[0093] (3.5) For feat2, horizontal average pooling and vertical average pooling are then used to obtain the average features feat2 in both directions. xavg and feat2 yavg The dimensions are b, c, h, 1 and b, c, 1, w, respectively;

[0094] (3.6) Use the horizontal dimension to expand the weighted module, and transfer feat1 pooled and feat2 xavg Weighted summation is applied, channel dimension size is halved, height is preserved, and feature2 is summed. xavg Same width and feature1 pooled Keeping them the same, we obtain a new feature map feat2. x_weight The dimensions are b, c / 2, h, w / 2; using the vertical dimension expansion weighting module, feat1 pooled and feat2 yavg Weighted summation is applied, channel dimension size is halved, width is maintained, and feature2 is summed. yavg Same height and feature1pooled Keeping them the same, we obtain a new feature map feat2. y_weight The dimensions are b, c / 2, h / 2, w;

[0095] (3.7) Using a multilayer perceptron to transfer feat2 x The number of channels is mapped to c / 2, and feat2 is transferred. x_weight with feat2 x Performing a dot product yields a new feature map feat2 x_fused ; using a multilayer perceptron to transfer feat2 y The number of channels is mapped to c / 2, and feat2 is transferred. y_weight with feat2 y Dot product yields feat2 y_fused ;

[0096] (3.8) Using the weighted module, with feat2 x_fused and feat2 y_fused We weighted feat2 to obtain the final output.

[0097] Furthermore, in step (3.6), the specific process of the horizontal dimension expansion weighting module is as follows:

[0098] (4.1) Reading feature map feat1 pooled At this point, the sizes of each dimension are denoted as b, c, h / 2, w / 2; the feature map feat2 is read. xavg At this point, the dimensions of each dimension are b, c, h, and 1.

[0099] (4.2) Using a multilayer perceptron to process the input feat2 xavg Perform mapping to obtain feat2 xmap Its dimensions are b, c / 2, h, 1;

[0100] (4.3) Using convolution pairs feat1 pooled Mapping yields a new feat1 map Its dimensions are b, c / 2, h / 2, w / 2;

[0101] (4.4) feat2 xmap Copy and expand along the fourth dimension, making the size of each dimension b, c / 2, h, w / 2;

[0102] (4.5) For feat1 map Perform interpolation along the third dimension to make the sizes of each dimension b, c / 2, h, w / 2;

[0103] (4.6) Add feat1 map and feat2xmap Perform a dot product to obtain the output feature feat2 x_weight The dimensions are b, c / 2, h, w / 2.

[0104] Furthermore, in step (3.6), the specific process of the vertical dimension expansion weighting module is as follows:

[0105] (5.1) Reading feature map feat1 pooled At this point, the sizes of each dimension are denoted as b, c, h / 2, w / 2; the feature map feat2 is read. yavg At this point, the dimensions of each dimension are b, c, 1, and w.

[0106] (5.2) Using a multilayer perceptron to process the input feat2 yavg Mapping yields feat2 ymap Its dimensions are b, c / 2, 1, w;

[0107] (5.3) Using convolution pairs feat1 pooled Mapping yields a new feat1 map2 Its dimensions are b, c / 2, h / 2, w / 2;

[0108] (5.4) feat2 ymap Copy and expand along the third dimension, making the size of each dimension b, c / 2, h / 2, w;

[0109] (5.5) For feat1 map2 Perform interpolation along the fourth dimension to make the sizes of each dimension b, c / 2, h / 2, w;

[0110] (5.6) Move feat1 map2 and feat2 ymap Perform a dot product to obtain the output feature feat2 y_weight The dimensions are b, c / 2, h / 2, and w.

[0111] Furthermore, in step (3.8), the specific steps of the weighting module are as follows:

[0112] (6.1) Use interpolation methods for feat2 x_fused After horizontal interpolation, the dimensions are b, c / 2, h, and w.

[0113] (6.2) Use interpolation methods for feat2 y_fused Perform vertical interpolation, and the resulting dimensions are b, c / 2, h, and w.

[0114] (6.3) Use convolution operations to apply them to feat2 respectively. x_fused and feat2y_fused Perform channel mapping to obtain results with dimensions b, c / 4, h, and w.

[0115] (6.4) feat2 x_fused and feat2 y_fused Adding them together gives feat2 fused ;

[0116] (6.5) Using convolution operations on feat2 fused After channel mapping, the dimensions become b, 1, h, and w;

[0117] (6.6) For feat2 fused The weighted output is obtained by multiplying the product of the two products (feat2 and feature2), and the dimensions are then processed to become b, c, h, and w.

[0118] Furthermore, in step 2, the specific process of RoIAlign extracting feature maps of interest at different resolutions is as follows:

[0119] (7.1) Using a RoIAlign sampler with a resolution of 7*7, sampling is performed on the fused feature map based on the RPN detection results. The sampling results are denoted as feat. small ;

[0120] (7.2) Using the RoIAlign sampler with a resolution of 14*14, sampling is performed on the feature map before fusion based on the RPN detection results. The sampling results are denoted as feat. medium ;

[0121] (7.3) Using the RoIAlign sampler with a resolution of 28*28, sampling is performed on the feature map before fusion based on the RPN detection results. The sampling results are denoted as feat. large .

[0122] Furthermore, in step 2, the specific process of the cross-attention fusion method at different resolutions is as follows:

[0123] (8.1) The feat with a width and height of 28*28 large The feature map is divided into a series of 2*2 small feature maps, which are fed into the transformer module to calculate self-attention. The results are then convolved to obtain the score of each small feature map.

[0124] (8.2) Sort the small feature maps according to the scores, select the top 1 / 4 of the small feature maps, and calculate their score in the original feature map. large The position in the middle;

[0125] (8.3) In the feature with a width and height of 14*14 mediumIn the feature map, the selected features large Small feature maps in the image are mapped to weighted features using a multilayer perceptron. medium On the corresponding pixel;

[0126] (8.4) The feat with a width and height of 14*14 medium The feature map is divided into a series of 2*2 small feature maps, which are fed into the transformer module to calculate self-attention. The results are then convolved to obtain the score of each small feature map.

[0127] (8.5) Sort the small feature maps according to the scores, select the top 1 / 4 of the small feature maps, and calculate their score in the original feature map. medium The position in the middle;

[0128] (8.5) In a feature with a width and height of 7*7 small In the feature map, the selected features medium Small feature maps in the image are mapped to weighted features using a multilayer perceptron. small On the corresponding pixel;

[0129] (8.6) Weighted features small Using bilinear interpolation, its size was expanded to 14x14, with feat medium Perform addition and then apply convolution mapping;

[0130] (8.7) Weighted features medium Using bilinear interpolation, its size was expanded to 28x28, and featured... large The two sides are added together, and the output is obtained by convolution mapping.

[0131] Furthermore, in step (8.1), the calculation process of the transformer module is as follows:

[0132] (9.1) Expand the dimensions of the feature map x into 4*1;

[0133] (9.2) Calculate self-attention for x using the formula shown below:

[0134] q = W q x (1)

[0135] k = W k x (2)

[0136] v = W v x (3)

[0137]

[0138] Attn normed = Norm(Attn) + x (5)

[0139] output = Norm(MLP(Attn) normed ))+Attn normed (6)

[0140] Among them, W q W k W v These are weight parameters, automatically determined by the network model during training. q, k, and v are variables in the calculation process. d is the size of the channel dimension of q. Norm represents the regularization function. MLP is a multilayer perceptron. output is the final output result.

[0141] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling, characterized in that, The method includes the following steps: Step 1: Obtain the remote sensing image instance segmentation dataset iSAID and preprocess the data in the dataset; Step 2: Build the Mask R-CNN network model. The initial multi-scale feature map of the image is obtained by the backbone network, the Swing Transformer, and the Feature Pyramid Network (FPN). The feature map is weighted using a two-dimensional weighting module to obtain a fused multi-scale feature map. The fused multi-scale feature map is fed into the Region Proposal Network (RPN) to generate a coarse bounding box for initial target location determination. The RoIAlign network at different resolutions is used to extract the feature map of interest at the target location in the initial feature map and the fused feature map. The feature of interest from the fused feature map is used for target classification and bounding box prediction. Cross-attention is calculated between the feature maps of interest before and after fusion to obtain the final feature map for predicting the segmentation result. Step 3: Use the remote sensing image instance segmentation data obtained in Step 1 to train the model built in Step 2; Step 4: Use the trained remote sensing image instance segmentation model to segment the remote sensing image and determine the location, size, and shape of the targets within it; In step 2, the specific process of the two-dimensional weighting module is as follows: (3.1) Sort the initial multi-scale feature maps generated in step 2 from largest to smallest according to their length or width; (3.2) The first four feature maps in the sorting results are designated as feature maps 1, 2, 3 and 4 respectively. The initial feature maps 1 and 2, 2 and 3, and 3 and 4 are grouped into a pair of feature map groups and processed one pair at a time. For each feature map group, the first feature map is designated as feat1 and the second as feat2. Based on the size of each dimension of feat2, they are designated as b, c, h and w respectively. Here, b is the batch size of the feature map, c is the number of channels of the feature map, and h and w are the height and width of the feature map. The batch size dimension is called the first dimension, the channel number dimension is called the second dimension, and the height and width dimensions are called the third and fourth dimensions respectively. In each dimension of feat1, the batch size and number of channels are the same as feat2, and the width and height are twice that of feat2. The size of each dimension is b, c, 2*h, 2*w. (3.3) For feat1, global max pooling is used for feature extraction, and the width and height are reduced to 1 / 4 of the original, while the batch size and number of channels remain unchanged to obtain a new feature map feat1. pooled The dimensions are b, c, h / 2, w / 2; (3.4) For feat2, convolution is performed using kernels with strides of (1,2) and (2,1) respectively, i.e., horizontal compression and vertical compression, to obtain the new feature map feat2. x and feat2 y Their dimensions are b, c, h, w / 2 and b, c, h / 2, w, respectively; (3.5) For feat2, horizontal average pooling and vertical average pooling are then used to obtain the average features feat2 in both directions. xavg and feat2 yavg The dimensions are b, c, h, 1 and b, c, 1, w, respectively; (3.6) Use the horizontal dimension to expand the weighted module, and transfer feat1 pooled and feat2 xavg Weighted summation is applied, channel dimension size is halved, height is preserved, and feature2 is summed. xavg Same width and feature1 pooled Keeping them the same, we obtain a new feature map feat2. x_weight The dimensions are b, c / 2, h, w / 2; using the vertical dimension expansion weighting module, feat1 pooled and feat2 yavg Weighted summation is applied, channel dimension size is halved, width is maintained, and feature2 is summed. yavg Same height and feature1 pooled Keeping them the same, we obtain a new feature map feat2. y_weight The dimensions are b, c / 2, h / 2, w; (3.7) Using a multilayer perceptron to transfer feat2 x The number of channels is mapped to c / 2, and feat2 is transferred. x_weight with feat2 x Performing a dot product yields a new feature map feat2 x_fused ; using a multilayer perceptron to transfer feat2 y The number of channels is mapped to c / 2, and feat2 is transferred. y_weight with feat2 y Dot product yields feat2 y_fused ; (3.8) Using the weighted module, with feat2 x_fused and feat2 y_fused We weighted feat2 to obtain the final output.

2. The remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling according to claim 1, characterized in that, The method for step 1 is as follows: Step 1-1: Obtain the remote sensing image instance segmentation dataset iSAID. This dataset contains 2806 remote sensing images, with 15 categories and 655451 instance objects. Steps 1-2: Crop the image, cutting the original image into 800*800 images with 200 pixels of overlap between adjacent images; modify the instance information in the dataset to fit the image size. Steps 1-3 involve re-dividing the cut images and labels to obtain training and validation sets for model training.

3. The remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling according to claim 1, characterized in that, In step (3.6), the specific process of the horizontal dimension expansion weighting module is as follows: (4.1) Reading feature map feat1 pooled At this point, the sizes of each dimension are denoted as b, c, h / 2, w / 2; the feature map feat2 is read. xavg At this point, the dimensions of each dimension are b, c, h, and 1. (4.2) Using a multilayer perceptron to process the input feat2 xavg Perform mapping to obtain feat2 xmap Its dimensions are b, c / 2, h, 1; (4.3) Using convolution pairs feat1 pooled Mapping yields a new feat1 map Its dimensions are b, c / 2, h / 2, w / 2; (4.4) feat2 xmap Copy and expand along the fourth dimension, making the size of each dimension b, c / 2, h, w / 2; (4.5) For feat1 map Perform interpolation along the third dimension to make the sizes of each dimension b, c / 2, h, w / 2; (4.6) Add feat1 map and feat2 xmap Perform a dot product to obtain the output feature feat2 x_weight The dimensions are b, c / 2, h, w / 2.

4. The remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling according to claim 1, characterized in that, In step (3.6), the specific process of the vertical dimension expansion weighting module is as follows: (5.1) Reading feature map feat1 pooled At this point, the sizes of each dimension are denoted as b, c, h / 2, w / 2; the feature map feat2 is read. yavg At this point, the dimensions of each dimension are b, c, 1, and w. (5.2) Using a multilayer perceptron to process the input feat2 yavg Mapping yields feat2 ymap Its dimensions are b, c / 2, 1, w; (5.3) Using convolution pairs feat1 pooled Mapping yields a new feat1 map2 Its dimensions are b, c / 2, h / 2, w / 2; (5.4) feat2 ymap Copy and expand along the third dimension, making its dimensions b, c / 2, h / 2, w; (5.5) For feat1 map2 Perform interpolation along the fourth dimension to make the sizes of each dimension b, c / 2, h / 2, w; (5.6) Move feat1 map2 and feat2 ymap Perform a dot product to obtain the output feature feat2 y_weight The dimensions are b, c / 2, h / 2, and w.

5. The remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling according to claim 1, characterized in that, In step (3.8), the specific steps of the weighting module are as follows: (6.1) Use interpolation methods for feat2 x_fused After horizontal interpolation, the dimensions are b, c / 2, h, and w. (6.2) Use interpolation methods for feat2 y_fused Perform vertical interpolation, and the resulting dimensions are b, c / 2, h, and w. (6.3) Use convolution operations to apply them to feat2 respectively. x_fused and feat2 y_fused Perform channel mapping to obtain results with dimensions b, c / 4, h, and w. (6.4) feat2 x_fused and feat2 y_fused Adding them together gives feat2 fused ; (6.5) Using convolution operations on feat2 fused After channel mapping, the dimensions become b, 1, h, and w; (6.6) For feat2 fused The weighted output is obtained by multiplying the product of feat2 and the product, and the dimensions are then b, c, h, and w.

6. The remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling according to claim 1, characterized in that, In step 2, the specific process of RoIAlign extracting feature maps of interest at different resolutions is as follows: (7.1) Using a RoIAlign sampler with a resolution of 7*7, sampling is performed on the fused feature map based on the RPN detection results. The sampling results are denoted as feat. small ; (7.2) Using the RoIAlign sampler with a resolution of 14*14, sampling is performed on the feature map before fusion based on the RPN detection results. The sampling results are denoted as feat. medium ; (7.3) Using the RoIAlign sampler with a resolution of 28*28, sampling is performed on the feature map before fusion based on the RPN detection results. The sampling results are denoted as feat. large .

7. The remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling according to claim 6, characterized in that, In step 2, the specific process of the cross-attention fusion method at different resolutions is as follows: (8.1) The feat with a width and height of 28*28 large The feature map is divided into a series of 2*2 small feature maps, which are fed into the transformer module to calculate self-attention. The results are then convolved to obtain the score of each small feature map. (8.2) Sort the small feature maps according to the scores, select the top 1 / 4 of the small feature maps, and calculate their score in the original feature map. large The position in the middle; (8.3) In the feature with a width and height of 14*14 medium In the feature map, the selected features large Small feature maps in the image are mapped to weighted features using a multilayer perceptron. medium On the corresponding pixel; (8.4) The feat with a width and height of 14*14 medium The feature map is divided into a series of 2*2 small feature maps, which are fed into the transformer module to calculate self-attention. The results are then convolved to obtain the score of each small feature map. (8.5) Sort the small feature maps according to the scores, select the top 1 / 4 of the small feature maps, and calculate their score in the original feature map. medium The position in the middle; (8.5) In a feature with a width and height of 7*7 small In the feature map, the selected features medium Small feature maps in the image are mapped to weighted features using a multilayer perceptron. small On the corresponding pixel; (8.6) Weighted features small Using bilinear interpolation, its size was expanded to 14x14, with feat medium Perform addition and then apply convolution mapping; (8.7) Weighted features medium Using bilinear interpolation, its size was expanded to 28x28, and featured... large The two sides are added together, and the output is obtained by convolution mapping.

8. The remote sensing image instance segmentation method based on multi-scale feature fusion and multi-resolution sampling according to claim 7, characterized in that, In step (8.1), the calculation process of the transformer module is as follows: (9.1) Expand the dimensions of the feature map x into 4*1; (9.2) Calculate self-attention for x using the formula shown below: ； ； ； ； ； ； Among them, W q W k W v These are weight parameters, automatically determined by the network model during training. q, k, and v are variables in the calculation process, and d is the size of the channel dimension of q. Represents the regularization function. It is a multilayer perceptron, and output is the final output result.

Images