Few-sample remote sensing image semantic segmentation method based on double-branch reinforcement network

Abstract

The invention discloses a few-sample remote sensing image semantic segmentation method based on a double-branch reinforcement network. The method comprises the following steps: firstly, building a backbone network, and respectively extracting middle-level and high-level support features and query features from a support image and a query image; secondly, constructing a prototype enhancement module based on feature customization and activation reconstruction, and inputting the intermediate support features and the query features into the module to obtain enhanced prototype features; thirdly, providing an enhancement module based on multi-level guidance, and inputting the advanced support features and the query features into the module to obtain a multi-level guidance graph; and finally, designing an encoder, and fusing the enhanced prototype features, the advanced query features and the multi-level guide graph to obtain a final segmentation result. The prototype enhancement module designed by the invention can deeply describe semantic features of a target category, and the target expression ability is improved; meanwhile, a proposed multi-level guide enhancement module can capture correlation among different levels of images, and the overall framework is enhanced through unbiased knowledge, so that good segmentation performance is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a semantic segmentation method for a few-sample remote sensing image based on a dual-branch reinforcement network. Background Art

[0002] Remote sensing image semantic segmentation is an important research direction in the field of remote sensing image processing. It refers to the pixel-level classification of images with semantic information, that is, each pixel in the image is divided according to its category, so as to achieve a deep understanding of the content of the remote sensing image. Remote sensing image semantic segmentation plays a key role in many fields, such as urban planning, agricultural monitoring, disaster assessment, etc. In addition, remote sensing image semantic segmentation is also of great significance in decision support, automated processing, and technology application. In recent years, deep learning technology has experienced explosive growth, and it has also been widely used in the field of remote sensing image semantic segmentation. However, training deep neural networks usually requires a large amount of finely labeled sample data, and the pixel-level data labeling task in remote sensing image semantic segmentation is very time-consuming and labor-intensive.

[0003] To this end, few-shot segmentation is proposed, which aims to solve the problem of image semantic segmentation when there are only a few labeled samples. Compared with traditional remote sensing image semantic segmentation methods, few-shot remote sensing image semantic segmentation attempts to achieve better segmentation results by using a small number of samples. At present, few-shot remote sensing image semantic segmentation methods are mainly divided into two categories: (1) prototype-level methods; (2) pixel-level methods.

[0004] Publication No. CN118154427A introduces a few-sample remote sensing image segmentation method, which includes spectral super-resolution reconstruction, image attention feature extraction, feature splicing, decoder decoding and other processes. First, spectral super-resolution reconstruction is achieved through a dynamic convolutional multi-column network, and the image space attention and spectral attention re-extraction are performed on the spectral super-resolution reconstruction features respectively, and then the two attention extracted features are spliced; then the segmentation results are obtained through a few-sample remote sensing image semantic segmentation decoder and a spectral super-resolution reconstruction decoder. This method uses the identifiable spectral information in hyperspectral images to improve the segmentation performance. However, this method involves the conversion between hyperspectral images and three-channel images, and there are problems such as information loss, band selection and color distortion. Therefore, the ability to express targets with a wide variety and complex expressions in remote sensing images is still very limited.

[0005] Ao et al. published a paper titled "Few-Shot Aerial Image Semantic Segmentation Leveraging PyramidCorrelation Fusion" in the 2023 IEEE Transactions on Geoscience and Remote Sensing. This paper proposes a pixel-level pyramid correlation fusion network for the task of few-shot remote sensing image segmentation. It calculates pixel-level semantic correspondence by calculating cosine similarity and Euclidean distance, and introduces a scale-aware cross-entropy loss function to improve the segmentation accuracy of small targets. However, the scale-aware cross-entropy function in this method focuses on the size of the target, and the segmentation effect of complex-shaped target objects is poor. At the same time, this method involves the calculation of correlation graphs during the segmentation process, which is easily interfered by noise.

[0006] In summary, the general few-sample remote sensing image semantic segmentation methods have many limitations, mainly manifested in:

[0007] (1) Poor ability to map target objects: Most mainstream methods extract features from support images, ignoring the rich category information in the query image. At the same time, pixel-level methods are easily affected by irrelevant information in the image, resulting in limited effective information.

[0008] (2) Poor generalization performance: Most current methods fail to fully consider the characteristics of remote sensing images, such as strong internal heterogeneity and high confusion between different categories. A large amount of target detail information will be lost during the processing, resulting in poor generalization performance. Summary of the invention

[0009] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a semantic segmentation method for remote sensing images of few samples based on a dual-branch reinforcement network. The prototype reinforcement module designed by this method can deeply characterize the semantic features of the target category, thereby improving the target expression ability; at the same time, the proposed multi-level guided reinforcement module can capture the correlation between images at different levels, and enhance the overall framework through unbiased knowledge, which not only avoids the problem of increasing the number of parameters, but also enhances the generalization ability of the model from a priori perspective, and obtains good segmentation performance.

[0010] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a semantic segmentation method of a few-sample remote sensing image based on a dual-branch reinforcement network, which includes six steps, as follows:

[0011] (1) Construct a few-sample remote sensing image semantic segmentation dataset and divide the dataset into a training set D train and the test set D test , D trainand D test They are composed of several groups of sample sets;

[0012] (2) Construct a query image-guided dual-branch enhanced few-shot remote sensing image semantic segmentation network. The network mainly consists of four parts: a backbone network, a prototype enhancement module based on feature customization and activation reconstruction, an enhancement module based on multi-level guidance, and a decoder. The network is trained on the training set D in step (1). train Train on the test set D test Test on;

[0013] (3) Using the backbone network in step (2), perform deep feature extraction on the support images and query images in each set of samples to obtain medium and high-level support features and query features;

[0014] (4) sending the intermediate support features and query features obtained in step (3) to the prototype enhancement module based on feature customization and activation reconstruction in step (2) to obtain enhanced prototype features;

[0015] (5) sending the high-level support features and high-level query features obtained in step (3) to the multi-level guidance-based reinforcement module in step (2) to obtain a multi-level guidance graph;

[0016] (6) The high-level query features in step (3), the enhanced prototype features in step (4), and the multi-level guidance map in step (5) are fed into the decoder in step (2) to achieve accurate semantic segmentation of the remote sensing image.

[0017] In step (1), the method for constructing a few-sample remote sensing image semantic segmentation dataset is as follows:

[0018] (1.1) Construct a few-sample remote sensing image semantic segmentation dataset Image = [Image1,…,Image i ,…,Image N ], and make the corresponding sample label Label = [Label1,…,Label i ,…,Label N ], where N represents N remote sensing images, Image i Represents the i-th remote sensing image, Label i Indicates the label corresponding to the i-th remote sensing image;

[0019] (1.2) Divide the dataset Image into a training set D train and the test set D test, assuming that the total number of samples is n, m images containing the training category are selected to construct the training set, and the remaining nm images are selected to construct the test set; then, the training set and the test set are randomly divided into several groups of sample sets according to the target category; each group of sample sets is represented as in, Indicated by the supporting image and corresponding tags The support set is composed of K, K represents the K-shot task; {(Q q ,M q )} represents the query image Q q and the corresponding label M q The query set formed.

[0020] Among them, the overall architecture design of the query image-guided dual-branch enhanced few-sample remote sensing image semantic segmentation network in step (2) is as follows:

[0021] (2.1) Build a pyramid scene parsing network (PSPNet) based on VGG-16 and ResNet-50 as the backbone network, and extract mid- and high-level features of the support image and the query image based on this network;

[0022] (2.2) Design prototype branch. The core of this branch is the prototype enhancement module based on feature customization and activation reconstruction. This module is divided into two parts: feature customization and activation reconstruction. In the feature customization part, the intermediate support features are and mid-level query features Associating them, we get the weight matrix w representing the similarity between the two, and use this matrix to get the reconstructed query features In the Activate Reconstruction section, add the Intermediate Query feature Pooling is performed to obtain the intermediate query feature vector, and the feature vector and the intermediate support feature vector are calculated. The similarity of Then activate the reconstructed query feature and Further combined to obtain the reconstructed support features Finally, the mask M will be supported S and refactoring support features Through mask average pooling (MAP), the enhanced prototype features can be finally obtained;

[0023] (2.3) Design the guidance branch. The core of this branch is a reinforcement module based on multi-level guidance. This module consists of four guidance information mining operations at different levels: First, through high-level support features and support mask M S Calculate the effective information matrix S and transform the advanced query features Convert it into a query matrix Q; then at each level, calculate the similarity matrix between S and Q to obtain the guide map at the current level, and fuse the guide map at the current level with the query matrix Q to obtain Q' as the query matrix for the next level; repeat the above process, and finally get four guide maps;

[0024] (2.4) Design a decoder that concatenates enhanced prototype features, high-level query features, and multi-level guidance maps in the channel dimension at four levels, and uses convolution to deeply fuse the three. Finally, semantic segmentation is achieved through atrous spatial pyramid pooling.

[0025] The specific process of obtaining enhanced prototype features based on feature customization and activation reconstruction of the prototype enhancement module in step (4) is as follows:

[0026] (4.1) Concatenate the extracted intermediate features in the channel dimension and obtain the intermediate query features through convolution dimensionality reduction. and mid-level support features This operation reduces the features to 256 dimensions, and the convolution kernel size is 1×1;

[0027] (4.2) The intermediate query features and mid-level support features Converted into a two-dimensional feature matrix by reorganization and transposition operations respectively and The intermediate query features Convert to a two-dimensional matrix W by redoing the operation Q , the formula is described as follows:

[0028]

[0029] Among them, conv represents the convolution operation, Γ represents the reorganization and transposition operation, and R represents the reorganization operation;

[0030] (4.3) The prototype enhancement module based on feature customization and activation reconstruction is mainly divided into two parts: feature customization and activation reconstruction. In the feature customization part, the intermediate support features and mid-level query features Associating them, we get the weight matrix w representing the similarity between the two. The formula is as follows:

[0031]

[0032] Among them, Φ represents matrix multiplication and softmax operation. Then use this matrix to calculate the reconstructed query features The formula is as follows:

[0033]

[0034] (4.4) In the activation reconstruction part of the prototype enhancement module, the intermediate query features are pooled to obtain the intermediate query feature vector, and the feature vector and the intermediate support feature vector are calculated. The similarity A between them is as follows:

[0035]

[0036] Among them, avgpool represents the average pooling function, F cosine Represents the cosine similarity calculation function. Assume an input feature map Input, use a pooling window of f×f and a step size s, and for the element Output(i,j) in the output feature map Output, its value is the result after average pooling, and avgpool is determined by the following formula:

[0037]

[0038] Among them, u and v represent the index of the traversal pooling window, i and j represent the position index of the element in the output feature map. In addition, F cosine The cosine similarity calculation formula is as follows:

[0039]

[0040] Among them, A and B represent and vectors of the same size in , i represents the i-th element in the vector, and n represents the length of the vector.

[0041] Then, the reconstructed query features are activated with similarity A and with Combined with refactored support features The formula is as follows:

[0042]

[0043] Here, ⊙ represents the Hadamard product.

[0044] (4.5) Using masked average pooling (MAP), the reconstructed support features Converted to the enhanced prototype feature pro, the formula is described as follows:

[0045]

[0046] Among them, MAP stands for masked average pooling.

[0047] The specific process of obtaining the guidance map based on the multi-level guidance reinforcement module in step (5) is as follows:

[0048] (5.1) Advanced Support Features and support mask M S Multiply them in the Hadamard product to obtain the effective information matrix S. The formula is as follows:

[0049]

[0050] Among them, ⊙ represents the Hadamard product, and Γ represents the reorganization and transposition operations.

[0051] Then, the advanced query feature Converted to query matrix Q, the formula is as follows:

[0052]

[0053] Among them, R represents the reorganization operation;

[0054] (5.2) Calculate the similarity between the effective information matrix S and the query matrix Q, and normalize them to obtain the guidance graph at the current level. The specific formula is as follows:

[0055]

[0056] sim′=max(sim)

[0057]

[0058] Here, ||·|| represents the norm operation, c is a hyperparameter, which is set to 1e -7 , map is the guide map at the current level;

[0059] (5.3) Multiply the guidance map at the current level by the query matrix Q in a Hadamard manner to obtain the query matrix Q' at the next level:

[0060] Q'=map⊙Q

[0061] (5.4) Advanced Support Features After scaling Q' to the next size, repeat (5.1) to (5.3) to obtain guidance maps at different levels.

[0062] Among them, in step (6), the decoder structure is designed as follows:

[0063] (6.1) At the current level, the high-level query features, enhanced prototype features and guidance maps are concatenated in the channel dimension and reduced to 256 dimensions through convolution operations with a convolution kernel size of 1×1;

[0064] (6.2) The reduced features are sent to the fusion module to obtain the fine features at the current level; the fusion module uses a 3×3 convolution kernel and a residual structure;

[0065] (6.3) The fine feature is concatenated with the feature after dimensionality reduction at the next level, and fused using convolution and residual operations; the convolution uses a 1×1 convolution kernel;

[0066] (6.4) Finally, the fine features at the four levels are concatenated in the channel dimension, and the final semantic segmentation result is obtained through atrous spatial pyramid pooling.

[0067] Beneficial effects: The present invention adopts the above technical solution and has the following beneficial effects:

[0068] (1) The prototype enhancement module in the method of the present invention integrates the query features into the supporting features, so that the generated prototype features can effectively capture the intrinsic characteristics and details of the target category, and more deeply characterize the semantic characteristics of the target category, thereby being able to show stronger expressiveness and higher accuracy when segmenting the target, and at the same time having a stronger ability to express the target.

[0069] (2) The multi-level guided reinforcement module in this method deeply explores the correlation between images at multiple abstract levels (layers) and constructs unbiased knowledge to provide the network with richer and more comprehensive information. This multi-level guidance and reinforcement not only improves the segmentation accuracy, but also significantly enhances the generalization ability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 The architecture diagram of the few-sample remote sensing image semantic segmentation network based on the dual-branch reinforcement network;

[0071] Figure 2 This is the internal structure diagram of the prototype reinforcement module;

[0072] Figure 3 Schematic diagram of guided information mining at one level in the multi-level guided reinforcement module. DETAILED DESCRIPTION

[0073] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0074] like Figure 1 As shown, the technical solution of the present invention is further described in detail as follows:

[0075] (1) Construct a few-sample remote sensing image semantic segmentation dataset and divide the dataset into a training set D train and the test set D test , D train and D test Each of them consists of several groups of samples:

[0076] (1.1) Construct a few-sample remote sensing image semantic segmentation dataset Image = [Image1,…,Image i,…,Image N ], and make the corresponding sample label Label = [Label1,…,Label i ,…,Label N ], where N represents N remote sensing images, Image i Represents the i-th remote sensing image, Label i Indicates the label corresponding to the i-th remote sensing image;

[0077] (1.2) Divide the dataset Image into a training set D train and the test set D test , assuming that the total number of samples is n, m images containing the training category are selected to construct the training set, and the remaining nm images are selected to construct the test set; then, the training set and the test set are randomly divided into several groups of sample sets according to the target category, and each group of sample sets is represented as in, Indicated by the supporting image and corresponding tags The support set is composed of K, K represents the K-shot task; {(Q q ,M q )} represents the query image Q q and the corresponding label M q The present invention uses the few-sample remote sensing image semantic segmentation public datasets iSAID and DLRSD for experiments. These two datasets contain 15 and 17 categories respectively.

[0078] (2) Construct a query image-guided dual-branch enhanced few-shot remote sensing image semantic segmentation network, which mainly consists of four parts: a backbone network, a prototype enhancement module based on feature customization and activation reconstruction, an enhancement module based on multi-level guidance, and a decoder. The specific structure is as follows Figure 1 As shown, the overall network workflow is as follows:

[0079] (2.1) Build a pyramid scene parsing network (PSPNet) based on VGG-16 and ResNet-50 as the backbone network, and extract mid- and high-level features of the support image and the query image based on this network;

[0080] (2.2) Design prototype branch. The core of this branch is the prototype enhancement module based on feature customization and activation reconstruction. This module is divided into two parts: feature customization and activation reconstruction. In the feature customization part, the intermediate support features are and mid-level query features Associating them, we get the weight matrix w representing the similarity between the two, and use this matrix to get the reconstructed query features In the Activate Reconstruction section, add the Intermediate Query feature Pooling is performed to obtain the intermediate query feature vector, and the feature vector and the intermediate support feature vector are calculated. The similarity of Then activate the reconstructed query feature and Further combined to obtain the reconstructed support features Finally, the mask M will be supported S and refactoring support features Through mask average pooling (MAP), the enhanced prototype features can be finally obtained;

[0081] (2.3) Design the guidance branch. The core of this branch is a reinforcement module based on multi-level guidance. This module consists of four guidance information mining operations at different levels: First, through high-level support features and support mask M S Calculate the effective information matrix S and transform the advanced query features Convert it into a query matrix Q; then at each level, calculate the similarity matrix between S and Q to obtain the guide map at the current level, and fuse the guide map at the current level with the query matrix Q to obtain Q' as the query matrix for the next level; repeat the above process, and finally get four guide maps;

[0082] (2.4) Design a decoder that concatenates enhanced prototype features, high-level query features, and multi-level guidance maps in the channel dimension at four levels, and uses convolution to deeply fuse the three. Finally, semantic segmentation is achieved through atrous spatial pyramid pooling.

[0083] (3) Using the backbone network in step (2), perform deep feature extraction on the support images and query images in each set of samples to obtain mid- and high-level support features and query features.

[0084] (4) The intermediate support features and query features obtained in step (3) are sent to the prototype enhancement module based on feature customization and activation reconstruction in step (2) to obtain enhanced prototype features, whose internal structure is as follows: Figure 2 As shown, the process of obtaining enhanced prototype features is as follows:

[0085] (4.1) Concatenate the extracted intermediate features in the channel dimension and obtain the intermediate query features through convolution dimensionality reduction. and mid-level support features This operation reduces the features to 256 dimensions, and the convolution kernel size is 1×1;

[0086] (4.2) The intermediate query features and mid-level support features Converted into a two-dimensional feature matrix by reorganization and transposition operations respectively and The intermediate query features Convert to a two-dimensional matrix W by redoing the operation Q , the formula is described as follows:

[0087]

[0088] Among them, conv represents the convolution operation, Γ represents the reorganization and transposition operation, and R represents the reorganization operation;

[0089] (4.3) The prototype enhancement module based on feature customization and activation reconstruction is mainly divided into two parts: feature customization and activation reconstruction. In the feature customization part, the intermediate support features and mid-level query features Associating them, we get the weight matrix w representing the similarity between the two. The formula is as follows:

[0090]

[0091] Among them, Φ represents matrix multiplication and softmax operation. Then use this matrix to calculate the reconstructed query features The formula is as follows:

[0092]

[0093] (4.4) In the activation reconstruction part of the prototype enhancement module, the intermediate query features are pooled to obtain the intermediate query feature vector, and the feature vector and the intermediate support feature vector are calculated. The similarity A between them is as follows:

[0094]

[0095] Among them, avgpool represents the average pooling function, F cosine Represents the cosine similarity calculation function. Assume an input feature map Input, use a pooling window of f×f and a step size s, and for the element Output(i,j) in the output feature map Output, its value is the result after average pooling, and avgpool is determined by the following formula:

[0096]

[0097] Among them, u and v represent the index of the traversal pooling window, i and j represent the position index of the element in the output feature map. In addition, F cosine The cosine similarity calculation formula is as follows:

[0098]

[0099] Among them, A and B represent and vectors of the same size in , i represents the i-th element in the vector, and n represents the length of the vector.

[0100] Then, the reconstructed query features are activated with similarity A and with Combined with refactored support features The formula is as follows:

[0101]

[0102] Here, ⊙ represents the Hadamard product.

[0103] (4.5) Using masked average pooling (MAP), the reconstructed support features Converted to the enhanced prototype feature pro, the formula is described as follows:

[0104]

[0105] Among them, MAP stands for masked average pooling.

[0106] (5) The high-level support features and high-level query features obtained in step (3) are sent to the multi-level guidance-based reinforcement module in step (2) to obtain a multi-level guidance graph. The schematic diagram of the guidance information mining of a certain level in the multi-level guidance reinforcement module is as follows: Figure 3 As shown, the specific process is as follows:

[0107] (5.1) Advanced Support Features and support mask M S Multiply them in the Hadamard product to obtain the effective information matrix S. The formula is as follows:

[0108]

[0109] Among them, ⊙ represents the Hadamard product, and Γ represents the reorganization and transposition operations.

[0110] Then, the advanced query feature Converted to query matrix Q, the formula is as follows:

[0111]

[0112] Among them, R represents the reorganization operation;

[0113] (5.2) Calculate the similarity between the effective information matrix S and the query matrix Q, and normalize them to obtain the guidance graph at the current level. The specific formula is as follows:

[0114]

[0115] sim'=max(sim)

[0116]

[0117] Here, ||·|| represents the norm operation, c is a hyperparameter, which is set to 1e -7 , map is the guide map at the current level;

[0118] (5.3) Multiply the guidance map at the current level by the query matrix Q in a Hadamard manner to obtain the query matrix Q' at the next level:

[0119] Q'=map⊙Q

[0120] (5.4) Advanced Support Features After scaling Q' to the next size, repeat (5.1) to (5.3) to obtain guidance maps at different levels.

[0121] (6) The high-level query features in step (3), the enhanced prototype features in step (4), and the multi-level guidance map in step (5) are sent to the decoder in step (2) to achieve accurate semantic segmentation of the remote sensing image; the decoder structure is designed as follows:

[0122] (6.1) At the current level, the high-level query features, enhanced prototype features and guidance maps are concatenated in the channel dimension and reduced to 256 dimensions through convolution operations with a convolution kernel size of 1×1;

[0123] (6.2) The reduced features are sent to the fusion module to obtain the fine features at the current level; the fusion module uses a 3×3 convolution kernel and a residual structure, and the specific formula is as follows:

[0124]

[0125] Among them, X0 is the feature after dimensionality reduction, conv1 represents 3×3 convolution, is the fine feature obtained at the current level;

[0126] (6.3) The fine feature is concatenated with the feature after dimensionality reduction at the next level, and fused using convolution and residual operations; the convolution uses a 1×1 convolution kernel, and the formula is described as follows:

[0127]

[0128] in, is the fine feature obtained, X i It is the feature after dimensionality reduction at the next level, cat means concatenation in the channel dimension, and conv2 means 1×1 convolution;

[0129] (6.4) Finally, the fine features of the four levels are concatenated in the channel dimension, and the final semantic segmentation result is obtained through dilated spatial pyramid pooling. Specifically, the concatenated features are input into the five branches of dilated spatial pyramid pooling: the first branch uses global average pooling to obtain global features; the second, third, and fourth branches obtain three sets of features of different scales through 1×1 convolution with a dilation rate of 0, 3×3 convolution with a dilation rate of 6, 3×3 convolution with a dilation rate of 12, and 3×3 convolution with a dilation rate of 18 respectively; the output features of the five branches are concatenated in the channel dimension, and the classification function is used to obtain the result. The classification function is:

[0130] class=argmax(S i )

[0131] Among them, S i is the probability that the input sample belongs to the i-th category.

[0132] The present invention uses two different few-sample remote sensing image semantic segmentation algorithms to compare with the proposed method. The comparison algorithms used are:

[0133] The pyramid correlation fusion network using a pixel-level method proposed by Ao et al. in "Few-Shot Aerial Image Semantic Segmentation LeveragingPyramid Correlation Fusion[J].IEEE Transactions on Geoscience and Remote Sensing, 2023." is referred to as Method 1.

[0134] The hypercorrelation squeeze network for constructing 4D convolutional tensor proposed by Min et al. in “Hypercorrelation Squeeze for Few-Shot Segmenation[C].International Conference on Computer Vision(ICCV).IEEE,2021.” is referred to as Method 2.

[0135] Table 1 shows the performance comparison of the three methods on two public few-sample remote sensing image semantic segmentation datasets iSAID and DLRSD. The mIoU index is a commonly used performance metric for few-sample remote sensing image semantic segmentation. The larger the value, the better the performance. It can be seen that the method proposed in this paper has the best effect on remote sensing image semantic segmentation.

[0136] Table 1 Comparison of accuracy indicators of different methods on iSAID and DLRSD datasets (%)

[0137]

Claims

1. A method for semantic segmentation of few-sample remote sensing images based on a dual-branch reinforcement network includes six steps, which are characterized by: (1) Construct a few-sample remote sensing image semantic segmentation dataset and divide the dataset into a training set D train and the test set D test , D train and D test They are composed of several groups of sample sets; (2) Construct a query image-guided dual-branch enhanced few-shot remote sensing image semantic segmentation network. The network mainly consists of four parts: a backbone network, a prototype enhancement module based on feature customization and activation reconstruction, an enhancement module based on multi-level guidance, and a decoder. The network is trained on the training set D in step (1). train Train on the test set D test Test on; (3) Using the backbone network in step (2), deep feature extraction is performed on the support images and query images in each set of samples to obtain medium and high-level support features and query features; (4) sending the intermediate support features and the intermediate query features obtained in step (3) to the prototype enhancement module based on feature customization and activation reconstruction in step (2) to obtain enhanced prototype features; (5) sending the high-level support features and high-level query features obtained in step (3) to the multi-level guidance-based reinforcement module in step (2) to obtain a multi-level guidance graph; (6) The high-level query features in step (3), the enhanced prototype features in step (4), and the multi-level guidance map in step (5) are fed into the decoder in step (2) to achieve accurate semantic segmentation of the remote sensing image.

2. According to the method for semantic segmentation of few-sample remote sensing images based on a dual-branch reinforcement network according to claim 1, in step (1), the method for constructing a semantic segmentation dataset of few-sample remote sensing images is as follows: (1.1) Construct a few-sample remote sensing image semantic segmentation dataset Image = [Image1,…,Image i ,…,Image N ], and make the corresponding sample label Label = [Label1,…,Label i ,…,Label N ], where N represents N remote sensing images, Image i Represents the i-th remote sensing image, Label i Indicates the label corresponding to the i-th remote sensing image; (1.2) Divide the dataset Image into a training set D train and the test set D test , assuming that the total number of samples is n, m images containing the training category are selected to construct the training set, and the remaining nm images are selected to construct the test set; then, the training set and the test set are randomly divided into several groups of sample sets according to the target category, and each group of sample sets is represented as in, Indicated by the supporting image and corresponding tags The support set is composed of K, K represents the K-shot task; {(Q q ,M q )} represents the query image Q q and the corresponding label M q The query set formed.

3. According to the method for semantic segmentation of few-sample remote sensing images based on a dual-branch enhanced network according to claim 1, in step (2), the overall architecture of the dual-branch enhanced few-sample remote sensing image semantic segmentation network guided by the query image is designed as follows: (2.1) Build a pyramid scene parsing network (PSPNet) based on VGG-16 and ResNet-50 as the backbone network, and extract mid- and high-level features of the support image and the query image based on this network; (2.2) Design prototype branch. The core of this branch is the prototype enhancement module based on feature customization and activation reconstruction. This module is divided into two parts: feature customization and activation reconstruction. In the feature customization part, the intermediate support features are and mid-level query features Associating them, we get the weight matrix w representing the similarity between the two, and use this matrix to get the reconstructed query features In the Activate Reconstruction section, change the mid-level query feature Pooling is performed to obtain the intermediate query feature vector, and the feature vector and the intermediate support feature vector are calculated. The similarity of Then the reconstructed query feature will be activated and Further combined to obtain the reconstructed support features Finally, the mask M will be supported S and refactoring support features Through mask average pooling (MAP), the enhanced prototype features can be finally obtained; (2.3) Design the guidance branch. The core of this branch is a reinforcement module based on multi-level guidance. This module consists of four guidance information mining operations at different levels: First, through high-level support features and support mask M S Calculate the effective information matrix S and transform the advanced query features Convert it into a query matrix Q; then at each level, calculate the similarity matrix between S and Q to obtain the guide map at the current level, and fuse the guide map at the current level with the query matrix Q to obtain Q' as the query matrix for the next level; Repeat the above process, and finally you can get four guide maps; (2.4) Design a decoder that concatenates enhanced prototype features, high-level query features, and multi-level guidance maps in the channel dimension at four levels, and uses convolution to deeply fuse the three. Finally, semantic segmentation is achieved through atrous spatial pyramid pooling.

4. According to the method for semantic segmentation of few-sample remote sensing images based on a dual-branch reinforcement network of claim 1, in step (4), the specific process of obtaining the enhanced prototype features based on the prototype reinforcement module of feature customization and activation reconstruction is as follows: (4.1) Concatenate the extracted intermediate features in the channel dimension and obtain the intermediate query features through convolution dimensionality reduction. and mid-level support features (4.2) The intermediate query features and mid-level support features Converted into a two-dimensional feature matrix by reorganization and transposition operations respectively and The intermediate query features Convert to a two-dimensional matrix W by redoing the operation Q , the formula is described as follows: in, conv represents the convolution operation, Γ represents the reorganization and transposition operation, and R represents the reorganization operation; (4.3) The prototype enhancement module based on feature customization and activation reconstruction is mainly divided into two parts: feature customization and activation reconstruction. In the feature customization part, the intermediate support features and mid-level query features Associating them, we get the weight matrix w representing the similarity between the two. The calculation formula is as follows: Among them, Φ represents matrix multiplication and softmax operation. Then use this matrix to calculate the reconstructed query features The formula is as follows: (4.4) In the activation reconstruction part of the prototype enhancement module, the intermediate query features are pooled to obtain the intermediate query feature vector, and the feature vector and the intermediate support feature vector are calculated. The similarity A between them is expressed as follows: Among them, avgpool represents the average pooling function, F cosine Represents the cosine similarity calculation function. Then, the reconstructed query features are activated with similarity A And compare the results with Combined with refactored support features The formula is as follows: Here, ⊙ represents the Hadamard product. (4.5) Using masked average pooling (MAP), the reconstructed support features Converted to the enhanced prototype feature pro, the formula is described as follows: Among them, MAP stands for masked average pooling.

5. According to the method for semantic segmentation of few-sample remote sensing images based on a dual-branch reinforcement network of claim 1, in step (5), the specific process of obtaining the guidance map based on the reinforcement module guided by multiple levels is as follows: (5.1) Advanced Support Features and support mask M S Multiply them in the Hadamard product to get the effective information matrix S. The formula is as follows: in, ⊙ represents the Hadamard product, Γ represents the reorganization and transposition operations; Then, the advanced query feature Converted to query matrix Q, the formula is as follows: Among them, R represents the reorganization operation; (5.2) Calculate the similarity between the effective information matrix S and the query matrix Q, and normalize them to obtain the guidance graph at the current level. The specific formula is as follows: sim′=max(sim) Among them, ||·|| represents the norm operation, c is a hyperparameter, and map is the guide map at the current level; (5.3) Multiply the guidance map at the current level by the query matrix Q in a Hadamard manner to obtain the query matrix Q' at the next level: Q'=map⊙Q (5.4) Advanced Support Features After scaling Q' to the next size, repeat steps (5.1) to (5.3) to obtain guidance maps at different levels.

6. According to the method for semantic segmentation of few-sample remote sensing images based on a dual-branch reinforcement network according to claim 1, in step (6), the decoder structure design method is as follows: (6.1) At the current level, the high-level query features, enhanced prototype features and guidance maps are concatenated in the channel dimension and reduced in dimension through convolution operations; (6.2) Send the reduced-dimensional features to the fusion module to obtain the fine features at the current level; (6.3) The fine features are concatenated with the features after dimension reduction at the next level, and fused using convolution and residual operations; (6.4) Finally, the fine features at the four levels are concatenated in the channel dimension, and the final semantic segmentation result is obtained through atrous spatial pyramid pooling.

Images