Few-shot remote sensing image semantic segmentation method based on double-branch reinforcement network

Abstract

The application discloses a few-shot remote sensing image semantic segmentation method based on a double-branch reinforcement network. First, a backbone network is built to extract middle and high-level support features and query features from support images and query images respectively; second, a prototype reinforcement module based on feature customization and activation reconstruction is constructed, the middle-level support features and the query features are input into the module to obtain reinforced prototype features; then, a reinforcement module based on multi-level guidance is proposed, the high-level support features and the query features are input into the module to obtain a multi-level guidance map; finally, an encoder is designed to fuse the reinforced prototype features, the high-level query features and the multi-level guidance map to obtain a final segmentation result. The prototype reinforcement module designed by the application can deeply depict semantic features of target categories and improve target expression capability; meanwhile, the multi-level guidance reinforcement module proposed by the application can capture correlations between different levels of images, enhance the overall framework through unbiased knowledge and obtain good segmentation performance.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and in particular relates to a few-shot semantic segmentation method for remote sensing images based on a dual-branch reinforcement network. Background Technology

[0002] Semantic segmentation of remote sensing images is an important research direction in the field of remote sensing image processing. It refers to the pixel-level classification of images containing semantic information, that is, dividing each pixel in the image according to its category to achieve a deeper understanding of the content of the remote sensing image. Semantic segmentation of remote sensing images plays a crucial role in many fields, such as urban planning, agricultural monitoring, and disaster assessment. Furthermore, it is also significant in decision support, automated processing, and technological applications. In recent years, deep learning technology has experienced explosive growth and has been widely used in the field of semantic segmentation of remote sensing images. However, training deep neural networks usually requires a large amount of finely labeled sample data, which is extremely time-consuming and labor-intensive for pixel-level data labeling tasks in semantic segmentation of remote sensing images.

[0003] To address this, few-shot segmentation has been proposed, aiming to solve the problem of semantic segmentation of images with only a small number of labeled samples. Compared with traditional remote sensing image semantic segmentation methods, few-shot remote sensing image semantic segmentation attempts to achieve better segmentation results by utilizing a small number of samples. Currently, few-shot remote sensing image semantic segmentation methods are mainly divided into two categories: (1) prototype-level methods; (2) pixel-level methods.

[0004] Publication number CN118154427A introduces a few-shot remote sensing image segmentation method, which includes spectral super-resolution reconstruction, image attention feature extraction, feature concatenation, and decoder decoding. First, spectral super-resolution reconstruction is achieved through a dynamic convolutional multi-column network. The spectral super-resolution reconstruction features are then subjected to image spatial attention re-extraction and spectral attention re-extraction, respectively, and the features extracted from the two attention processes are concatenated. Next, the segmentation result is obtained through a few-shot remote sensing image semantic segmentation decoder and a spectral super-resolution reconstruction decoder. This method utilizes discernible spectral information from hyperspectral images, improving segmentation performance. However, this method involves conversion between hyperspectral images and three-channel images, resulting in problems such as information loss, band selection, and color distortion. Therefore, its ability to represent diverse and complex targets in remote sensing images remains quite limited.

[0005] In 2023, Ao et al. published a paper titled "Few-Shot Aerial Image Semantic Segmentation Leveraging Pyramid Correlation Fusion" in IEEE Transactions on Geoscience and Remote Sensing. This paper proposes a pixel-level pyramidal correlation fusion network for few-shot remote sensing image segmentation tasks. It calculates pixel-level semantic correspondences by using cosine similarity and Euclidean distance, and introduces a scale-aware cross-entropy loss function to improve the segmentation accuracy of small targets. However, the scale-aware cross-entropy function in this method focuses primarily on target size, resulting in poor segmentation performance for complex-shaped objects. Furthermore, the method involves calculating correlation graphs during segmentation, making it susceptible to noise interference.

[0006] In summary, the main limitations of general few-sample remote sensing image semantic segmentation methods are as follows:

[0007] (1) Poor ability to map target objects: Most mainstream methods extract features from supporting images, ignoring the rich category information in the query image; at the same time, pixel-level methods are easily affected by irrelevant information in the image, resulting in limited effective information obtained.

[0008] (2) Poor generalization performance of the method: Most current methods fail to fully consider the characteristics of remote sensing images, such as strong intra-class heterogeneity and high degree of confusion between different classes. A large amount of target detail information will be lost during the processing, resulting in poor generalization effect. Summary of the Invention

[0009] Objective: To address the problems existing in the prior art, this invention provides a few-shot semantic segmentation method for remote sensing images based on a dual-branch reinforcement network. The prototype reinforcement module designed in this method can deeply characterize the semantic features of the target category, thereby improving the target's expressive ability. Simultaneously, the proposed multi-level guided reinforcement module can capture the correlation between images at different levels, enhancing the overall framework through unbiased knowledge. This not only avoids the problem of increased parameter count but also enhances the model's generalization ability from a priori perspective, achieving good segmentation performance.

[0010] Technical Solution: To achieve the objectives of this invention, the technical solution adopted is: a few-shot remote sensing image semantic segmentation method based on a dual-branch reinforcement network, comprising six steps, as follows:

[0011] (1) Construct a few-sample remote sensing image semantic segmentation dataset, and divide the dataset into a training set D. train and test set D test D trainand D test Each consists of several sets of samples;

[0012] (2) Construct a query-guided dual-branch enhanced few-shot remote sensing image semantic segmentation network. This network mainly consists of four parts: a backbone network, a prototype enhancement module based on feature customization and activation reconstruction, a multi-level guided enhancement module, and a decoder. The training set D of this network in step (1) is used for the following steps. train Training is performed on the test set D. test Test on;

[0013] (3) Using the backbone network in step (2), deep feature extraction is performed on the support images and query images in each set of samples to obtain intermediate and high-level support features and query features.

[0014] (4) Send the intermediate support features and query features obtained in step (3) into the prototype enhancement module based on feature customization and activation reconstruction in step (2) to obtain enhanced prototype features.

[0015] (5) Input the advanced support features and advanced query features obtained in step (3) into the multi-level guidance-based reinforcement module in step (2) to obtain the multi-level guidance graph;

[0016] (6) Input the advanced query features in step (3), the enhanced prototype features in step (4), and the multi-level guide map in step (5) into the decoder in step (2) to achieve accurate semantic segmentation of the remote sensing image.

[0017] The method for constructing a few-sample remote sensing image semantic segmentation dataset in step (1) is as follows:

[0018] (1.1) Construct a few-sample remote sensing image semantic segmentation dataset Image = [Image1, ..., Image2] i Image N ], and create corresponding sample labels Label = [Label1, ..., Label i ,…,Label N ], where N represents N remote sensing images, Image i Label represents the i-th remote sensing image. i This represents the label corresponding to the i-th remote sensing image;

[0019] (1.2) Divide the dataset Image into training set D train and test set D testLet the total number of samples be n. We select m images containing the training category to construct the training set, and the remaining n images to construct the test set. Subsequently, both the training and test sets are randomly divided into several sample sets according to the target category. Each sample set is represented as... in, Indicates the supporting image and corresponding tags The support set is formed, where K represents the K-shot task; {(Q q M q )} represents the query image Q q and corresponding tag M q The query set that is formed.

[0020] In step (2), the overall architecture design of the query image-guided dual-branch enhanced few-shot remote sensing image semantic segmentation network is as follows:

[0021] (2.1) Construct a pyramid scene parsing network (PSPNet) based on VGG-16 and ResNet-50 as the backbone network, and extract mid-to-high-level features of the supporting image and the query image based on this network;

[0022] (2.2) Design the prototype branch, the core of which is a prototype enhancement module based on feature customization and activation refactoring. This module is divided into two parts: feature customization and activation refactoring. In the feature customization part, intermediate support features will be implemented. and intermediate query features By performing the association, a weight matrix w representing the similarity between the two is obtained. This matrix is ​​then used to obtain the reconstructed query features. In the activation and reconstruction section, intermediate query features will be... Pooling is performed to obtain the intermediate query feature vector, and the feature vector is then compared with the intermediate support features. Activating the reconstructed query features based on similarity Then activate the reconstructed query features. and Further combining the reconstructed support features Finally, mask M will be supported. S and refactoring support features Through masked average pooling (MAP), the enhanced prototype features can be obtained in the end.

[0023] (2.3) Design the guiding branch. The core of this branch is a reinforcement module based on multi-level guidance. This module consists of four guiding information mining operations at different levels: First, through high-level support features and support mask M S Calculate the effective information matrix S, and simultaneously incorporate advanced query features. The process is then transformed into a query matrix Q. Next, at each level, the similarity matrix between S and Q is calculated to obtain the guide map at the current level. The guide map at the current level is then fused with the query matrix Q to obtain Q', which serves as the query matrix for the next level. This process is repeated until four guide maps are obtained.

[0024] (2.4) Design a decoder that concatenates enhanced prototype features, advanced query features and multi-level guide maps in the channel dimension at four levels, and uses convolution to deeply fuse the three. Finally, semantic segmentation is achieved through hollow spatial pyramid pooling.

[0025] In step (4), the specific process by which the prototype enhancement module based on feature customization and activation reconstruction obtains enhanced prototype features is as follows:

[0026] (4.1) The intermediate-level features extracted along the channel dimension are concatenated, and the intermediate-level query features are obtained by dimensionality reduction through convolution. and intermediate support features This operation reduces the features to 256 dimensions, and the convolutional kernel size is 1×1;

[0027] (4.2) Incorporate intermediate query features and intermediate support features Transformed into a two-dimensional feature matrix through recombination and transpose operations respectively. and Intermediate query features Transformed into a two-dimensional matrix W through recombination operations. Q The formula is described as follows:

[0028]

[0029] Where conv represents the convolution operation, Γ represents the recombination and transpose operation, and R represents the recombination operation;

[0030] (4.3) The prototype enhancement module based on feature customization and activation reconstruction is mainly divided into two parts: feature customization and activation reconstruction. In the feature customization part, intermediate support features are... and intermediate query features By performing the association, we obtain the weight matrix w representing the similarity between the two, as described in the following formula:

[0031]

[0032] Here, Φ represents matrix multiplication and softmax operations. The reconstructed query features are then calculated using this matrix. The formula is expressed as follows:

[0033]

[0034] (4.4) In the activation and reconstruction part of the prototype reinforcement module, the intermediate query features are pooled to obtain the intermediate query feature vector, and the feature vector and the intermediate support features are calculated. The similarity A between them is calculated using the following formula:

[0035]

[0036] Where avgpool represents the average pooling function, F cosine Let `avgpool` represent the cosine similarity calculation function. Given an input feature map `Input`, a pooling window of size `f×f`, and a stride `s`, the value of an element `Output(i,j)` in the output feature map `Output` is the result of average pooling. `avgpool` is determined by the following formula:

[0037]

[0038] Where u and v represent the indices of the pooling window, and i and j represent the position indices of elements in the output feature map. Furthermore, F cosine The formula for calculating the cosine similarity is as follows:

[0039]

[0040] Where A and B represent respectively and In a vector, i represents the i-th element and n represents the length of the vector.

[0041] Subsequently, the reconstructed query features were activated using similarity A. and Combined with the reconstructed supporting features The formula is as follows:

[0042]

[0043] Where ⊙ represents the Hadamarda.

[0044] (4.5) Using Masked Average Pooling (MAP), the reconstructed supporting features are... The transformed prototype feature 'pro' is described by the following formula:

[0045]

[0046] MAP stands for Mask Average Pooling.

[0047] The specific process of obtaining the guidance graph based on the multi-level guidance enhancement module in step (5) is as follows:

[0048] (5.1) Advanced support features and support mask M S The effective information matrix S is obtained by multiplying the results using the Hadamard product, as described in the following formula:

[0049]

[0050] Where ⊙ represents the Hadamard product, and Γ represents the recombination and transpose operations.

[0051] Then, advanced query features Convert to the query matrix Q, as shown in the following formula:

[0052]

[0053] Where R represents a recombination operation;

[0054] (5.2) Calculate the similarity between the effective information matrix S and the query matrix Q, and normalize it to obtain the guidance graph at the current level. The specific formula is as follows:

[0055]

[0056] sim′=max(sim)

[0057]

[0058] Where ||·|| denotes the norm operation, and c is a hyperparameter set to 1e. -7 `map` is the guide map at the current level;

[0059] (5.3) Multiply the current level's guidance graph (map) with the query matrix Q using the Hadamard method to obtain the next level's query matrix Q':

[0060] Q'=map⊙Q

[0061] (5.4) Advanced support features After scaling Q' to the next size, repeat steps (5.1) to (5.3) to obtain guide maps at different levels.

[0062] In step (6), the decoder structure is designed as follows:

[0063] (6.1) At the current level, the advanced query features, enhanced prototype features and guide graph are concatenated in the channel dimension and reduced to 256 dimensions through convolution operation with a convolution kernel size of 1×1.

[0064] (6.2) The dimensionality-reduced features are fed into the fusion module to obtain the refined features at the current level; the fusion module uses 3×3 convolution kernels and residual structures;

[0065] (6.3) The refined feature is concatenated with the feature after the next level of dimensionality reduction, and then fused using convolution and residual operations; the convolution uses a 1×1 convolution kernel;

[0066] (6.4) Finally, the fine features at the four levels are concatenated along the channel dimension, and the final semantic segmentation result is obtained by hollow space pyramid pooling.

[0067] Beneficial effects: The present invention, by adopting the above technical solution, has the following beneficial effects:

[0068] (1) The prototype enhancement module in the method of the present invention integrates the query features into the support features, so that the prototype features generated can effectively capture the intrinsic characteristics and details of the target category, and more deeply depict the semantic features of the target category, thereby showing stronger expressiveness and higher accuracy when segmenting the target, and also stronger expressive ability of the target.

[0069] (2) The multi-level guided reinforcement module in this method deeply explores the correlation of images at multiple abstract levels (layers) and constructs unbiased knowledge to provide the network with richer and more comprehensive information. This multi-level guidance and reinforcement not only improves the accuracy of segmentation, but also significantly enhances the generalization ability of the network. Attached Figure Description

[0070] Figure 1 This is a diagram of the semantic segmentation network architecture for few-shot remote sensing images based on a dual-branch reinforcement network.

[0071] Figure 2 This is a diagram of the internal structure of the prototype enhancement module;

[0072] Figure 3 This is a schematic diagram illustrating the mining of guidance information at a specific level within a multi-level guidance enhancement module. Detailed Implementation

[0073] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0074] like Figure 1 As shown, the technical solution of the present invention is further described in detail below:

[0075] (1) Construct a few-sample remote sensing image semantic segmentation dataset, and divide the dataset into a training set D. train and test set D test D train and D test Each consists of several sample sets:

[0076] (1.1) Construct a few-sample remote sensing image semantic segmentation dataset Image = [Image1, ..., Image2] iImage N ], and create corresponding sample labels Label = [Label1, ..., Label i ,…,Label N ], where N represents N remote sensing images, Image i Label represents the i-th remote sensing image. i This represents the label corresponding to the i-th remote sensing image;

[0077] (1.2) Divide the dataset Image into training set D train and test set D test Let the total number of samples be n. We select m images containing the training category to construct the training set, and the remaining n images to construct the test set. Subsequently, both the training and test sets are randomly divided into several sample sets according to the target category, each sample set denoted as [image 1]. in, Indicates the supporting image and corresponding tags The support set is formed, where K represents the K-shot task; {(Q q M q )} represents the query image Q q and corresponding tag M q The query set is constructed. This invention uses the few-sample remote sensing image semantic segmentation public datasets iSAID and DLRSD for experiments. These two datasets contain 15 and 17 categories, respectively.

[0078] (2) A query-guided, dual-branch enhanced few-shot remote sensing image semantic segmentation network is constructed. This network mainly consists of four parts: a backbone network, a prototype enhancement module based on feature customization and activation reconstruction, a multi-level guided enhancement module, and a decoder. The specific structure is as follows: Figure 1 As shown, the overall network workflow is as follows:

[0079] (2.1) Construct a pyramid scene parsing network (PSPNet) based on VGG-16 and ResNet-50 as the backbone network, and extract mid-to-high-level features of the supporting image and the query image based on this network;

[0080] (2.2) Design the prototype branch, the core of which is a prototype enhancement module based on feature customization and activation refactoring. This module is divided into two parts: feature customization and activation refactoring. In the feature customization part, intermediate support features will be implemented. and intermediate query features By performing the association, a weight matrix w representing the similarity between the two is obtained. This matrix is ​​then used to obtain the reconstructed query features. In the activation and reconstruction section, intermediate query features will be... Pooling is performed to obtain the intermediate query feature vector, and the feature vector is then compared with the intermediate support features. Activating the reconstructed query features based on similarity Then activate the reconstructed query features. and Further combining the reconstructed support features Finally, mask M will be supported. S and refactoring support features Through masked average pooling (MAP), the enhanced prototype features can be obtained in the end.

[0081] (2.3) Design the guiding branch. The core of this branch is a reinforcement module based on multi-level guidance. This module consists of four guiding information mining operations at different levels: First, through high-level support features and support mask M S Calculate the effective information matrix S, and simultaneously incorporate advanced query features. The process is then transformed into a query matrix Q. Next, at each level, the similarity matrix between S and Q is calculated to obtain the guide map at the current level. The guide map at the current level is then fused with the query matrix Q to obtain Q', which serves as the query matrix for the next level. This process is repeated until four guide maps are obtained.

[0082] (2.4) Design a decoder that concatenates enhanced prototype features, advanced query features and multi-level guide maps in the channel dimension at four levels, and uses convolution to deeply fuse the three. Finally, semantic segmentation is achieved through hollow spatial pyramid pooling.

[0083] (3) Using the backbone network in step (2), perform deep feature extraction on the support images and query images in each set of samples to obtain intermediate and advanced support features and query features.

[0084] (4) Input the intermediate support features and query features obtained in step (3) into the prototype enhancement module based on feature customization and activation reconstruction in step (2) to obtain the enhanced prototype features, the internal structure of which is as follows: Figure 2 As shown, the process for obtaining enhanced prototype features is as follows:

[0085] (4.1) The intermediate-level features extracted along the channel dimension are concatenated, and the intermediate-level query features are obtained by dimensionality reduction through convolution. and intermediate support features This operation reduces the features to 256 dimensions, and the convolutional kernel size is 1×1;

[0086] (4.2) Incorporate intermediate query features and intermediate support features Transformed into a two-dimensional feature matrix through recombination and transpose operations respectively. and Intermediate query features Transformed into a two-dimensional matrix W through recombination operations. Q The formula is described as follows:

[0087]

[0088] Where conv represents the convolution operation, Γ represents the recombination and transpose operation, and R represents the recombination operation;

[0089] (4.3) The prototype enhancement module based on feature customization and activation reconstruction is mainly divided into two parts: feature customization and activation reconstruction. In the feature customization part, intermediate support features are... and intermediate query features By performing the association, we obtain the weight matrix w representing the similarity between the two, as described in the following formula:

[0090]

[0091] Here, Φ represents matrix multiplication and softmax operations. The reconstructed query features are then calculated using this matrix. The formula is expressed as follows:

[0092]

[0093] (4.4) In the activation and reconstruction part of the prototype reinforcement module, the intermediate query features are pooled to obtain the intermediate query feature vector, and the feature vector and the intermediate support features are calculated. The similarity A between them is calculated using the following formula:

[0094]

[0095] Where avgpool represents the average pooling function, F cosine Let `avgpool` represent the cosine similarity calculation function. Given an input feature map `Input`, a pooling window of size `f×f`, and a stride `s`, the value of an element `Output(i,j)` in the output feature map `Output` is the result of average pooling. `avgpool` is determined by the following formula:

[0096]

[0097] Where u and v represent the indices of the pooling window, and i and j represent the position indices of elements in the output feature map. Furthermore, F cosine The formula for calculating the cosine similarity is as follows:

[0098]

[0099] Where A and B represent respectively and In a vector, i represents the i-th element and n represents the length of the vector.

[0100] Subsequently, the reconstructed query features were activated using similarity A. and Combined with the reconstructed supporting features The formula is as follows:

[0101]

[0102] Where ⊙ represents the Hadamarda.

[0103] (4.5) Using Masked Average Pooling (MAP), the reconstructed supporting features are... The transformed prototype feature 'pro' is described by the following formula:

[0104]

[0105] MAP stands for Mask Average Pooling.

[0106] (5) Input the advanced support features and advanced query features obtained in step (3) into the multi-level guidance-based reinforcement module in step (2) to obtain the multi-level guidance graph. The schematic diagram of guidance information mining at a certain level in the multi-level guidance reinforcement module is shown below. Figure 3 As shown, the specific process is as follows:

[0107] (5.1) Advanced support features and support mask M S The effective information matrix S is obtained by multiplying the results using the Hadamard product, as described in the following formula:

[0108]

[0109] Where ⊙ represents the Hadamard product, and Γ represents the recombination and transpose operations.

[0110] Then, advanced query features Convert to the query matrix Q, as shown in the following formula:

[0111]

[0112] Where R represents a recombination operation;

[0113] (5.2) Calculate the similarity between the effective information matrix S and the query matrix Q, and normalize it to obtain the guidance graph at the current level. The specific formula is as follows:

[0114]

[0115] sim' = max(sim)

[0116]

[0117] Where ||·|| denotes the norm operation, and c is a hyperparameter set to 1e. -7 `map` is the guide map at the current level;

[0118] (5.3) Multiply the current level's guidance graph (map) with the query matrix Q using the Hadamard method to obtain the next level's query matrix Q':

[0119] Q'=map⊙Q

[0120] (5.4) Advanced support features After scaling Q' to the next size, repeat steps (5.1) to (5.3) to obtain guide maps at different levels.

[0121] (6) The advanced query features in step (3), the enhanced prototype features in step (4), and the multi-level guide map in step (5) are fed into the decoder in step (2) to achieve accurate semantic segmentation of the remote sensing image; the decoder structure is designed as follows:

[0122] (6.1) At the current level, the advanced query features, enhanced prototype features and guide graph are concatenated in the channel dimension and reduced to 256 dimensions through convolution operation with a convolution kernel size of 1×1.

[0123] (6.2) The dimensionality-reduced features are fed into the fusion module to obtain the refined features at the current level; the fusion module uses a 3×3 convolution kernel and a residual structure, and the specific formula is as follows:

[0124]

[0125] Where X0 represents the reduced-dimensional features, and conv1 represents a 3×3 convolution. These are the refined features obtained at the current level;

[0126] (6.3) This refined feature is concatenated with the feature after the next level of dimensionality reduction, and then fused using convolution and residual operations; the convolution uses a 1×1 kernel, and the formula is described as follows:

[0127]

[0128] in, To obtain the fine features, X i For the features after dimensionality reduction at the next level, cat indicates concatenation along the channel dimension, and conv2 indicates 1×1 convolution;

[0129] (6.4) Finally, the fine features at the four levels are concatenated along the channel dimension, and the final semantic segmentation result is obtained through dilated spatial pyramid pooling. Specifically, the concatenated features are input into the five branches of dilated spatial pyramid pooling: the first branch uses global average pooling to obtain global features; the second, third, and fourth branches obtain three sets of features at different scales through 1×1 convolution with a dilation rate of 0, 3×3 convolution with a dilation rate of 6, 3×3 convolution with a dilation rate of 12, and 3×3 convolution with a dilation rate of 18, respectively; the output features of the five branches are concatenated along the channel dimension, and the result is obtained using a classification function, which is:

[0130] class = argmax(S i )

[0131] Among them, S i Let be the probability that the input sample belongs to the i-th class.

[0132] This invention compares the proposed method with two different few-sample remote sensing image semantic segmentation algorithms. The comparison algorithms selected are:

[0133] Ao et al. proposed a pixel-level pyramid correlation fusion network, referred to as Method 1, in "Few-Shot Aerial Image Semantic Segmentation Leveraging Pyramid Correlation Fusion [J]. IEEE Transactions on Geoscience and Remote Sensing, 2023".

[0134] Min et al. proposed a hypercorrelation squeezing network for constructing 4D convolutional tensors in “Hypercorrelation Squeeze for Few-Shot Segmentation[C]. International Conference on Computer Vision (ICCV). IEEE, 2021.”, referred to as Method 2.

[0135] Table 1 compares the performance of the three methods on two publicly available few-shot remote sensing image semantic segmentation datasets, iSAID and DLRSD. The mIoU metric is a commonly used performance measure for few-shot remote sensing image semantic segmentation; a higher value indicates better performance. Therefore, the method proposed in this invention achieves the best results in remote sensing image semantic segmentation.

[0136] Table 1. Comparison of accuracy metrics (%) of different methods on the iSAID and DLRSD datasets.

[0137]

Claims

1. A few-shot remote sensing image semantic segmentation method based on double-branch reinforcement network, comprising six steps, characterized in that: (1) Construct a few-shot remote sensing image semantic segmentation dataset, divide the dataset into a training set D train and a test set D test , D train and D test each consist of several groups of sample sets; (2) Construct a dual-branch reinforcement few-shot remote sensing image semantic segmentation network based on query image guidance, which mainly includes four parts: backbone network, prototype reinforcement module based on feature customization and activation reconstruction, reinforcement module based on multi-level guidance, and decoder. The network is trained in the training set D train in step (1), and tested in the test set D test . (3) The backbone network in step (2) is used to extract deep features of support images and query images in each sample set to obtain middle and high-level support features and middle and high-level query features; (4) The middle-level support features and middle-level query features obtained in step (3) are sent to the prototype reinforcement module in step (2) based on feature customization and activation reconstruction to obtain reinforced prototype features; (5) The high-level support features and high-level query features obtained in step (3) are sent to the reinforcement module in step (2) based on multi-level guidance to obtain multi-level guidance maps; (6) The high-level query features in step (3), the reinforced prototype features in step (4) and the multi-level guidance maps in step (5) are sent to the decoder in step (2) to realize accurate semantic segmentation of remote sensing images.

2. The few-shot remote sensing image semantic segmentation method based on the dual-branch reinforced network according to claim 1, characterized in that, The method for constructing a few-shot remote sensing image semantic segmentation dataset in step (1) is as follows: (1.1) Construct a few-shot remote sensing image semantic segmentation dataset Image = [Image1, …, Image i , …, Image N ], and make the corresponding sample label Label = [Label1, …, Label i , …, Label N ], where N represents N remote sensing images, Image i represents the i-th remote sensing image, and Label i represents the label corresponding to the i-th remote sensing image; (1.2) Divide the dataset Image into training set D train and testing set D test , let the total number of samples be n, select m images containing training classes to construct the training set, and the remaining n-m images to construct the testing set; then, the training set and the testing set are randomly divided into several groups of sample sets according to the target class, each group of sample sets is represented as wherein, represents a support set composed of support images and corresponding labels , K represents a K-shot task; {(Q q ,M q )} represents a query set composed of query images Q q and corresponding labels M q .

3. The dual-branch reinforced network-based few-shot remote sensing image semantic segmentation method according to claim 1, characterized in that, The overall architecture design of the double-branch reinforcement few-shot remote sensing image semantic segmentation network based on query image guidance in step (2) is as follows: (2.1) A pyramid scene parsing network (PSPNet) based on VGG-16 and ResNet-50 is built as a backbone network to extract middle and high-level features of support images and query images based on the network; (2.2) Design a prototype branch, the core of which is a prototype enhancement module based on feature customization and activation reconstruction: the module is divided into two parts: feature customization and activation reconstruction. In the feature customization part, the intermediate support features and the intermediate query features are associated to obtain a weight matrix w representing the similarity between the two, and the reconstructed query features are obtained using the matrix. In the activation reconstruction part, the intermediate query features are pooled to obtain an intermediate query feature vector, and the similarity between the feature vector and the intermediate support features is calculated to activate the reconstructed query features Then the activated reconstructed query features are further combined with to obtain the reconstructed support features Finally, the support mask M S and the reconstructed support features are averaged by mask pooling (MAP), and the enhanced prototype features can be finally obtained. (2.3) Design a guidance branch, the core of which is a multi-level guidance-based reinforcement module, which is composed of four guidance information mining operations at different levels: first, through the high-level support features and support mask M S Calculate the effective information matrix S, while converting the high-level query features into a query matrix Q; then at each level, calculate the similarity matrix between S and Q to obtain the guidance map at the current level, and fuse the guidance map at the current level with the query matrix Q to obtain Q' as the query matrix at the next level; The above process is repeated, and finally four guidance maps can be obtained; (2.4) A decoder is designed to splice the reinforced prototype features, high-level query features and multi-level guidance maps in the channel dimension at four levels, and use convolution to deeply fuse the three, and finally realize semantic segmentation through an empty spatial pyramid pooling.

4. The few-shot remote sensing image semantic segmentation method based on the dual-branch reinforced network according to claim 1, characterized in that, The specific process of obtaining reinforced prototype features by the prototype reinforcement module based on feature customization and activation reconstruction in step (4) is as follows: (4.1) concatenating the extracted intermediate-level features in the channel dimension, and respectively obtaining middle-level query features and middle-level support features by convolutional dimension reduction and middle-level support features (4.2) The mid-level query features and the mid-level support features are converted into two-dimensional feature matrices by recombination and transposition operations, respectively and and the mid-level query features are converted into a two-dimensional matrix W by a recombination operation Q The formula is described as follows: Wherein, conv represents convolution operation, Γ represents reorganization and transposition operation, and R represents reorganization operation; (4.3) The prototype reinforcement module based on feature customization and activation reconstruction is mainly divided into two parts: feature customization and activation reconstruction. In the feature customization part, the intermediate support features and the intermediate query features are associated to obtain a weight matrix w representing the similarity between the two, and the calculation formula is as follows: where Φ denotes matrix multiplication and softmax operation, and then the matrix is used to calculate the reconstructed query feature The formula is expressed as follows: (4.4) In the activation reconstruction part of the prototype reinforcement module, the intermediate query feature is obtained by pooling, and the similarity A between the feature vector and the intermediate support feature is calculated, which is expressed by the following formula: wherein avgpool denotes an average pooling function, F cosine denotes a cosine similarity computation function; Subsequently, the reconstructed query feature is activated with the similarity A and the results are combined with the reconstructed support feature The formula is as follows: Wherein, represents Hadamard product; (4.5) With Masked Average Pooling (MAP), the reconstructed support features are converted into boosted prototype features pro, described by the following formula: wherein MAP denotes mask average pooling, M S denotes support mask.

5. The few-shot remote sensing image semantic segmentation method based on the dual-branch reinforced network according to claim 1, characterized in that, The specific process of obtaining guidance maps by the reinforcement module based on multi-level guidance in step (5) is as follows: (5.1) Advanced support features and support mask M S are multiplied in Hadamard product fashion to obtain the valid information matrix S, described by the equation: Wherein, represents Hadamard product, and Γ represents reorganization and transposition operation; Then, the advanced query feature is converted into a query matrix Q, as follows: Wherein, R represents reorganization operation; (5.2) The similarity between the effective information matrix S and the query matrix Q is calculated and normalized to obtain the guidance map at the current level, and the specific formula is as follows: sim'=max(sim) Wherein, sim represents similarity, ||·|| represents norm operation, c is a hyperparameter, and map is the guidance map at the current level; (5.3) The guidance map map at the current level is multiplied with the query matrix Q in Hadamard manner to obtain the query matrix Q' at the next level: Q'=map⊙Q (5.4) Scaling up the advanced support features And Q' is scaled down to the next size, repeat steps (5.1) ~ (5.3), you can get the guide map at different levels.

6. The few-shot remote sensing image semantic segmentation method based on a double-branch reinforced network according to claim 1, characterized in that, The structure design method of the decoder in step (6) is as follows: (6.1) At the current level, the high-level query features, the reinforced prototype features and the guidance map are spliced in the channel dimension, and the dimension is reduced through convolution operation; (6.2) The reduced features are sent to the fusion module to obtain fine features at the current level; (6.3) The fine features are spliced with the features after dimension reduction of the next level, and convolution and residual operation are used for fusion; (6.4) Finally, the fine features at four levels are spliced in the channel dimension, and the final semantic segmentation result is obtained through the empty space pyramid pooling.

Images