Few-Shot Remote Sensing Image Segmentation Method Based on Progressive Parsing and Commonsense Distillation

Abstract

The present invention provides a few-shot remote sensing image segmentation method based on progressive parsing and shared distillation. A corresponding semantic segmentation network is designed, which mainly includes a feature extraction network, a progressive parsing module, and a shared distillation module. Among them, the progressive parsing module gradually parses the support mask into multiple valuable sub-regions to provide more information about the difficult parts in the segmentation objects, which can effectively solve the problem of large intra-class diversity in remote sensing images; the shared distillation module extracts rich prototype representations using the sharedness between the base classes and the new classes, and guides prototype fusion through similarity, which can overcome the deficiency of low object-background contrast in images. The present invention can effectively improve the problems of incomplete segmentation target main body and misactivation of irrelevant interference objects that occur when segmenting remote sensing images, improve the recall rate of segmentation objects, and suppress irrelevant interference factors.

Description

Technical Field

[0001] The present invention belongs to the technical field of image semantic segmentation, and particularly relates to a few-shot remote sensing image segmentation method based on progressive parsing and commonality distillation. Background Art

[0002] The purpose of semantic segmentation is to identify the semantic category of each pixel in a given image. Fully convolutional neural networks (FCNs) are pioneering research in the era of deep learning. Based on FCNs, research related to semantic segmentation can be further divided into multiple sub-directions, and the most extensive research mainly focuses on the aggregation of multi-scale information. For example, "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs" designed an atrous spatial pyramid pooling module to jointly capture and integrate multi-scale features using atrous convolution; "Dual attention network for scene segmentation" proposed a dual attention mechanism network to model the semantic correlation in the spatial dimension and channel dimension respectively, and adaptively integrate local features and global features to achieve feature aggregation. Most semantic segmentation tasks require tens of thousands of labeled data for training, and the trained models can only produce outputs for known classes and are not easily extended to segmentation prediction tasks for unknown classes. These unfavorable factors hinder the further development of semantic segmentation. Moreover, most research only involves the analysis and processing of natural images, and there is less research on remote sensing images. Due to the large intra-class diversity and low target-background contrast of remote sensing images, existing few-shot segmentation methods cannot give good results for this complex feature of remote sensing images, and there will be difficult problems such as incomplete segmentation of the target main body and incorrect activation of irrelevant interference objects when segmenting remote sensing images. Summary of the Invention

[0003] To overcome the deficiencies of the prior art, the present invention provides a few-shot remote sensing image segmentation method based on progressive parsing and commonality distillation. A corresponding semantic segmentation network is designed, mainly including a feature extraction network, a progressive parsing module, and a commonality distillation module. Among them, the progressive parsing module gradually parses the support mask into multiple valuable sub-regions to provide more information about the difficult parts in the segmentation object, which can effectively solve the problem of large intra-class diversity in remote sensing images; the commonality distillation module extracts rich prototype representations using the commonality between the base classes and the new classes, and guides the prototype fusion through similarity, which can overcome the deficiency of low object-background contrast in the images. The present invention can effectively improve the problems of incomplete segmentation target main body and incorrect activation of irrelevant interference objects when segmenting remote sensing images, improve the recall rate of the segmentation object, and suppress irrelevant interference factors.

[0004] A few-shot semantic segmentation method based on progressive parsing and commonality distillation, characterized by the following steps:

[0005] Step 1: Randomly divide the publicly available iSAID dataset into three subsets equally. Among them, any two subsets are used as the training dataset, and the other subset is used as the test dataset. The dataset contains support images and their support masks, query images and their query masks;

[0006] Step 2: Given a support image and a query image Use a feature extraction network with shared weights to extract their intermediate feature maps F s and F q , where H represents the height of the image and W represents the width of the image; the feature extraction network uses a VGG-16 network or a ResNet-50 network;

[0007] Step 3: Input the intermediate feature map F s of the support image and its original support mask Y s into the progressive parsing module, and output a number of parsing prototypes p p ;

[0008] The processing process of the progressive parsing module is as follows: First, perform global average pooling on the intermediate feature map to obtain the initial prototype p0 of its foreground region. Then, iteratively calculate the prototype p p as follows:

[0009]

[0010] where p pDenote the p-th prototype, where p = 0, ..., np, and np is the maximum number of parsing times, with a value range of 1 to 10; (x, y) represents the spatial position of a point in the graph, where x = 1, 2, …, W and y = 1, 2, …, H. Denote the intermediate feature map F s The value of the point at position (x, y) in it. 1[·] is an indicator function that outputs 1 when the condition is met and 0 otherwise. Denote the support mask Y used in the p-th parsing s；p The value of the point at position (x, y) in it. Initially, Y s；0 = Y s When performing the p-th parsing, use the support prototype p obtained from the (p - 1)-th parsing p-1 and the intermediate feature map F s to perform a similarity measurement to obtain a probability map. When the value of the point at position (x, y) in the probability map is greater than the threshold τ, set the value of the point at position (x, y) in the support mask Y s；p-1 used in the (p - 1)-th parsing to 0 to obtain a new support mask Y s；p The value range of the threshold τ is 0.3 to 0.9.

[0011] Step 4: Calculate the probability map M according to the following formula s :

[0012]

[0013] where (x, y) represents the spatial position of a point in the graph, p′ ∈ {0, ..., np - 1} is the prototype index, denote the value of the point at position (x, y) in the probability map M s p p′ denote the p′-th prototype calculated in Step 3, denote the value of the point at position (x, y) in the intermediate feature map F s ||·|| represents the L2 norm of the vector;

[0014] Step 5: Calculate the prediction mask according to the following formula

[0015]

[0016] where p″ = p - 1 represents the index of the probability map and the prediction mask, with a value range of 1 to np - 1; denote the p″-th prediction mask the value of the point at position (x, y) in it;

[0017] Step 6: Calculate the new support mask Y s ′ according to the following formula:

[0018]

[0019] Among them, represents the support mask Y′ s；p″+1 the value of the point at position (x, y) in

[0020] Step 7: Calculate the final support prototype p according to the following formula s :

[0021]

[0022] where Conv(·) represents a 1×1 convolution operation, represents the concatenation operation along the channel dimension, and p p represents the p-th prototype calculated in Step 3, p = 0,..., np;

[0023] Step 8: Input the support prototype p s into the commonality distillation module, and output a number of new class prototypes where l represents the new class prototype index, l ∈ {1,..., ns}, and ns represents the number of new class prototypes; the commonality distillation module includes a number of base class memory units subject to normal distribution, and uses the number of base classes most relevant to the prototype p s as the output of the new class prototypes;

[0024] Step 9: Perform weighted fusion on the support prototype p s and all new class prototypes to obtain the fused prototype

[0025]

[0026] where β is the first fusion coefficient, set to β = 0.7, α is the second fusion coefficient, set to α = 1 / ns, l = 1,..., ns;

[0027] Step 10: Input the fused prototype and the intermediate feature map F of the query image q into the decoder network for segmentation prediction, and output the segmentation mask; the decoding network is a network with multi-scale feature aggregation function;

[0028] Step 11: Use the support image, query image, support mask, and query mask in the training dataset as inputs, and train all network modules according to the above Steps 2 - 10 to obtain a trained network, where the network loss L is set as follows during training:

[0029] L = L seg + ηL PPR(7)

[0030] Among them, L seg is the binary cross-entropy loss of the segmentation mask and the query mask output by the network. η is the balance coefficient, set to η = 0.5, and L PPR is the progressive parsing regularization loss, which is calculated according to the following formula:

[0031]

[0032] Among them, BCE(·,·) represents the binary cross-entropy loss, is the concatenation operation along the channel dimension;

[0033] Step 12: Input the support image, query image, and support mask in the test dataset into the trained network, and the output segmentation mask is its semantic segmentation result.

[0034] Furthermore, the fusion prototype described in step 9 is calculated according to the following formula:

[0035]

[0036] Among them, l = 1,..., ns, α (l) represents the second fusion coefficient of the l-th one, and is calculated according to the following formula:

[0037]

[0038] Among them, w( l ) represents the l-th fusion weight of the new class prototype, and is calculated according to the following formula:

[0039]

[0040] Among them, λ0 = 2, λ1 = 1, γ0 = 5, and γ1 = 5 are all constants, represents the l-th new class prototype and the similarity with the support prototype p s ; represents the l-th new class prototype and the similarity with the intermediate feature map F q of the query image.

[0041] The beneficial effects of the present invention are as follows: By adopting the method of gradually parsing the support mask into multiple valuable sub-regions, more information about the difficult-to-process parts in the segmentation object can be provided, effectively solving the problem of large intra-class diversity in remote sensing images; extracting rich prototype representations using the commonality between the base class and the new class, and through the processing process of similarity-guided prototype fusion, the deficiency of low target-background contrast in the image can be overcome; the present invention can effectively improve the problems of incomplete segmentation target main body and incorrect activation of irrelevant interference objects that occur when segmenting remote sensing images, improve the recall rate of the segmentation object, suppress irrelevant interference factors, provide a more effective solution for improving the unsatisfactory segmentation of remote sensing images by existing few-shot segmentation models, and show good application prospects and potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of the processing process of the few-shot remote sensing image semantic segmentation network based on progressive parsing and commonality distillation of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0043] The present invention will be further described below in conjunction with the drawings and embodiments, and the present invention includes but is not limited to the following embodiments.

[0044] The present invention provides a few-shot semantic segmentation method based on progressive parsing and commonality distillation, and its specific implementation process is as follows:

[0045] 1. Dataset preparation

[0046] The present invention constructs a benchmark dataset based on the publicly available iSAID dataset. This dataset includes 655,451 fine-annotated object instances of 15 geospatial categories in 2,806 high-resolution images. The iSAID dataset is randomly divided into three equal subsets. Among them, any two subsets are used as the training dataset, and the other subset is used as the test dataset. The dataset contains support images and their support masks, query images and their query masks.

[0047] 2. Network construction

[0048] The present invention constructs a few-shot remote sensing image semantic segmentation network based on progressive parsing and commonality distillation, which mainly includes three parts: a backbone network (feature extraction network), a progressive parsing module, and a commonality distillation module. As Figure 1 shown, its specific processing process is as follows:

[0049] (1) Given a support image and a query image The backbone network with shared weights, that is, the feature extraction network, is used to separately extract the intermediate feature maps F s and F q, where H represents the height of the image and W represents the width of the image; the feature extraction network adopts the VGG-16 network or the ResNet-50 network.

[0050] (2) Support the intermediate feature map F of the image s and its original support mask Y s Input to the progressive parsing module, and output several parsing prototypes p p .

[0051] The processing process of the progressive parsing module is as follows: first, before the parsing starts (t0), the intermediate feature map is globally averaged pooled to obtain the initial prototype p0 of its foreground area. During the parsing process, the cosine similarity between the prototype and the supporting features is evaluated. Based on the preset threshold τ and the previously parsed mask, the under-segmented area in the probability map is further mined, and the newly obtained mask with the corresponding features is used for the next parsing until the maximum parsing number np (t np ) until finally we get p np . Iterate and calculate the prototype p as above p The process can be specifically expressed by the formula as follows:

[0052]

[0053] Among them, p p represents the pth prototype, p=0,...,np, np is the maximum number of parsing times, ranging from 1 to 10; (x,y) represents the spatial position of the point in the figure, x=1,2,...,W, y=1,2,...,H, Represents the intermediate feature map F s The value of the point at position (x, y) in the graph, 1[·] is an indicator function, which outputs 1 if the condition is met and 0 if it is not met; Indicates the support mask Y used in the pth parsing s；p The value of the point at position (x, y) in the middle, where Y is initially s；0 =Y s , when parsing for the pth time, the supporting prototype p obtained by parsing for p-1 times is p-1 and the intermediate feature map F s The similarity measurement is performed to obtain a probability map. When the value of the point at position (x, y) in the probability map is greater than the threshold τ, the support mask Y used in the p-1th analysis is s；p-1 The value of the point at position (x, y) is set to 0, and the new support mask Y is obtained. s；p , the threshold τ ranges from 0.3 to 0.9.

[0054] (3) The probability map M is calculated according to the following formula s :

[0055]

[0056] Among them, (x, y) represents the spatial position of a point in the figure, and p′∈{0,...,np - 1} is the prototype index. Represents the probability map M s The value of the point at position (x, y) in, p p′ Represents the p′-th prototype calculated in step 3. Represents the intermediate feature map F s The value of the point at position (x, y) in, ||·|| represents the L2 norm of the vector.

[0057] (4) Calculate the predicted mask according to the following formula

[0058]

[0059] Among them, p″ = p - 1 represents the index of the probability map and the predicted mask, and the value range is 1~np - 1; Represents the p″-th predicted mask The value of the point at position (x, y) in.

[0060] (5) Calculate the new support mask Y′ according to the following formula s :

[0061]

[0062] Among them, Represents the support mask Y′ s；p″+1 The value of the point at position (x, y) in.

[0063] (6) After the parsing process is fully executed, considering that all the prototypes generated by parsing are local descriptions of difficult-to-process components and represent partial features of the support image, the present invention further integrates all the parsed prototypes {p1,...,p np} and the overall initial prototype p0 through concatenation and convolution operations to generate a more discriminative class descriptor, that is, calculate the final support prototype p according to the following formula s :

[0064]

[0065] Among them, Conv(·) represents a 1×1 convolution operation. Represents the concatenation operation along the channel dimension, p p Represents the p-th prototype calculated in step 3, p = 0,...,np.

[0066] (7) The commonality distillation module stores a certain number of base class memory units Assume that the memory units of each base class follow a normal distribution where μ i and σ i represent the mean and standard deviation respectively. Considering the incompleteness of the new class representation and the intra-class diversity of remote sensing images in the few-shot paradigm, the present invention uses the base classes most relevant to the support prototypes for prototype sampling

[0067] Input the support prototype p s into the commonality distillation module, and use several base classes most relevant to the prototype p s as the output of the new class prototype, denoted as the new class prototype where l represents the new class prototype index, l ∈ {1,..., ns}, and ns represents the number of new class prototypes. This process can be expressed by the formula:

[0068]

[0069] where * ∈ {1,..., nb} represents the index of the selected base class, and the standard deviation of this class is used to construct the normal distribution of the new class μ n represents the mean of the new class. In this way, the base class knowledge obtained from a large number of labeled samples is transferred to the new class, providing multiple robust representations for the challenging segmentation tasks in the few-shot setting

[0070] (8) Subsequently, for the support prototype p s output by the progressive parsing module and the new class prototype refined by the commonality distillation module perform weighted fusion based on the following formula to obtain the fused prototype

[0071]

[0072] where β is the first fusion coefficient, set to β = 0.7, and α is the second fusion coefficient, which can be set to a fixed value, i.e., α = 1 / ns, l = 1,..., ns, or can be adaptively determined as follows:

[0073]

[0074] where l = 1,..., ns, α (l) represents the l-th second fusion coefficient, and is calculated according to the following formula:

[0075]

[0076] where w( l ) represents the l-th fusion weight of the new class prototype, and is calculated according to the following formula:

[0077]

[0078] Among them, λ0 = 2, λ1 = 1, γ0 = 5, and γ1 = 5 are all constants, represents the l-th new class prototype and the support prototype p s similarity. represents the l-th new class prototype and the intermediate feature map F of the query image q similarity.

[0079] (9) Finally, input the fused prototype and the intermediate feature map F of the query image q into the decoder network for segmentation prediction, and output the segmentation mask; the decoding network is a network with multi-scale feature aggregation function.

[0080] 3. Network training

[0081] Use the support images, query images, support masks, and query masks in the training dataset as inputs, and train all network modules according to the above steps 2 - 10 to obtain a trained network.

[0082] To improve the effectiveness of the mask parsing process and promote the generation of more representative local descriptors, the present invention imposes an additional segmentation constraint on the progressive parsing module, namely the Progressive Parsing Regularization (PPR) loss, which is calculated as follows:

[0083]

[0084] Among them, BCE(·,·) represents the binary cross-entropy loss, is the concatenation operation along the channel dimension;

[0085] The total loss L of the network during training is as follows:

[0086] L = L seg + ηL PPR (23)

[0087] Among them, L seg is the binary cross-entropy loss between the segmentation mask output by the network and the query mask, η is the balance coefficient, and is set to η = 0.5.

[0088] 4. Network inference

[0089] Input the support images, query images, and support masks in the test dataset into the trained network, and the output segmentation mask is the semantic segmentation result.

Claims

1. A few-shot remote sensing image segmentation method based on progressive parsing and co-ownership distillation, characterized in that The steps are as follows: Step 1: Randomly and equally divide the publicly available iSAID dataset into three subsets. Among them, any two subsets are selected as the training datasets, and the other subset is used as the test dataset. The dataset contains support images and their support masks, query images and their query masks; Step 2: Given a support image and a query image Use a feature extraction network with shared weights to separately extract their intermediate feature maps F s and F q , where H represents the height of the image and W represents the width of the image; the feature extraction network uses a VGG-16 network or a ResNet-50 network; Step 3: Input the intermediate feature map F that supports images s and its original support mask Y s into the progressive parsing module, and output several parsing prototypes p p ; The processing procedure of the progressive parsing module is as follows: First, perform global average pooling on the intermediate feature map to obtain the initial prototype p0 of its foreground region, and then, iteratively calculate the prototype p according to the following formula p :[[]]END]] where p p represents the p-th prototype, where p = 0, ..., np, and np is the maximum number of parsing times with a value range of 1 to 10; (x, y) represents the spatial position of a point in the figure, where x = 1, 2, …, W and y = 1, 2, …, H. represents the value of the point at position (x, y) in the intermediate feature map F s , and 1[·] is an indicator function that outputs 1 when the condition is met and 0 otherwise. represents the value of the point at position (x, y) in the support mask Y s；p used in the p-th parsing. Initially, Y s；0 = Y s . During the p-th parsing, the support prototype p p-1 obtained from the (p - 1)-th parsing and the intermediate feature map F s are used for similarity measurement to obtain a probability map. When the value of the point at position (x, y) in the probability map is greater than the threshold τ, the value of the point at position (x, y) in the support mask Y s；p-1 used in the (p - 1)-th parsing is set to 0 to obtain a new support mask Y s；p , where the value range of the threshold τ is 0.3 to 0.

9. Step 4: Calculate the probability map M according to the following formula s :[[]]END]] Among them, (x, y) represents the spatial position of a point in the figure, p′ ∈ {0,..., np - 1} is the prototype index, represents the value of the point at position (x, y) in the probability graph M s , p p′ represents the p′-th prototype calculated in step 3, represents the value of the point at position (x, y) in the intermediate feature map F s ||·|| represents the L2 norm of a vector; Step 5: Calculate the predicted mask according to the following formula where p″ = p - 1 represents the indices of the probability map and the prediction mask, with values ranging from 1 to np - 1; represents the p″-th prediction mask at the point (x, y); Step 6: Calculate the new support mask Y s ': Among them, represents the support mask Y'; s；p″+1 the value of the point at the position (x, y) in Step 7: Calculate the final support prototype p according to the following formula s :[[]]END]] where Conv(·) represents a 1×1 convolution operation, represents a concatenation operation along the channel dimension, p p represents the p-th prototype calculated in step 3, p = 0, ..., np; Step 8: Input the support prototype p s into the commonality distillation module, and output a number of new class prototypes where l represents the new class prototype index, l ∈ {1,..., ns}, and ns represents the number of new class prototypes; the commonality distillation module includes a number of base class memory units subject to a normal distribution, and uses the several base classes most relevant to the prototype p s as the output of the new class prototypes; Step 9: Perform weighted fusion on the support prototype p s and all new class prototypes to obtain a fused prototype Among them, the first β fusion coefficient is set to β = 0.7, and the second α fusion coefficient is set to α = 1 / ns, where l = 1,..., ns; Step 10: Input the intermediate feature map F of the fusion prototype and the query image into the decoder network for segmentation prediction to output a segmentation mask; the decoder network is a network with multi-scale feature aggregation function; q ​ Step 11: Use the support images, query images, support masks, and query masks in the training dataset as inputs, and train all network modules according to Steps 2 - 10 above to obtain a trained network. Among them, when training, set the network loss L as follows: L = L seg + ηL PPR (7) Among them, L seg is the binary cross-entropy loss of the segmentation mask and the query mask output by the network. η is the balance coefficient, set to η = 0.5, and L PPR is the progressive parsing regularization loss, which is calculated according to the following formula: where BCE(·, ·) represents binary cross-entropy loss, is a concatenation operation along the channel dimension; Step 12: Input the support images, query images, and support masks in the test dataset into the trained network, and the output segmentation mask is its semantic segmentation result.

2. The few-shot remote sensing image segmentation method based on progressive parsing and co-ownership distillation according to claim 1, wherein: The fusion prototype described in Step 9 Calculated according to the following formula: where \(l = 1,\ldots,n_s\), \(\alpha\) (l) represents the \(l\)-th fusion coefficient two and is calculated by the following formula: where, w (l) represents the l-th fusion weight of the new class prototype and is calculated as follows: Among them, λ0 = 2, λ1 = 1, γ0 = 5, and γ1 = 5 are all constants. represents the l-th new class prototype similarity with the support prototype p s and represents the l-th new class prototype similarity with the intermediate feature map F of the query image q and

Images