A small sample semantic segmentation method based on information interaction enhancement

Abstract

The application belongs to the technical field of computer vision, and particularly relates to a small sample semantic segmentation method based on information interaction enhancement; including: acquiring a support image and a corresponding true value mask, and a query image with segmentation; adopting ResNet to process the support image and the query image to obtain support medium and high-level features and query medium and high-level features; adopting a query prior generation module, a guide attention module and a spatial information interaction attention module to process the true value mask of the support image, support medium-level features, support high-level features, query medium-level features and query high-level features to obtain four kinds of output features; inputting the four kinds of output features of the three modules into a multi-scale fusion network for processing to obtain refined query features; inputting the refined query features into a decoder for processing to obtain a predicted segmentation result of the query image; the application can make the model more robust and better segment target objects.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a few-sample semantic segmentation method based on information interaction enhancement. Background Technology

[0002] Thanks to the rapid development of a series of supervised convolutional neural network architectures, deep learning has made significant progress in semantic segmentation. However, most models employ fully supervised learning, meaning their performance depends not only on the quality and quantity of labeled data but also on their poor generalization ability when segmenting unknown categories. This is particularly problematic for intensive prediction tasks, where data collection and labeling consume substantial time and resources. Against this backdrop, few-shot learning techniques have been introduced into semantic segmentation by experts, leading to the development of few-shot semantic segmentation methods that effectively alleviate these issues.

[0003] Few-shot semantic segmentation uses several support images containing the same category and their masks to segment target objects in a query image. Therefore, effectively extracting and utilizing the information provided by the support images to guide the segmentation of the query image is a key factor affecting the performance of few-shot semantic segmentation. Currently, the most popular approach is prototype-based architecture, which extracts representative prototypes by performing masked average pooling on the target region in the support images. These prototypes are then used to perform feature matching with the query image to guide the segmentation of the target object. This method has proven effective, as seen in PFENet, PANet, and ASGNet. However, this method also has drawbacks. Because the generated support prototypes are singular and lack spatial information, directly using the prototypes can lead to semantic ambiguity when there are significant differences in the appearance or shape of the target object between the support and query images, resulting in inaccurate segmentation. Although some methods generate prototypes for multiple categories, such as PMMs and PAMs, they are not optimal in pixel-level feature matching between the support prototypes and query features, and they do not consider fully utilizing the semantic information contained in the support and query images to compensate for intra-class differences and increase the model's generalization ability.

[0004] Therefore, it is necessary to provide a few-sample semantic segmentation method that makes full use of the semantic information of the supporting image and the query image, enhances the information interaction between the two, and improves the robustness of the model. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a few-sample semantic segmentation method based on enhanced information interaction, which includes:

[0006] S1: Obtain the supporting images and their corresponding truth masks, and the segmented query images;

[0007] S2: ResNet is used as the backbone feature extraction network to process the supporting image and the query image to obtain supporting intermediate features, supporting high-level features, query intermediate features, and query high-level features;

[0008] S3: The query prior generation module processes the truth mask of the supported image, the supported intermediate features, the supported high-level features, and the query high-level features to obtain the query prior mask and the supported prototype feature vector.

[0009] S4: The guided attention module is used to process the ground truth mask and intermediate support features of the supporting image to obtain the attention feature map;

[0010] S5: The spatial information interaction attention module is used to process the ground truth mask of the supporting image, the supporting intermediate features and the query intermediate features to obtain the final query features;

[0011] S6: Input the query prior mask, support prototype feature vector, attention feature map and final query features into a multi-scale fusion network for processing to obtain refined query features;

[0012] S7: Input the refined query features into the decoder for processing to obtain the predicted segmentation result of the query image.

[0013] Preferably, step S3 specifically includes:

[0014] The query prior mask is obtained by performing a Hadamard product between the supporting high-level features and the ground truth mask of the supporting image, and then performing a pixel-level cosine similarity calculation with the query high-level features.

[0015] The support intermediate features and the ground truth mask of the support image are subjected to masked average pooling to obtain the support prototype feature vector.

[0016] Furthermore, the formula for obtaining the supporting prototype feature vector is:

[0017]

[0018] Among them, V S ζ represents the support prototype feature vector, and M represents the adjustment. S Size and The function for size matching, M S This indicates the truth mask corresponding to the supported image. This indicates support for intermediate features, ⊙ represents the Hadamard product, and MAP(·) represents the mask average pooling operation.

[0019] Preferably, step S4 specifically includes:

[0020] The support intermediate features and the ground truth mask of the support image are subjected to masked average pooling to obtain the support prototype feature vector.

[0021] Convolution and feature activation are performed on the supporting prototype feature vectors to obtain the attention vectors;

[0022] The attention feature map is obtained by performing Hadamard product on the intermediate features and the attention vector.

[0023] Preferably, step S5 specifically includes:

[0024] The support features are obtained by performing a Hadamard product between the ground truth mask of the support image and the support intermediate features.

[0025] Linear mapping is performed on the supporting features to obtain the supporting linear features; linear mapping is performed on the intermediate features of the query to obtain the first query linear features and the second query linear features;

[0026] The affinity matrix is ​​obtained by performing matrix multiplication on the supporting linear features and the first query linear features;

[0027] The affinity matrix is ​​multiplied with the second query linear feature, and then residual-connected with the query intermediate feature to obtain the first query feature;

[0028] The first query feature is subjected to average pooling and max pooling operations on the channel dimension, and the two pooling features are concatenated to obtain the concatenated feature.

[0029] Convolutional and feature activation operations are performed on the concatenated features to obtain a spatial attention map;

[0030] The second query feature is obtained by performing a Hadamard product between the spatial attention map and the intermediate query features.

[0031] The first query feature and the second query feature are combined to obtain the final query feature;

[0032] Furthermore, the formula for fusing the first query feature and the second query feature is as follows:

[0033]

[0034] in, Indicates the final query characteristics. Indicates the first query feature. This indicates the second query feature. Indicates the concatenation operation, f 1×1 (·) indicates a convolution operation with a kernel size of 1×1.

[0035] Preferably, step S6 specifically includes:

[0036] The final query features and attention feature maps are downsampled to obtain four new query features and four new attention feature maps at different scales.

[0037] The support prototype feature vectors are expanded to have the same shape as the four new query features at different scales, resulting in four new support prototype feature vectors at different scales.

[0038] The query prior mask is adjusted to have the same spatial size as the four new attention feature maps at different scales, resulting in four new query prior masks at different scales.

[0039] The new query features, new attention feature maps, new support prototype feature vectors, and new query prior masks are concatenated along the channel dimension. The concatenated results are then convolved to obtain four coarse query features at different scales.

[0040] Feature interaction processing was performed on four coarse query features at four different scales to obtain four initial refined query features;

[0041] Four initial refined query features are concatenated, and the concatenated results are then fused and dimensionality reduced through convolution to obtain the final refined query features.

[0042] Preferably, the decoder includes a 3×3 convolution with residual connections and a classifier.

[0043] The beneficial effects of this invention are as follows: The few-sample semantic segmentation method based on information interaction enhancement provided by this invention can segment target objects with a small amount of labeled data, reducing the time and cost of data collection and labeling; through an efficient spatial information interaction attention network, the semantic information interaction between the support image and the query image is enhanced, further enriching the features of the query image while making the target class pixels to be segmented more prominent, thereby reducing the negative impact of intra-class interference; by guiding the support features through the mask of the support image, the network is more inclined to learn class-related information, which can effectively alleviate the problem of new class information being suppressed by base class information during the segmentation process and improve the generalization of the model; through the interaction of feature information at different scales, the query image features obtained are richer and more refined, effectively improving the accuracy of the model. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall network structure used for few-sample semantic segmentation in this invention;

[0045] Figure 2 This is a schematic diagram of the network structure of the attention-guiding module in this invention;

[0046] Figure 3 This is a schematic diagram of the network structure of the spatial information interaction attention module in this invention;

[0047] Figure 4 This is a schematic diagram of the multi-scale feature fusion network structure in this invention;

[0048] Figure 5 This is a schematic diagram of the network module structure for feature interaction at two scales in this invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] This invention proposes a few-sample semantic segmentation method based on enhanced information interaction, the method comprising the following:

[0051] S1: Obtain supporting image I S And the corresponding truth mask M S Query image with segmentation I q .

[0052] S2: ResNet is used as the backbone feature extraction network to process the supporting image and the query image to obtain supporting intermediate features, supporting high-level features, query intermediate features, and query high-level features.

[0053] Preferably, a ResNet-50 pre-trained model is used as the backbone feature extraction network, and the overall network model structure diagram is as follows. Figure 1 As shown, the model concatenates the features output from block_2 and block_3 along the channel dimension, then performs feature fusion and dimensionality reduction through 1×1 convolution to obtain intermediate features for the support and query images; the features output from block_4 are used as high-level features for the support and query images. This process is represented as follows:

[0054]

[0055]

[0056]

[0057]

[0058] Among them, I S I q These are respectively supporting images and querying images, B i f represents the feature output by the i-th block; 1×1 (·) indicates a convolution operation with a kernel size of 1×1; These respectively indicate support for intermediate features, support for advanced features, query for intermediate features, and query for advanced features; This indicates a splicing operation.

[0059] S3: The query prior generation module processes the truth mask of the supported image, the supported intermediate features, the supported high-level features, and the query high-level features to obtain the query prior mask and the supported prototype feature vector.

[0060] In order to better activate the target class pixels in the query image and obtain richer query image features, this invention constructs a query prior generation module.

[0061] The process by which the prior generation module processes ground truth masks for supported images, supports intermediate features, supports high-level features, and queries high-level features includes:

[0062] The query prior mask is obtained by performing a Hadamard product between the supporting high-level features and the ground truth mask of the supporting image, followed by pixel-level cosine similarity calculation with the query high-level features. Because the supported image features are multiplied by the prior mask using the Hadamard product, the supported image features only retain the features related to the target class pixels. Therefore, the higher the similarity value of the query feature when calculating its similarity, the greater the probability that the pixel is the target pixel.

[0063] To better activate target class pixels in the query image, support prototype feature vector V is extracted using masked average pooling. S Specifically: Masked average pooling is performed using the ground truth mask of the supporting intermediate features and the supporting image.

[0064]

[0065] Among them, V S ζ represents the support prototype feature vector, and M represents the adjustment. S Size and The function for size matching, M S This indicates the truth mask corresponding to the supported image. This indicates support for intermediate features, ⊙ represents the Hadamard product, MAP(·) represents mask average pooling operation, and C represents the number of feature channels.

[0066] S4: The guided attention module is used to process the ground truth mask and intermediate support features of the supporting image to obtain the attention feature map.

[0067] This invention constructs a guided attention module, aiming to guide the model to better learn and understand category-related information by utilizing supporting images and corresponding masks, thus solving the problem that new class information is suppressed by base class information during segmentation due to the scarcity of new class data. Figure 2As shown, the process by which the attention module processes the ground truth mask and supporting intermediate features of the image includes:

[0068] The guided attention module has two inputs. M S These are the supporting intermediate features and the corresponding binary ground truth masks of their supporting images, respectively. First, masked average pooling is performed on the supporting intermediate features and the ground truth masks of the supporting images to extract the supporting prototype feature vectors that are only relevant to the target category.

[0069]

[0070] A series of convolutional operations and feature activations are performed on the supporting prototype feature vector to obtain the attention vector V. A :

[0071]

[0072] Where σ represents the sigmoid activation function; Conv(·) represents a series of convolution operations.

[0073] By applying attention vectors to support features to suppress irrelevant category information features, an attention feature map G with representative relevant category information is obtained. S Specifically: it will support Hadamard product of intermediate features and attention vectors.

[0074]

[0075] Where H represents the height of the feature map and W represents the width of the feature map.

[0076] S5: The spatial information interaction attention module is used to process the ground truth mask of the supporting image, the supporting intermediate features, and the query intermediate features to obtain the final query features.

[0077] This invention constructs a spatial information interaction attention module, aiming to highlight the features of the target region in the query features. While fully utilizing the spatial and detailed information provided by the supporting image, it performs pixel-level information interaction between the supporting and query images, thereby obtaining more pixel similarity discrimination information that is applied to the query image features. For example... Figure 3 As shown, the spatial information interaction attention module processes the ground truth mask, supported intermediate features, and queried intermediate features of the image, including:

[0078] The spatial information interaction attention module has three inputs, F S M S F q ,in This module is divided into two parts.

[0079] Part 1: M S Applying this to support image features yields F1 support features that retain only information about the target object. S This is beneficial for more accurately focusing on the features of the target object in subsequent steps; specifically: performing a Hadamard product between the ground truth mask of the supporting image and the supporting intermediate features.

[0080]

[0081] Perform a linear mapping on the supporting features to obtain the supporting linear features. Perform a linear mapping on the intermediate features of the query to obtain the first linear feature φ(F). q ) and the second query linear feature g(F) q ); φ(F q ),

[0082] The support linear feature and the first query linear feature are multiplied by matrix to obtain the affinity matrix of the support feature and the query feature.

[0083]

[0084] in, This represents matrix multiplication.

[0085] Due to F1 S Since only the foreground features of the target category are retained, pixels belonging to the target category features will have a greater weight in A during the calculation of pixel similarity between supporting features and query features.

[0086] Applying A to the query feature and adding a residual join yields the query feature. F1 q It contains richer feature information, and the features of the target category pixels are more prominent; specifically: the affinity matrix is ​​multiplied with the second query linear feature, and a residual connection is made with the intermediate query feature to obtain the first query feature F1. q :

[0087]

[0088] Part Two: To better capture the spatial information of query features, the last branch performs a test on the first query feature F1. q We perform average pooling and max pooling along the channel dimension to obtain the pooling features:

[0089]

[0090]

[0091] Where AvgPool(·) and MaxPool(·) represent average pooling and max pooling operations, respectively.

[0092] The result The concatenation is performed along the channel dimension, and the concatenated features are fused using a 7×7 convolution. Finally, feature activation is performed to obtain a spatial attention map.

[0093]

[0094] Among them, f 7×7 (·) indicates a convolution operation with a kernel size of 7×7.

[0095] Applying SA to the query features yields a new query feature (the second query feature). Specifically: Perform a Hadamard product between the spatial attention map and the intermediate features of the query.

[0096]

[0097] By combining the second query feature and the first query feature obtained from the first part, the final query feature is obtained.

[0098] The characteristics of the genus and the target class are more prominent, and they also have richer spatial feature information.

[0099]

[0100] S6: Input the query prior mask, support prototype feature vector, attention feature map and final query features into a multi-scale fusion network for processing to obtain refined query features.

[0101] This invention constructs a multi-scale fusion network, aiming to fuse... V S G S , This is used to better activate target class pixels in the query image and enrich query features to achieve more accurate segmentation of target objects. In other words, it solves the problem of inaccurate segmentation caused by excessive scale differences between target objects in the support image and the query image by effectively fusing multi-scale contextual information of the target object.

[0102] The overall structure of the converged network is as follows Figure 4 As shown, it has 4 inputs These are: the final query feature, supporting prototype feature vector, attention feature map representing category information, and query prior mask. The network output is a refined query feature with rich information. The process by which the multi-scale fusion network processes the query prior mask, supporting prototype feature vector, attention feature map, and final query feature includes:

[0103] For the final query features and attention feature map G S Downsampling was performed to obtain four new query features F at different scales. i =[F 1 ,F 2 ,F 3 ,F 4 ] and four new attention feature maps G at different scales i =[G 1 G 2 G 3 G 4 The four sizes are [60×60, 30×30, 15×15, 8×8].

[0104] Will support prototype feature vector V S Expanding to have the same shape as the four new query features at different scales, we obtain four new support prototype feature vectors V at different scales. i =[V 1 V 2 V 3 V 4 ].

[0105] Query prior mask Adjusting the size to match the spatial size of the four new attention feature maps at different scales yields four new query prior masks M at different scales. i =[M 1 M 2 M 3 M 4 ].

[0106] Concatenate new query features F along the channel dimension i New attention feature map G i New support prototype feature vector V i And the new query prior mask M i The concatenated results are then subjected to simple feature fusion and channel reduction via 1×1 convolution to obtain coarse query features F at four different scales. q i =[F q 1 ,F q 2 ,F q3 ,F q 4 ],

[0107] Interactive processing of four coarse query features at different scales yields four initial refined query features. Specifically, during the fusion of multi-scale context, there are four branches in the horizontal direction. A top-down fusion approach is adopted, passing detailed information from the refined features to the coarse features with lower resolution. (See figure.) Its purpose is to enable feature interaction between two different scales, using auxiliary features F as input from other upper-level branches. aux Enrich the features of the current branch F i Obtain refined query features F R :

[0108] F R =R(F aux ,F i )

[0109] Structural diagram as follows Figure 5 As shown, firstly, the auxiliary features containing more details are adjusted to match the shape of the current feature. Then, they are concatenated along the channel dimension, and a 1×1 convolution is used to extract the auxiliary feature F. aux Extract useful information and then compare it with the current feature F. i Residual connections are performed. Secondly, to reduce the loss of channel information in the query image, the residually connected features are processed through a channel attention mechanism to better extract channel feature information. Finally, after two 3×3 convolutions and the addition of residual connections, the final refined query feature F is obtained. R .

[0110] Features at these four scales from top to bottom F q i =[F q 1 ,F q 2 ,F q 3 ,F q 4 Both pairs of features interacted a total of 7 times. The first branch from top to bottom did not have any auxiliary features interacting with it, while the rest all followed the same rules. The regulations stipulate that four initial refined query features F, containing multi-scale features and contextual information, are obtained. R i =[F R 1 ,F R 2 ,F R 3,F R 4 ].

[0111] F R 1 F R 2 F R 3 F R 4 The spatial size is adjusted to H×W, and four initial refined query features are concatenated. The concatenated results are then fused and dimensionality reduced using a 1×1 convolution to obtain the final refined query features.

[0112] S7: Input the refined query features into the decoder for processing to obtain the predicted segmentation result of the query image.

[0113] The obtained refined query features F final The final predicted segmentation result is obtained through a decoder, which contains a 3×3 convolution with residual connections and a classifier.

[0114] Through the above operations, this invention achieves semantic segmentation of the query image, which makes full use of the semantic information of the supporting image and the query image, enhances the information interaction between the two, and enhances the robustness of the model.

[0115] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0116] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A few-sample semantic segmentation method based on enhanced information interaction, characterized in that, include: S1: Obtain the supporting images and their corresponding truth masks, and the segmented query images; S2: ResNet is used as the backbone feature extraction network to process the supporting image and the query image to obtain supporting intermediate features, supporting high-level features, query intermediate features, and query high-level features; S3: The query prior generation module processes the truth mask of the supported image, the supported intermediate features, the supported high-level features, and the query high-level features to obtain the query prior mask and the supported prototype feature vector. S4: The guided attention module is used to process the ground truth mask and intermediate support features of the supporting image to obtain the attention feature map; The specific process includes: performing masked average pooling on the supporting intermediate features and the ground truth mask of the supporting image to obtain the supporting prototype feature vector; Convolution and feature activation are performed on the supporting prototype feature vectors to obtain the attention vectors; The Hadamard product is performed on the intermediate features and the attention vector to obtain the attention feature map. S5: The spatial information interaction attention module is used to process the ground truth mask, supporting intermediate features, and query intermediate features of the supporting image to obtain the final query features; the specific process includes: performing Hadamard product on the ground truth mask and supporting intermediate features of the supporting image to obtain the supporting features; Linear mapping is performed on the supporting features to obtain the supporting linear features; linear mapping is performed on the intermediate features of the query to obtain the first query linear features and the second query linear features; The affinity matrix is ​​obtained by performing matrix multiplication on the supporting linear features and the first query linear features; The affinity matrix is ​​multiplied with the second query linear feature, and then residual-connected with the query intermediate feature to obtain the first query feature; The first query feature is subjected to average pooling and max pooling operations on the channel dimension, and the two pooling features are concatenated to obtain the concatenated feature. Convolutional and feature activation operations are performed on the concatenated features to obtain a spatial attention map; The second query feature is obtained by performing a Hadamard product between the spatial attention map and the intermediate query features. The first query feature and the second query feature are combined to obtain the final query feature; S6: Input the query prior mask, support prototype feature vector, attention feature map and final query features into a multi-scale fusion network for processing to obtain refined query features; S7: Input the refined query features into the decoder for processing to obtain the predicted segmentation result of the query image.

2. The few-sample semantic segmentation method based on information interaction enhancement according to claim 1, characterized in that, Step S3 specifically includes: The query prior mask is obtained by performing a Hadamard product between the supporting high-level features and the ground truth mask of the supporting image, and then performing a pixel-level cosine similarity calculation with the query high-level features. The support intermediate features and the ground truth mask of the support image are subjected to masked average pooling to obtain the support prototype feature vector.

3. The few-sample semantic segmentation method based on information interaction enhancement according to claim 2, characterized in that, The formula for obtaining the supporting prototype feature vector is: ； in, This indicates that the prototype feature vector is supported. Indicates adjustment Size and Functions for size matching This indicates the truth mask corresponding to the supported image. This indicates support for intermediate features. Represents the Hadama product. This indicates the mask average pooling operation.

4. The few-sample semantic segmentation method based on information interaction enhancement according to claim 1, characterized in that, The formula for combining the first query feature and the second query feature is: ； in, Indicates the final query characteristics. Indicates the first query feature. This indicates the second query feature. This indicates a splicing operation. This indicates a convolution operation with a kernel size of 1×1.

5. The few-sample semantic segmentation method based on information interaction enhancement according to claim 1, characterized in that, Step S6 specifically includes: The final query features and attention feature maps are downsampled to obtain four new query features and four new attention feature maps at different scales. The support prototype feature vectors are expanded to have the same shape as the four new query features at different scales, resulting in four new support prototype feature vectors at different scales. The query prior mask is adjusted to have the same spatial size as the four new attention feature maps at different scales, resulting in four new query prior masks at different scales. The new query features, new attention feature maps, new support prototype feature vectors, and new query prior masks are concatenated along the channel dimension. The concatenated results are then convolved to obtain four coarse query features at different scales. Feature interaction processing was performed on four coarse query features at four different scales to obtain four initial refined query features; Four initial refined query features are concatenated, and the concatenated results are then fused and dimensionality reduced through convolution to obtain the final refined query features.

6. The few-sample semantic segmentation method based on information interaction enhancement according to claim 1, characterized in that, The decoder includes a 3×3 convolution with residual connections and a classifier.

Images