Small sample segmentation method based on multi-scale self-attention feature fusion

Abstract

The application discloses a small sample segmentation method based on multi-scale self-attention feature fusion. Firstly, a pre-trained twin CNN on Imagenet is used as a backbone network to extract features of support and query samples; high-level features and middle-level features are respectively input into a target activation module and a multi-scale self-attention feature fusion module to respectively extract target activation maps and enhanced feature representations of the query samples; the multi-scale self-attention feature fusion module is used to fuse the enhanced query features of each scale, so that coarse-grained information and fine-grained information are effectively fused to better highlight the edge profile and position of a target object; and the fused multi-scale query feature maps are used for final small sample segmentation prediction, and the last segmentation result is obtained through multiple iteration training.

Description

TECHNICAL FIELD

[0001] The present application relates to a sample segmentation technology, in particular to a small sample segmentation method based on multi-scale self-attention feature fusion. BACKGROUND

[0002] Small sample segmentation is an active problem in the field of computer vision and multimedia, which aims to segment the unobserved query image based on a small number of annotated support samples. It is well known that the core problem of small sample segmentation is how to use the annotation information from the support image to guide the query image segmentation.

[0003] However, the main challenge of the small sample segmentation problem is the spatial (e.g. pose, size) inconsistency between the support objects and the query objects. In order to alleviate this challenge, existing methods mainly adopt prototype learning and spatial consistency learning to improve the richness of query feature representation. The prototype learning method usually first adopts a twin convolutional neural network to represent the support and query images, and then uses a mask global average pooling to generate a support prototype. Recent work has also studied some spatial consistency learning methods to alleviate the spatial inconsistency problem between support images and query images. For support image guided query image, they make predictions at different scales, but there is a lack of rich interaction between multiple scales.

[0004] In summary, the existing small sample segmentation has the following problems:

[0005] (1) The support feature guided query feature segmentation process is single;

[0006] (2) The query feature representation between multiple scales does not get sufficient information interaction. SUMMARY

[0007] The purpose of the present application is to solve the problems existing in the prior art, and to provide a small sample segmentation method based on multi-scale self-attention feature fusion. From the perspective of multi-scale information fusion, self-attention is used to aggregate information between different scales, greatly improving the efficiency of information interaction, effectively fusing coarse-grained information and fine-grained information, and better highlighting the edge contour and position of the target object.

[0008] Technical scheme: The small sample segmentation method based on multi-scale self-attention feature fusion of the present application comprises the following steps:

[0009] Step (1), input the query image and the support image with consistent categories, and use a pre-trained convolutional twin neural network as a feature extractor to extract features from the query image and the support image, respectively, to obtain high-level query features, intermediate query features, high-level support features and intermediate support features;

[0010] Step (2), converting the high-level query feature and the high-level support feature containing semantic information into a target activation map Y through cosine similarity calculation q , that is, calculating the cosine similarity of each pixel point in the high-level query feature with all pixel points in the high-level support feature respectively, and then selecting the maximum as the value of the pixel point of the current target activation map; the target activation map Y q shows the probability of each pixel belonging to the target class.

[0011] Step (3), multiplying the intermediate support feature obtained in step (1) with the corresponding true value mask at the corresponding position to obtain a corresponding true value map, filtering the complex background information in the support image through the true value map, and then obtaining a support prototype P of the support image through global pooling operation s .

[0012] Step (4), inputting the intermediate query feature obtained in step (1), the target activation map obtained in step (2), and the support prototype obtained in step (3) into a multi-scale self-attention feature fusion module to obtain a new query feature for final segmentation.

[0013] Step (5), making a loss between the new query feature map obtained in step (4) and the real query segmentation map, and obtaining the final segmentation result through multiple iterations of training.

[0014] Further, the step (1) extracts features of the query image and the support image through a convolutional twin neural network pre-trained on an ImageNet dataset, thereby obtaining consistent feature expressions of the query image and the support image.

[0015] Further, the specific process of obtaining the target activation map in step (2) is as follows:

[0016] Define each pixel point in the high-level support feature and the high-level query feature as: and For each pixel point f i q , i∈[1,hw], the maximum similarity between it and all support pixel points is taken as the corresponding value The formula (1) of the prior target activation map is as follows:

[0017]

[0018]

[0019] Wherein, hw represents the number of feature pixel points.

[0020] Further, the formula for generating a mask global pooling in step (3) is as formula (2):

[0021] Ps =GAP(F s ⊙M s (2)

[0022] Where GAP represents global pooling operation; F s and M s These are represented as intermediate support features and their corresponding ground truth maps. Step (2) first filters out background information from the support image, highlighting the information of the object to be segmented; secondly, this support prototype integrates all the support feature information into a whole, thus better guiding the segmentation of the query image.

[0023] Furthermore, the detailed process of step (4) is as follows:

[0024] First, at the four scales [8, 15, 30, 60], the intermediate query features, supporting prototypes, and target activation maps are merged along the channel dimension. A linear transformation is then used to map the channel size of all four scales to 256, resulting in new merged query features at the four scales [8, 15, 30, 60]. These represent the four different feature scales mentioned above;

[0025] Then, the feature maps at four different scales Expanding all features into feature vectors, such as 8*8*256 (h*w*c) features expanding to 64*256, and 16*16*256 features expanding to 256*256, there are a total of (60*60+30*30+15*15+8*8=4789) feature vectors, which are the query features X for all pixels at all scales. Q .

[0026] Finally, a multi-scale self-attention feature fusion module is used to fully interact the features of pixels at all scales, where the features X of all pixels are combined. Q Q, K, and V are obtained through three linear transformations, and the new query features after fusion are shown in formula (3):

[0027]

[0028] Q, K, and V are all intermediate computational quantities. We use Q and K to calculate weights and aggregate information with weights and V. τ is the balance parameter that controls the Softmax function. Ins.Norm represents the instance normalization operation.

[0029] The multi-scale self-attention feature fusion module comprises a Self-Attention (SA) layer, a residual layer and a normalization layer, the input token of the Self-Attention (SA) layer is set to 4819, and each token is 1*1*256 (w*h*c) in size; then the weight of the aggregated information is obtained through two full connection layers and a softmax, then the multi-scale information is aggregated with V, the original information is merged through residual connection, and finally, instance normalization is performed through Ins.Norm.

[0030] Further, in the iteration process of the step (5), random gradient is adopted for 200 iterations, the batch is set to 8 in each iteration, the model learning rate is 0.0025, the decay rate is 0.0001, until 200 iterations, and the best model is saved for final testing.

[0031] Beneficial effects: the application is used for small sample segmentation tasks, different scale information obtained through feature enhancement is further interacted, the self-attention calculation of the information between multiple scales is performed, so that the transmission and interaction of information are effectively performed.

[0032] (1) The application can capture the correlation of features in an image and explore the interaction between images to learn the consistent representation of the support and query images.

[0033] (2) The application is a new efficient feature enhancement network, which can better guide the information expression of the query feature by the support feature.

[0034] (3) The existing multi-scale fusion module usually adopts a top-down fusion method, which is a hierarchical fusion method, and the adjacent scales are fused layer by layer; while the application converts the feature maps of multiple scales into feature vectors, then completes the fusion operation of multiple scales through a self-attention mechanism, realizes the complementary effect between multiple scale features, highlights useful information, suppresses redundant information, and further enhances the representation ability of the feature map. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 It is a whole segmentation flowchart of the application;

[0036] Figure 2 It is a multi-scale self-attention feature fusion diagram in the embodiment. DETAILED DESCRIPTION

[0037] The technical scheme of the application will be described in detail below, but the protection scope of the application is not limited to the described embodiments.

[0038] The current small sample semantic segmentation is to improve the generalization ability of the model, so that the model can quickly adapt to new scenes. Some papers have proved that the middle-level features (i.e. block-2 and block-3) of the convolutional neural network (such as: resnet50) are more conducive to the generalization ability of the model, and the high-level feature semantic information is stronger, which is not conducive to the generalization of the trained model, but reduces the generalization performance of the model, so the application adopts high-level features to generate target activation maps (not participating in the whole training of the model, only providing some prior information), and the multi-scale self-attention feature fusion module of the application is obtained by downsampling the middle-level features, which is irrelevant to the high-level features, so the generalization ability of the model is retained; in order to express the query feature more richly, the application downsamples the middle-level features to different scales, so that the support information guides the query information at different scales.

[0039] As shown in Figure 1 , a small sample segmentation method based on multi-scale self-attention feature fusion, comprising the following steps:

[0040] Step (1), input the query image and support image with consistent categories, use the pre-trained convolutional twin neural network as the feature extractor to extract the features of the query image and support image respectively, to obtain high-level query features, middle-level query features, high-level support features and middle-level support features;

[0041] Step (2), the high-level query features and high-level support features containing semantic information are converted into target activation map Y q by cosine similarity calculation, that is, the cosine similarity of each pixel point in the high-level query feature and all pixel points in the high-level support feature is calculated, and then the maximum is selected as the value of the pixel point of the current target activation map; the target activation map Y q shows the probability of each pixel belonging to the target class;

[0042] Step (3), multiply the middle-level support features obtained in step (1) with the corresponding true value mask at the corresponding position to obtain the corresponding true value map, filter the complex background information in the support image through the true value map, and then obtain the support prototype P s of the support image through the global pooling operation.

[0043] Step (4), input the middle-level query features obtained in step (1), the target activation map obtained in step (2) and the support prototype obtained in step (3) into the multi-scale self-attention feature fusion module, and then obtain the new query features for final segmentation;

[0044] Step (5), make loss between the new query feature map obtained in step (4) and the real query segmentation map, and obtain the final segmentation result through multiple iterations.

[0045] Embodiment 1

[0046] This embodiment includes the following steps:

[0047] Step (1), using Resnet-50 pre-training model as the basic network to extract the high-level features and the middle-level features of the query image and the support image respectively, specifically:

[0048] Resnet-50 pre-training model is used as the backbone network to extract the features of the support original image and the query image, the features obtained by the backbone network block-4 are used as high-level features, and the features obtained by block-2 and block-3 are stacked together in the channel dimension, and the channel dimension is compressed to 256 through 1*1 convolution.

[0049] Step (2), obtaining the target activation map, which is used to segment the query image;

[0050] The high-level support features and the high-level query features are defined as: and For each f i q The maximum similarity between it and all support pixel points is taken as the corresponding value c q The formula (1) of the prior target activation map is as follows:

[0051]

[0052]

[0053] Finally, Y q is converted into the same spatial size as the original feature Figure 1 , and hw represents the number of feature pixels.

[0054] Step (3), generating a support prototype for global guided query image segmentation;

[0055] The generation of the support prototype is realized by mask global pooling, as shown in formula (2):

[0056] P s = GAP (F s ⊙M s ) (2)

[0057] Wherein, GAP represents the global pooling operation; F s and M s respectively represent the middle-level support features and the true value map of the corresponding size.

[0058] Step (4), the scale self-attention feature fusion model considers the interaction between the query image and the support image while considering the interaction between different scales;

[0059] First, the middle-level query feature, the support prototype, and the target activation map are concatenated together at the four scales of [8, 15, 30, 60], and the channel sizes of the four scales are mapped to 256 through linear transformation to obtain new merged query features at the four scales of [8, 15, 30, 60] representing the four different feature scale sizes as above;

[0060] Then the feature maps of the four different scales are all expanded into feature vectors, such as the 8*8*256 (h*w*c) feature is expanded into 64*256, the 16*16*256 feature is expanded into 256*256, and a total of 4789 feature vectors, i.e., query features X of all scale pixel points Q .

[0061] Finally, all scale pixel point features are fully interacted through the multi-scale self-attention feature fusion module, in which all pixel point features X Q Q, K, and V are obtained through three linear transformations respectively, and the new query feature after fusion is shown in formula (3):

[0062]

[0063] where τ is a balance parameter that controls the Softmax function; Ins. Norm represents the instance normalization operation.

[0064] As shown in Figure 2 , the embodiment includes an encoder, a multi-scale self-attention feature fusion module, and a decoder, and the support image Support feature and the query image Query feature are middle-level features extracted by Resnet50 from the support sample; the masked GAP module filters out the background information of the support feature using the ground truth map of the support image, and then obtains the support prototype (1*1*c) using global average pooling; then the query feature, the support prototype, and the target activation map are all sampled to four spatial sizes, respectively [60*60, 30*30, 15*15, 8*8]; the query feature, the support prototype, and the target activation map are concatenated together in the channel dimension at each scale, and then the channel number is reduced to 256 through 1*1 convolution to obtain new query features at each scale; finally, the feature maps at each scale are straightened into feature vectors, and then all the feature vectors are input into the multi-scale self-attention feature fusion module to complete the information interaction between the scales and generate the final query feature representation for segmentation prediction of the query image.

[0065] Figure 2 The four scales [8, 15, 30, 60] are selected to achieve the enhancement of multi-scale features; the multi-scale self-attention feature fusion module is used to complete the multi-scale fusion operation in one step, the multi-scale self-attention feature fusion module of the application can complete long-distance information interaction, so that not only the information within the scale can be learned, but also multiple scales can learn from each other, which is more convenient and efficient than hierarchical fusion, and richer query feature expression can also be learned, thereby facilitating the final prediction segmentation.

[0066] In the embodiment, the model is iterated 200 times by using a random gradient, the batch is set to 8 in each iteration, the learning rate of the model is 0.0025, the decay rate is 0.0001, and until 200 iterations, the best model is saved for the final test.

Claims

1. A small sample segmentation method based on multi-scale self-attention feature fusion, characterized in that: The method comprises the following steps: Step (1), input the query image and the support image of the same category, and use a pre-trained convolutional twin neural network as a feature extractor to extract features of the query image and the support image to obtain high-level query features, middle-level query features, high-level support features and middle-level support features; Step (2), converting the high-level query feature and the high-level support feature containing semantic information into a target activation map through cosine similarity calculation , that is, calculating the cosine similarity of each pixel point in the high-level query feature with all pixel points in the high-level support feature, and then selecting the maximum as the value of the pixel point of the current target activation map; the target activation map shows the probability of each pixel belonging to the target class. Step (3), multiplying the middle-level support feature obtained in step (1) with the corresponding position of the true value mask corresponding thereto to obtain a corresponding true value graph, filtering the complex background information in the support image through the true value graph, and then obtaining a support prototype of the support image through a global pooling operation ; Step (4), input the middle-level query features obtained in step (1), the target activation map obtained in step (2) and the support prototype obtained in step (3) into a multi-scale self-attention feature fusion module to obtain new query features for final segmentation, and the detailed process is as follows: First, the mid-level query feature, support prototype and target activation map are concatenated together in the channel dimension on the four scales of [8, 15, 30, 60], and the channel sizes of the four scales are all mapped to 256 through linear transformation, to obtain the new merged query features on the four scales of [8, 15, 30, 60] k = {1, 2, 3, 4} represents the four different feature scale sizes as above. Then, four feature maps of different scales are obtained All the feature vectors are expanded, and there are (60*60+30*30+15*15+8*8) feature vectors in total, i.e., the query features of all scale pixel points ; Finally, all scale pixel features are fully interacted by the multi-scale self-attention feature fusion module, in which all pixel features are fused as formula (3) shows: Q, K and V are obtained by three linear transformations respectively, and the new query feature after fusion is shown in formula (3): (3) Q, K and V are intermediate calculation quantities, Q and K are used to calculate weights, and the weights and V are used to aggregate information, tau is a balance parameter for controlling the Softmax function, and Ins.Norm represents an instance normalization operation; The multi-scale self-attention feature fusion module comprises a Self-Attention layer, a residual layer and a normalization layer, the input token of the Self-Attention layer is set to 4819, and the size of each token is 1x1x256 (w x h x c); then the token is input into two fully connected layers and a softmax to obtain the weights of the aggregated information, then the weights are used to aggregate the multi-scale information with V, the original information is merged through residual connection, and finally the Ins.Norm is used for instance normalization operation. Step (5), make a loss between the new query feature map obtained in step (4) and the real query segmentation map, and obtain the final segmentation result through multiple iterations.

2. The small sample segmentation method based on multi-scale self-attention feature fusion according to claim 1, characterized in that: In step (1), the query image and the support image are extracted by a convolutional twin neural network pre-trained on an ImageNet dataset, so as to obtain consistent feature expressions of the query image and the support image.

3. The small sample segmentation method based on multi-scale self-attention feature fusion according to claim 1, characterized in that: The specific process of obtaining the target activation map in step (2) is as follows: Each pixel point in the definition of the advanced support feature and the advanced query feature is respectively: and ; for each pixel point , , the maximum similarity between it and all support pixel points is taken as the corresponding value , and the formula (1) of the prior target activation map is obtained as follows: (1) ； wherein, represents the number of high-level query feature pixels.

4. The small sample segmentation method based on multi-scale self-attention feature fusion according to claim 1, characterized in that: In step (3), the mask global pooling generation formula is as formula (2): (2) where GAP represents a global pooling operation; and are denoted as the mid-level support features and their corresponding ground truth maps of size, respectively.

5. The small sample segmentation method based on multi-scale self-attention feature fusion according to claim 1, characterized in that: In step (5), random gradient is used for 200 iterations in the iteration process, the batch is set to 8 in each iteration, the model learning rate is 0.0025, the decay rate is 0.0001, until 200 iterations, and the best model is saved for final testing.

Images