Small sample semantic segmentation method and system based on inter-class relationship

Abstract

The invention belongs to the technical field of semantic segmentation, and particularly discloses a small sample semantic segmentation method and system based on inter-class relationships, and the method comprises the following steps: obtaining pictures of a query set and a feature set, and extracting features of the query set and features of a support set; setting a feature auto-aggregation (FSA) module, a cross support and query function aggregation (CSQFA) module and a priori mask pattern (PMF) module, constructing a cross support set and query set feature aggregation network, and obtaining aggregation features, dense pixel-by-pixel attention features of the cross support set and the query set; and inputting the dense pixel-by-pixel attention features crossing the support set and the query set into a priori mask pattern PMF module to obtain a final priori mask for final classification. By adopting the technical scheme, the CSQFA and the FASA expand the difference between the classes by expanding the transformer, reduce the difference between the classes and implement pixel-level dense matching, and the PMF is used for improving a prototype mode and mining background information in the prototype mode so that the prior mask is more accurate.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic segmentation, and relates to a small sample semantic segmentation method and system based on inter-class relationship. Background Art

[0002] In recent years, deep convolutional neural networks have made great progress in computer vision tasks such as object detection, semantic segmentation, style transfer, and panoptic segmentation. However, these advances not only rely on deep networks, but also rely heavily on large-scale well-annotated datasets. In some areas where data is in urgent need, such as medical image segmentation, the performance of deep convolutional neural networks has not achieved the desired results, especially when the model is expanded to classes that have not been seen during the model training process.

[0003] In order to solve the above problems, weakly supervised semantic segmentation and many other methods have been tried, but the problem of poor model scalability has not been solved. Humans are born with the ability to quickly identify new categories using a small number of samples through existing life prior knowledge. Inspired by this phenomenon, researchers have tried to develop a paradigm for the learning ability of small sample semantic segmentation (FSS) models, which learn a segmentation pattern on the existing labeled base classes (classes used during training).

[0004] The main challenge of small-sample semantic segmentation models is how to utilize the existing information in the support set. Most of the previous work has been along the prototype direction, that is, the information in the support set is compressed into a prototype vector through global pooling or clustering and the prototype vector is used to match the features of the query set. However, the prototype approach easily leads to the loss of details in the support set. Summary of the invention

[0005] The purpose of the present invention is to provide a small sample semantic segmentation method and system based on inter-class relationship to make the priori mask more accurate.

[0006] In order to achieve the above object, the basic scheme of the present invention is: a small sample semantic segmentation method based on inter-class relationship, comprising the following steps:

[0007] Get the query set and feature set images;

[0008] Input the query set and feature set images into the feature extractor to extract query set features and support set features;

[0009] Set up feature self-aggregation FSA module, cross support and query function aggregation CSQFA module and prior mask graph PMF module to build a feature aggregation network across support set and query set;

[0010] Input the query set features and the support features into the feature self-aggregation FSA module, reshape the query set features and the support features into multiple local features in spatial position, add the local feature vector and the corresponding position vector as the input of the transformer, and obtain the self-aggregation feature;

[0011] The aggregated features are fed into the cross-support and query feature aggregation CSQFA module to obtain dense pixel-wise attention features across the support set and query set;

[0012] The dense pixel-by-pixel attention features across the support set and query set are fed into the prior mask graph PMF module to obtain the final prior mask for final classification.

[0013] The working principle and beneficial effects of this basic solution are as follows: This technical solution proposes CSQFAN (cross-support set and cross-set feature aggregation network) to solve the problem of small sample semantic segmentation, which includes three modules: FSA, CSQFA and PMF. CSQFA and FASA expand the gap between classes and reduce the gap within classes by expanding the transformer and implement pixel-level dense matching. The role of FSA is to increase the similarity of similar targets in the same image, and CSQFA realizes the interaction between support set features and query set features. PMF is used to mine the relationship between classes to replace the traditional prior mask. PMF improves the prototype method and mines the background information to make the prior mask more accurate.

[0014] Furthermore, the query set features and the support features are input into the feature self-aggregation FSA module, and the method for obtaining the self-aggregation features is as follows:

[0015] Reshape the query set features and support features into r local features in spatial position, k = h × w, h, w represent the height and width of the input features respectively, and k represents the kth sampling point;

[0016] Add the local feature vector and the corresponding position vector as the input Z of the transformer in the feature self-aggregation FSA module S , The vector representing the pixel point, E pos is a vector using sinusoidal position encoding;

[0017] The output of the FSA module is based on the standard attention operation in the transformer encoder:

[0018]

[0019] Among them, Attention represents the attention operation, Q, K, and V represent query, key, and value vectors respectively, and d kis the dimension of the key vector, and

[0020]

[0021] in, is the result obtained by adding the local vector and the sine code to obtain z, and then calculating it through the attention formula, that is, the result after the self-attention operation; W q , W k , W v are learnable parameters;

[0022] After a normalization layer and a multi-layer perceptron:

[0023]

[0024] The traditional visualization transformer takes a series of image blocks as a sequence as input based on the standard transformer. In contrast, this scheme adds the local feature vector and the corresponding position vector as the input of the transformer to increase the similarity of similar objects in the same image.

[0025] Furthermore, we obtain dense pixel-by-pixel attention features across the support set and query set as:

[0026]

[0027]

[0028] Convert the three-dimensional input features into two dimensions, and use each pixel as a token to obtain Q, K, and V after a linear mapping.

[0029] in, Q is the dense pixel-by-pixel attention feature obtained by aggregating the query set features S. is the dense pixel-by-pixel attention feature obtained by aggregating S with the query set feature Q; Q, K, V represent query, key, and value vectors respectively; F Q and F S represent query set features and support set features respectively, W q ,W k ,W v ∈R d×d is a learnable parameter, d is the hidden layer dimension;

[0030] CSQFA module through Q q S k To measure the similarity of all support set pixels to a specific query set pixel, and S vMultiplication aggregation from F S Foreground information, where Q q W q F Q ,S k W k F S ,S v W v F S , W q , W k , W v is a learnable parameter, F Q is the query set feature, F S is the support set feature; Q q Represents the Query obtained by changing the query set, S k represents the key obtained by changing the support set, S v Represents the value obtained by changing the support set;

[0031] Interchange the backing set and query set, sharing the same parameters.

[0032] Dense pixel-by-pixel attention features across support sets and query sets are obtained to achieve the interaction between support set features and query set features.

[0033] Furthermore, the dense pixel-by-pixel attention features across the support set and the query set are input into the prior mask graph PMF module to obtain the final prior mask. The specific steps are:

[0034] Mask out high-level support features with corresponding masks The superscript i represents high-level features, s represents support set, c, h, and w represent channel dimensions, height, and width respectively. and high-level queryset features Expand F∈R c×k , k = h × w;

[0035] Based on the sampled pixel and each pixel of the query set feature, the cosine similarity calculation is used to measure the similarity between the foreground pixel in the support set and each pixel in the query set. The subscript SQ represents the calculation of cosine similarity between the query set and the support set, and the superscript FG represents foreground;

[0036] right After standardization, the threshold is set to 0.3 for sampling, and sampling points with confidence levels lower than 0.3 are considered potential background points;

[0037] Calculate the cosine similarity between the background point vector and the query set feature vector in the query set to obtain a series of The superscript BG represents the potential background, k represents the kth sampling point, and the subscript QQ represents the cosine similarity between the query set itself and its own vector;

[0038] To mine the relevance of its own foreground points, the same operation is performed, and the threshold is set to 0.7, and a series of

[0039] Get the final prior mask:

[0040]

[0041] Among them, PM stands for Pri or Mask, which is the final prior mask, ReL is an activation function, and FG i Represents the foreground of the i-th sampling point.

[0042] The dense pixel-by-pixel attention features across the support set and query set are fed into the prior mask graph PMF module to obtain the final prior mask with higher accuracy.

[0043] The present invention also provides a small sample semantic segmentation system based on inter-class relationship, comprising a data acquisition module and a processing module, wherein the data acquisition module acquires a query set and a feature set image, and an output end of the data acquisition module is connected to the processing module;

[0044] The processing module executes the method of the present invention, obtains an optimized priori mask, and performs small sample semantic segmentation.

[0045] The system has a simple structure and obtains optimized prior masks, which is beneficial for small sample semantic segmentation.

[0046] Furthermore, the processing module includes a feature self-aggregation FSA module, a cross support and query function aggregation CSQFA module and a priori mask pattern PMF module which are connected in sequence.

[0047] The processing module is provided with three modules, namely FSA, CSQFA and PMF, which cooperate with each other for easy use.

[0048] Furthermore, the feature self-aggregation FSA module includes a self-attention mechanism and a position encoding unit;

[0049] The input of the feature self-aggregation FSA module is the query set features and the support set features. The features of the support set are masked, and the features of the query set are not processed. The query set features and the support set features are expanded into c*k vectors and added to the position code, and then the self-aggregation features are obtained through the self-attention mechanism, where k = h×w, c, h, w represent the channel dimensions, height, and width respectively, and k represents the kth sampling point.

[0050] Furthermore, the cross support and query function aggregation CSQFA module includes a cross attention mechanism unit.

[0051] Furthermore, the priori mask pattern PMF module includes a calculation unit for sampling and cosine similarity of foreground points and background points, and the output of the cross support and query function aggregation CSQFA module is connected in series with the priori mask pattern PMF module and then connected to the decoder.

[0052] Simple structure and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a structural schematic diagram of the small sample semantic segmentation method based on inter-class relationship of the present invention;

[0054] Figure 2 It is a structural schematic diagram of FSA and CSQFA of the small sample semantic segmentation method based on inter-class relationship of the present invention;

[0055] Figure 3 It is a structural diagram of the PMF of the small sample semantic segmentation method based on inter-class relationship of the present invention. DETAILED DESCRIPTION

[0056] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0057] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0058] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0059] The present invention discloses a small sample semantic segmentation method based on inter-class relationship, such as Figure 1 As shown, the following steps are included:

[0060] Get the query set and feature set images;

[0061] Input the query set and feature set images into the pre-trained feature extractor with fixed parameters to extract the query set features and support set features F S and F Q (Q stands for query set, S stands for support set);

[0062] like Figure 2 As shown in the figure, a feature self-aggregation FSA (Feature Self Aggregation) module, a cross support and query feature aggregation CSQFA (Cross Support and Query Feature Aggregation) module and a priori mask feature PMF (Pr ior mask feature) module are set to build a feature aggregation network across support sets and query sets; the role of FSA is to increase the similarity of similar targets in the same image, CSQFA realizes the interaction between support set features and query set features, and PMF is used to mine inter-class relationships to replace the traditional prior mask.

[0063] Input the query set features and the support features into the feature self-aggregation FSA module, reshape the query set features and the support features into multiple local features in spatial position, add the local feature vector and the corresponding position vector as the input of the transformer, and obtain the self-aggregation feature;

[0064] The aggregated features are fed into the cross-support and query feature aggregation CSQFA module to obtain dense pixel-wise attention features across the support set and query set;

[0065] The dense pixel-by-pixel attention features across the support set and query set are fed into the prior mask graph PMF module to obtain the final prior mask for final classification.

[0066] In a preferred embodiment of the present invention, the query set features and the support features are input into the feature self-aggregation FSA module to obtain the self-aggregation features by:

[0067] Reshape the query set features and support features into r local features in spatial position, k = h × w, h, w represent the height and width of the input features respectively, and k represents the kth sampling point;

[0068] The traditional visualization transformer takes a series of image blocks as a sequence as input based on the standard transformer, while the present invention adds the local feature vector and the corresponding position vector as the input Z of the transformer in the feature self-aggregation FSA module. S , The vector representing the pixel point, E pos is a vector using sinusoidal position encoding;

[0069] The output of the FSA module is based on the standard attention operation in the transformer encoder:

[0070]

[0071] Among them, Attention represents the attention operation, Q, K, and V represent query, key, and value vectors respectively, and d k is the dimension of the key vector, and

[0072]

[0073] in, is the result obtained by adding the local vector and the sine code to obtain z, and then calculating it through the attention formula, that is, the result after the self-attention operation; W q , W k , W v are learnable parameters;

[0074] After a normalization layer (Layer Normalization, LN) and a multi-layer perceptron (Multi-LayerPerceptron, MLP):

[0075]

[0076] In a preferred embodiment of the present invention, dense pixel-by-pixel attention features across the support set and the query set are obtained as follows:

[0077]

[0078] Convert the three-dimensional input features into two dimensions, and use each pixel as a token to obtain Q, K, and V after a linear mapping.

[0079] in, Q is the dense pixel-by-pixel attention feature obtained by aggregating the query set features S. is the dense pixel-by-pixel attention feature obtained by aggregating S with the query set feature Q; Q, K, V represent query, key, and value vectors respectively; F Q and F S represent query set features and support set features respectively, W q ,W k ,W v ∈R d×dis a learnable parameter, d is the hidden layer dimension;

[0080] CSQFA module through Q q S k To measure the similarity of all support set pixels to a specific query set pixel, and S v Multiplication aggregation from F S Foreground information, where Q q W q F Q ,S k W k F S ,S v W v F S , W q , W k , W v is a learnable parameter, F Q is the query set feature, F S is the support set feature; Q q Represents the Query obtained by changing the query set, S k represents the key obtained by changing the support set, S v Represents the value obtained by changing the support set;

[0081] Interchange the backing set and query set, sharing the same parameters.

[0082] In a preferred embodiment of the present invention, Figure 3 As shown in Figure 1, the dense pixel-by-pixel attention features across the support set and the query set are input into the prior mask graph PMF module to obtain the final prior mask. The specific steps are:

[0083] The prior mask has many extended uses in the FFS task because it can roughly locate the target category in the query set only by the support set and the mask corresponding to the support set.

[0084] Mask out high-level support features with corresponding masks The superscript i represents high-level features, s represents support set, c, h, and w represent channel dimensions, height, and width respectively. and high-level queryset features Expand F∈R c×k , k = h × w;

[0085] Based on the sampled pixel and each pixel of the query set feature, the cosine similarity calculation is used to measure the similarity between the foreground pixel in the support set and each pixel in the query set. The subscript SQ represents the calculation of cosine similarity between the query set and the support set, and the superscript FG represents foreground;

[0086] The backgrounds in the same picture are similar. In order to find these potential background points, After standardization, the threshold is set to 0.3 for sampling, and sampling points with confidence levels lower than 0.3 are considered potential background points;

[0087] Calculate the cosine similarity between the background point vector and the query set feature vector in the query set to obtain a series of The superscript BG represents the potential background, k represents the kth sampling point, and the subscript QQ represents the cosine similarity between the query set itself and its own vector;

[0088] To mine the relevance of its own foreground points, the same operation is performed, and the threshold is set to 0.7, and a series of

[0089] Get the final prior mask:

[0090]

[0091] Among them, PM stands for Pri or Mask, which is the final prior mask, ReL is an activation function, and FG i Represents the foreground of the i-th sampling point.

[0092] The present invention also provides a small sample semantic segmentation system based on inter-class relations, including a data acquisition module and a processing module, the data acquisition module acquires query set and feature set images, and the output end of the data acquisition module is electrically connected to the processing module.

[0093] The processing module executes the method of the present invention, obtains an optimized priori mask, and performs small sample semantic segmentation. The system has a simple structure, obtains an optimized priori mask, and is conducive to small sample semantic segmentation.

[0094] In a preferred embodiment of the present invention, the processing module includes a feature self-aggregation FSA module, a cross support and query function aggregation CSQFA module and a priori mask pattern PMF module connected in sequence. The processing module is provided with three modules, FSA, CSQFA and PMF, and each module cooperates for convenient use.

[0095] More preferably, the feature self-aggregation FSA module includes a self-attention mechanism and a position encoding unit. The input of the feature self-aggregation FSA module is the query set features and the support set features. The features of the support set are masked, and the features of the query set are not processed. The query set features and the support set features are expanded into a c*k vector and added to the position encoding, and then the self-aggregation features are obtained through the self-attention mechanism, where k=h×w, c, h, w represent the channel dimensions, height, and width, respectively, and k represents the kth sampling point.

[0096] The Cross Support and Query Feature Aggregation (CSQFA) module includes a cross attention mechanism unit, which improves the input output and parameters based on the Transformer structure.

[0097] The prior mask pattern PMF module includes a calculation unit for sampling foreground points and background points and cosine similarity. The output of the cross support and query function aggregation CSQFA module is connected in series with the prior mask pattern PMF module and then electrically connected to the decoder. Cosine similarity is the result of multiplying vectors a and b divided by the product of the vector modulus of a and b. It reflects the angle between the two vectors. The smaller the angle, the closer the directions of the two vectors.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0099] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A small sample semantic segmentation method based on inter-class relations, characterized in that: The steps include: Get the query set and feature set images; Input the query set and feature set images into the feature extractor to extract query set features and support set features; Set up feature self-aggregation FSA module, cross support and query function aggregation CSQFA module and prior mask graph PMF module to build a feature aggregation network across support set and query set; Input the query set features and the support features into the feature self-aggregation FSA module, reshape the query set features and the support features into multiple local features in spatial position, add the local feature vector and the corresponding position vector as the input of the transformer, and obtain the self-aggregation feature; The aggregated features are fed into the cross-support and query feature aggregation CSQFA module to obtain dense pixel-wise attention features across the support set and query set; The dense pixel-by-pixel attention features across the support set and query set are fed into the prior mask graph PMF module to obtain the final prior mask for final classification.

2. The small sample semantic segmentation method based on inter-class relationship according to claim 1, characterized in that: The query set features and support features are input into the feature self-aggregation FSA module to obtain the self-aggregation features as follows: Reshape the query set features and support features into r local features in spatial position, k = h × w, h, w represent the height and width of the input features respectively, and k represents the kth sampling point; Add the local feature vector and the corresponding position vector as the input Z of the transformer in the feature self-aggregation FSA module S , The vector representing the pixel point, E pos is a vector using sinusoidal position encoding; The output of the FSA module is based on the standard attention operation in the transformer encoder: Among them, Attention represents the attention operation, Q, K, and V represent query, key, and value vectors respectively, and d k is the dimension of the key vector, and in, is the result obtained by adding the local vector and the sine code to obtain z, and then calculating it through the attention formula, that is, the result after the self-attention operation; W q , W k , W v are learnable parameters; After a normalization layer and a multi-layer perceptron:

3. The small sample semantic segmentation method based on inter-class relationship according to claim 1, characterized in that: Get dense pixel-wise attention features across the support set and query set as: Convert the three-dimensional input features into two dimensions, and use each pixel as a token to obtain Q, K, and V after a linear mapping. in, is the dense pixel-by-pixel attention feature obtained by aggregating the query set features S, is the dense pixel-by-pixel attention feature obtained by aggregating S with the query set feature Q; Q, K, V represent query, key, and value vectors respectively; F Q and F S represent query set features and support set features respectively, W q ,W k ,W v ∈R d×d is a learnable parameter, d is the hidden layer dimension; CSQFA module through Q q S k To measure the similarity of all support set pixels to a specific query set pixel, and S v Multiplication aggregation from F S Foreground information, where Q q W q F Q ,S k W k F S ,S v W v F S , W q , W k , W v is a learnable parameter, F Q is the query set feature, F S is the support set feature; Q q Represents the Query obtained by changing the query set, S k represents the key obtained by changing the support set, S v Represents the value obtained by changing the support set; Interchange the backing set and query set, sharing the same parameters.

4. The small sample semantic segmentation method based on inter-class relationship according to claim 1, characterized in that: The dense pixel-by-pixel attention features across the support set and the query set are input into the prior mask graph PMF module to obtain the final prior mask. The specific steps are: Mask out high-level support features with corresponding masks The superscript i represents high-level features, s represents support set, c, h, and w represent channel dimensions, height, and width respectively. and high-level queryset features Expand F∈R c×k , k = h × w; Based on the sampled pixel and each pixel of the query set feature, the cosine similarity calculation is used to measure the similarity between the foreground pixel in the support set and each pixel in the query set. The subscript SQ represents the calculation of cosine similarity between the query set and the support set, and the superscript FG represents foreground; right After standardization, the threshold is set to 0.3 for sampling, and sampling points with confidence levels lower than 0.3 are considered potential background points; Calculate the cosine similarity between the background point vector and the query set feature vector in the query set to obtain a series of The superscript BG represents the potential background, k represents the kth sampling point, and the subscript QQ represents the cosine similarity between the query set itself and its own vector; To mine the relevance of its own foreground points, the same operation is performed, and the threshold is set to 0.7, and a series of Get the final prior mask: Among them, PM stands for Prior Mask, which is the final prior mask; ReLU is an activation function, FG i Represents the foreground of the i-th sampling point.

5. A small sample semantic segmentation system based on inter-class relations, characterized in that: It includes a data acquisition module and a processing module, wherein the data acquisition module acquires the query set and the feature set image, and the output end of the data acquisition module is connected to the processing module; The processing module executes the method described in any one of claims 1-4, obtains an optimized priori mask, and performs small sample semantic segmentation.

6. The small sample semantic segmentation system based on inter-class relationship according to claim 5, characterized in that: The processing module includes a feature self-aggregation FSA module, a cross support and query function aggregation CSQFA module and a priori mask pattern PMF module which are connected in sequence.

7. The small sample semantic segmentation system based on inter-class relationship according to claim 6, characterized in that: The feature self-aggregation FSA module includes a self-attention mechanism and a position encoding unit; The input of the feature self-aggregation FSA module is the query set features and the support set features. The features of the support set are masked, and the features of the query set are not processed. The query set features and the support set features are expanded into c*k vectors and added to the position code, and then the self-aggregation features are obtained through the self-attention mechanism, where k = h×w, c, h, w represent the channel dimensions, height, and width respectively, and k represents the kth sampling point.

8. The small sample semantic segmentation system based on inter-class relationship according to claim 6, characterized in that: The cross support and query function aggregation CSQFA module includes a cross attention mechanism unit.

9. The small sample semantic segmentation system based on inter-class relationship according to claim 6, characterized in that: The priori mask pattern PMF module includes a calculation unit for sampling and cosine similarity of foreground points and background points, and the output of the cross support and query function aggregation CSQFA module is connected in series with the priori mask pattern PMF module and then connected to the decoder.

Images