Small sample semantic segmentation method and system based on inter-class relationship

Abstract

The application belongs to the technical field of semantic segmentation, and specifically discloses a small sample semantic segmentation method and system based on inter-class relationship, which comprises the following steps: acquiring query set and feature set pictures, extracting query set features and support set features; setting a feature self-aggregation FSA module, a cross-support and query function aggregation CSQFA module and a prior mask figure PMF module, constructing a cross-support set and query set feature aggregation network, obtaining aggregated features, dense pixel-by-pixel attention features of the cross-support set and the query set, inputting the dense pixel-by-pixel attention features of the cross-support set and the query set into the prior mask figure PMF module, and obtaining a final prior mask for final classification. According to the technical scheme, the CSQFA and the FASA expand the transformer to enlarge the difference between classes and reduce the difference within the class, and implement pixel-level dense matching, and the PMF improves the prototype mode and excavates background information to make the prior mask more accurate.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of semantic segmentation, and relates to a small sample semantic segmentation method and system based on inter-class relationship. BACKGROUND

[0002] In recent years, deep convolutional neural networks have made great progress in computer vision tasks such as object detection, semantic segmentation, style transfer, and panorama segmentation. However, these advances not only rely on deep networks, but also greatly depend on large-scale labeled datasets. In some data-starved fields such as medical image segmentation, the performance of deep convolutional neural networks has not reached the desired results, especially when the model is expanded to classes that have not been seen during model training.

[0003] To solve the above problems, weakly supervised semantic segmentation and many ways have been tried, but the problem of poor model scalability has not been solved. Humans are born with the ability to quickly recognize new classes using a small number of examples based on existing life prior knowledge. Inspired by this phenomenon, researchers have tried to develop a paradigm for small sample semantic segmentation (FSS) models that learn a segmentation pattern on existing labeled base classes (classes used during training).

[0004] The main challenge of small sample semantic segmentation models is how to utilize the existing information of the support set. Previous work mostly follows the prototype direction, that is, the information in the support set is compressed into a prototype vector through global pooling or clustering, and the prototype vector is used to match the features of the query set. However, the prototype method can easily lead to the loss of details in the support set. SUMMARY

[0005] The purpose of the present application is to provide a small sample semantic segmentation method and system based on inter-class relationship, which makes the prior mask more accurate.

[0006] To achieve the above purpose, the basic scheme of the present application is as follows: a small sample semantic segmentation method based on inter-class relationship, comprising the following steps:

[0007] Obtain the query set and the feature set pictures;

[0008] Input the query set and the feature set pictures into the feature extractor to extract the query set features and the support set features;

[0009] Set the feature self-aggregation FSA module, the cross-support and query function aggregation CSQFA module, and the prior mask figure PMF module to construct a cross-support set and query set feature aggregation network;

[0010] input the query set features and the support set features into a feature self-aggregation FSA module, reshape the query set features and the support set features in spatial positions into a plurality of local features, add the local feature vector and the corresponding position vector as the input of the transformer, and obtain the self-aggregation features;

[0011] input the aggregation features into a cross-support and query function aggregation CSQFA module, and obtain dense pixel-by-pixel attention features across the support set and the query set;

[0012] input the dense pixel-by-pixel attention features across the support set and the query set into a prior mask figure PMF module, and obtain the final prior mask for final classification.

[0013] The working principle and beneficial effects of the basic scheme are that the CSQFAN (cross-support set and cross-set feature aggregation network) is proposed to solve the small sample semantic segmentation problem, which includes three modules of FSA, CSQFA and PMF. The CSQFA and the FASA expand the transformer to expand the difference between classes and reduce the difference within the class and implement pixel-level dense matching. The FSA increases the similarity of the same class targets in the same picture. The CSQFA realizes the interaction of the support set features and the query set features. The PMF mines the inter-class relationship to replace the traditional prior mask. The PMF improves the prototype method and mines the background information to make the prior mask more accurate.

[0014] Further, the query set features and the support set features are input into the feature self-aggregation FSA module, and the method for obtaining the self-aggregation features is:

[0015] The query set features and the support set features are reshaped into r local features in spatial positions, k = h x w, h and w represent the height and width of the input features respectively, and k represents the kth sampling point;

[0016] The local feature vector and the corresponding position vector are added as the input Z of the transformer in the feature self-aggregation FSA module S , The vector represents the pixel point, and E pos is a vector using sinusoidal position encoding.

[0017] The output of the FSA module is based on the standard attention operation in the transformer encoder:

[0018]

[0019] Among them, Attention represents the attention operation, Q, K, and V represent the query, key, and value vectors respectively, and d kis the dimension of the key vector, and

[0020]

[0021] in, The result obtained by adding the local vector and the sine code to obtain z, and then calculating it through the attention formula, that is, the result after the self-attention operation; W q , W k , W v are learnable parameters;

[0022] After a normalization layer and a multilayer perceptron:

[0023]

[0024] The traditional visualization transformer based on the standard transformer takes a series of image blocks as a sequence as input. In contrast, this scheme adds the local feature vector and the corresponding position vector as the input of the transformer, increasing the similarity of similar objects in the same image.

[0025] Furthermore, we obtain dense pixel-wise attention features across the support set and query set as:

[0026]

[0027]

[0028] Convert the three-dimensional input features into two dimensions, and use each pixel as a token to obtain Q, K, and V after a linear mapping.

[0029] in, is the dense pixel-by-pixel attention feature obtained by aggregating the query set features S, is the dense pixel-by-pixel attention feature obtained by aggregating S with the query set feature Q; Q, K, V represent query, key, and value vectors respectively; F Q and F S represent query set features and support set features respectively, W q ,W k ,W v ∈R d×d is a learnable parameter, d is the hidden layer dimension;

[0030] CSQFA module through Q q S k To measure the similarity of all support set pixels to a specific query set pixel, and S vMultiplication aggregation from F S Foreground information, where Q q W q F Q ,S k W k F S ,S v W v F S , W q , W k , W v is a learnable parameter, F Q is the query set feature, F S is the support set feature; Q q Represents the query obtained by changing the query set, S k represents the key obtained by changing the support set, S v Represents the value obtained by changing the support set;

[0031] Interchange the backing set and query set, sharing the same parameters.

[0032] Dense pixel-by-pixel attention features are obtained across the support set and query set, realizing the interaction between support set features and query set features.

[0033] Furthermore, the dense pixel-by-pixel attention features across the support set and query set are input into the prior mask graph PMF module to obtain the final prior mask. The specific steps are:

[0034] Mask the high-level support features with the corresponding masks The superscript i represents high-level features, s represents support set, c, h, w represent channel dimension, height, and width respectively; and high-level queryset features Expand into F∈R c×k , k = h × w;

[0035] Based on the sampling pixel and each pixel of the query set feature, the similarity between the foreground pixel in the support set and each pixel in the query set is measured by cosine similarity calculation The subscript SQ represents the calculation of cosine similarity between the query set and the support set, and the superscript FG represents the foreground;

[0036] right After standardization, the threshold is set to 0.3 for sampling, and sampling points with confidence levels lower than 0.3 are considered potential background points;

[0037] Calculate the cosine similarity between the background point vector and the query set feature vector in the query set, and get a series of The superscript BG represents the potential background, k represents the kth sampling point, and the subscript QQ represents the cosine similarity between the query set itself and its own vector;

[0038] To explore the correlation of its own foreground points, the same operation is performed and the threshold is set to 0.7, and a series of

[0039] Get the final prior mask:

[0040]

[0041] Among them, PM stands for Pri or Mask, which is the final prior mask, ReL is an activation function, and FG i Represents the foreground of the i-th sampling point.

[0042] The dense pixel-by-pixel attention features across the support set and query set are fed into the prior mask graph PMF module to obtain the final prior mask with higher accuracy.

[0043] The present invention also provides a small sample semantic segmentation system based on inter-class relationship, comprising a data acquisition module and a processing module, wherein the data acquisition module acquires a query set and a feature set image, and an output end of the data acquisition module is connected to the processing module;

[0044] The processing module executes the method of the present invention, obtains an optimized priori mask, and performs small sample semantic segmentation.

[0045] This system has a simple structure and obtains optimized prior masks, which is beneficial for small sample semantic segmentation.

[0046] Furthermore, the processing module includes a feature self-aggregation FSA module, a cross support and query function aggregation CSQFA module and a priori mask pattern PMF module which are connected in sequence.

[0047] The processing module is equipped with three modules: FSA, CSQFA and PMF, which are coordinated to facilitate use.

[0048] Furthermore, the feature self-aggregation FSA module includes a self-attention mechanism and a position encoding unit;

[0049] The input of the feature self-aggregation FSA module is the query set features and the support set features. The support set features are masked and the query set features are not processed. The query set features and the support set features are expanded into a c*k vector and added to the position code. Then, the self-aggregated features are obtained through the self-attention mechanism, where k = h×w, c, h, w represent the channel dimension, height, and width respectively, and k represents the kth sampling point.

[0050] Furthermore, the cross support and query function aggregation CSQFA module includes a cross attention mechanism unit.

[0051] Furthermore, the priori mask pattern PMF module includes a calculation unit for foreground point and background point sampling and cosine similarity, and the output of the cross support and query function aggregation CSQFA module is connected in series with the priori mask pattern PMF module and then connected to the decoder.

[0052] Simple structure and easy to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the structure of the small sample semantic segmentation method based on inter-class relationship of the present invention;

[0054] Figure 2 Schematic diagram of the structure of FSA and CSQFA of the small sample semantic segmentation method based on inter-class relationship of the present invention;

[0055] Figure 3 Schematic diagram of the structure of the PMF of the small sample semantic segmentation method based on inter-class relationship of the present invention. DETAILED DESCRIPTION

[0056] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0057] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0058] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0059] The present invention discloses a small sample semantic segmentation method based on inter-class relationship, such as Figure 1 As shown, the following steps are included:

[0060] Get the query set and feature set images;

[0061] Input the query set and feature set images into the pre-trained feature extractor with fixed parameters to extract the query set features and support set features F S and F Q (Q stands for query set, S stands for support set);

[0062] like Figure 2 As shown in the figure, a feature self-aggregation FSA (Feature Self Aggregation) module, a cross support and query feature aggregation CSQFA (Cross Support and Query Feature Aggregation) module and a priori mask feature PMF (Prior mask feature) module are set to build a feature aggregation network across support sets and query sets; the role of FSA is to increase the similarity of similar targets in the same image, CSQFA realizes the interaction between support set features and query set features, and PMF is used to mine inter-class relationships to replace the traditional prior mask.

[0063] Input the query set features and support features into the feature self-aggregation FSA module, reshape the query set features and support features into multiple local features in spatial position, add the local feature vector and the corresponding position vector as the input of the transformer to obtain the self-aggregation feature;

[0064] The aggregated features are fed into the Cross-Support and Query Feature Aggregation (CSQFA) module to obtain dense pixel-wise attention features across the support set and query set.

[0065] The dense pixel-wise attention features across the support set and query set are fed into the prior mask graph PMF module to obtain the final prior mask for final classification.

[0066] In a preferred embodiment of the present invention, the query set features and the support features are input into the feature self-aggregation FSA module to obtain the self-aggregation features by:

[0067] Reshape the query set features and support features into r local features in spatial position, k = h × w, h, w represent the height and width of the input features respectively, and k represents the kth sampling point;

[0068] The traditional visualization transformer takes a series of image blocks as a sequence as input based on the standard transformer, while the present invention adds the local feature vector and the corresponding position vector as the input Z of the transformer in the feature self-aggregation FSA module. S , vector representing a pixel point, E pos is a vector using sine position encoding;

[0069] The output of the FSA module is based on the standard attention operation in the transformer encoder:

[0070]

[0071] where Attention represents the attention operation, Q, K, V represent query, key, value vectors respectively, d k is the dimension of the key vector, and is obtained

[0072]

[0073] where, is the result obtained by adding the local vector to the sine encoding, and then performing the attention formula operation, that is, the result after self-attention operation; W q , W k , W v are learnable parameters;

[0074] After a standardization layer (Layer Normalization, LN) and a multi-layer perceptron (Multi-Layer Perceptron, MLP):

[0075]

[0076] In a preferred scheme of the present application, the dense pixel-by-pixel attention feature obtained across the support set and the query set is:

[0077]

[0078] The three-dimensional input feature is converted into two dimensions, and each pixel point is taken as a Token after a linear mapping to obtain Q, K, and V;

[0079] where, is the dense pixel-by-pixel attention feature obtained by aggregating Q using the query set feature S, is the dense pixel-by-pixel attention feature obtained by aggregating S using the query set feature Q; Q, K, and V represent query, key, and value vectors, respectively; F Q and F S represent the query set feature and the support set feature, respectively, W q , W k , W v ∈R d×dare learnable parameters, d is the dimension of hidden layer;

[0080] The CSQFA module measures the similarity of all support set pixels to a particular query set pixel by Q q S k is multiplied by S v to aggregate foreground information from F S , where Q q represents W q F Q , S k represents W k F S , S v represents W v F S , W q , W k , W v are learnable parameters, F Q is the query set feature, F S is the support set feature; Q q represents Query obtained by changing the query set, S k represents key obtained by changing the support set, S v represents value obtained by changing the support set;

[0081] The support set and the query set are interchanged, and the same parameters are shared in the process.

[0082] In a preferred scheme of the application, as shown in Figure 3 , dense pixel-by-pixel attention features across the support set and the query set are input into a prior mask graph PMF module to obtain a final prior mask, and the specific steps are as follows:

[0083] The prior mask has many extended uses in the FFS task, because it can roughly locate the target class in the query set only by the support set and the mask corresponding to the support set.

[0084] The high-level support set feature is hidden by the corresponding mask wherein the superscript i represents the high-level feature, s represents the support set, c, h, and w represent the channel dimension, height, and width, respectively; and and the high-level query set feature are unfolded into F∈R c×k , k = h x w;

[0085] Based on the sampling pixel point and each pixel point of the query set feature, the similarity of the foreground pixel point in the support set and each pixel point in the query set is calculated by cosine similarity wherein the subscript SQ represents that the cosine similarity is calculated between the query set and the support set, and the superscript FG represents the foreground;

[0086] The background in the same picture has similarity, in order to find out these potential background points, the similarity between the feature vector of the background point and the feature vector of the query set is calculated, and a series of After standardization, the threshold is set to 0.3 for sampling, and the sampling point with a confidence lower than 0.3 is considered as a potential background point.

[0087] In the query set, the cosine similarity of the vector of the background point and the feature vector of the query set is calculated, and a series of The superscript BG represents potential background, k represents the kth sampling point, and the subscript QQ represents the query set itself and the vector calculation cosine similarity.

[0088] In order to mine the relevance of the foreground points of the query set, the same operation is performed, and the threshold is set to 0.7, and a series of

[0089] The final prior mask is obtained:

[0090]

[0091] Wherein, PM represents Prior Mask, which is the final prior mask, ReL is an activation function, FG i represents the foreground of the ith sampling point.

[0092] The application also provides a small sample semantic segmentation system based on inter-class relationship, comprising a data acquisition module and a processing module, the data acquisition module acquires query set and feature set pictures, and the output end of the data acquisition module is electrically connected with the processing module.

[0093] The processing module executes the method described in the application, obtains an optimized prior mask, and performs small sample semantic segmentation. The system has simple structure, can obtain an optimized prior mask, and is beneficial to small sample semantic segmentation.

[0094] In a preferred scheme of the application, the processing module comprises a feature self-aggregation FSA module, a cross-support and query function aggregation CSQFA module and a prior mask figure PMF module connected in sequence. The FSA, CSQFA and PMF modules are arranged in the processing module, and the modules cooperate, which is beneficial to use.

[0095] More preferably, the feature self-aggregation FSA module comprises a self-attention mechanism and a position encoding unit, the input of the feature self-aggregation FSA module is the query set feature and the support set feature, the feature of the support set is shielded by a mask, the feature of the query set is not processed, the query set feature and the support set feature are expanded into c*k vectors and added with the position encoding, and then the self-aggregation feature is obtained through the self-attention mechanism, wherein k=h×w, c, h and w represent channel dimension, height and width respectively, and k represents the kth sampling point.

[0096] The Cross Support and Query Feature Aggregation (CSQFA) module includes a cross-attention mechanism unit, which improves the input, output, and parameters based on the Transformer structure.

[0097] The prior mask pattern (PMF) module includes a unit that calculates cosine similarity between foreground and background point samples. The output of the Cross Support and Query Function Aggregation (CSQFA) module is connected in series with the prior mask pattern (PMF) module and then electrically connected to the decoder. Cosine similarity is the product of multiplying vectors a and b, divided by the product of the moduli of vectors a and b. It reflects the angle between two vectors; the smaller the angle, the closer the two vectors are in their directions.

[0098] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0099] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A small sample semantic segmentation method based on inter-class relationship, characterized by: The steps include: Get the query set and feature set images; Input the query set and feature set images into the feature extractor to extract query set features and support set features; Set up the feature self-aggregation FSA module, the cross-support and query function aggregation CSQFA module and the prior mask graph PMF module to build a feature aggregation network across the support set and query set; Input the query set features and support set features into the feature self-aggregation FSA module, reshape the query set features and support set features into multiple local features in terms of spatial position, and add the local feature vector and the corresponding position vector as the input of the transformer to obtain the self-aggregation feature; The aggregated features are fed into the Cross-Support and Query Feature Aggregation (CSQFA) module to obtain dense pixel-wise attention features across the support set and query set. The dense pixel-by-pixel attention features across the support set and query set are input into the prior mask graph PMF module to obtain the final prior mask for final classification; Input query set features and support set features into the feature self-aggregation FSA module to obtain the self-aggregation features as follows: Reshape the query set features and support set features into r local features in spatial position, k = h × w, h, w represent the height and width of the input features respectively, and k represents the kth sampling point; Add the local feature vector and the corresponding position vector as the input Z of the transformer in the feature self-aggregation FSA module S , The vector representing the pixel point, E pos is a vector using sinusoidal position encoding; The output of the FSA module is based on the standard attention operation in the transformer encoder: Among them, Attention represents the attention operation, Q, K, and V represent query, key, and value vectors respectively, and d k is the dimension of the key vector, obtained in, The result obtained by adding the local vector and the sine code to obtain z, and then calculating it through the attention formula, that is, the result after the self-attention operation; W q , W k , W v are learnable parameters; After a normalization layer and a multilayer perceptron: The dense pixel-by-pixel attention features across the support set and query set are input into the prior mask graph PMF module to obtain the final prior mask. The specific steps are: Mask the high-level support features with the corresponding masks The superscript i represents high-level features, s represents support set, c, h, and w represent channel dimensions, height, and width respectively. and high-level queryset features Expand into F∈R c×k , k = h × w; Based on the sampling pixel and each pixel of the query set feature, the similarity between the foreground pixel in the support set and each pixel in the query set is measured by cosine similarity calculation The subscript SQ represents the calculation of cosine similarity between the query set and the support set, and the superscript FG represents the foreground; right After standardization, the threshold is set to 0.3 for sampling, and sampling points with confidence levels lower than 0.3 are considered potential background points; Calculate the cosine similarity between the background point vector and the query set feature vector in the query set, and get a series of The superscript BG represents the potential background, k represents the kth sampling point, and the subscript QQ represents the cosine similarity between the query set itself and its own vector; To explore the correlation of its own foreground points, the same operation is performed and the threshold is set to 0.7, and a series of Get the final prior mask: Among them, PM stands for Prior Mask, which is the final prior mask; ReLU is an activation function, FG i Represents the foreground of the i-th sampling point.

2. The small sample semantic segmentation method based on inter-class relationship according to claim 1, characterized in that: Get dense pixel-wise attention features across the support set and query set as: Convert the three-dimensional input features into two dimensions, and use each pixel as a token to obtain Q, K, and V after a linear mapping. in, is the dense pixel-by-pixel attention feature obtained by aggregating the query set features S, is the dense pixel-by-pixel attention feature obtained by aggregating S with the query set feature Q; Q, K, V represent query, key, and value vectors respectively; F Q and F S represent query set features and support set features respectively, W q , W k ,W v ∈R d×d is a learnable parameter, d is the hidden layer dimension; CSQFA module through Q q S k To measure the similarity of all support set pixels to a specific query set pixel, and S v Multiplication aggregation from F S Foreground information, where Q q W q F Q ,S k W k F S ,S v W v F S , W q , W k , W v is a learnable parameter, F Q is the query set feature, F S is the support set feature; Q q Represents the query obtained by changing the query set, S k represents the key obtained by changing the support set, S v Represents the value obtained by changing the support set; Interchange the backing set and query set, sharing the same parameters.

3. A small sample semantic segmentation system based on inter-class relations, characterized by: It includes a data acquisition module and a processing module. The data acquisition module obtains the query set and the feature set image. The output end of the data acquisition module is connected to the processing module. The processing module executes the method described in claim 1 or 2, obtains an optimized priori mask, and performs small sample semantic segmentation.

4. The small sample semantic segmentation system based on inter-class relationship according to claim 3, characterized in that The processing module includes a feature self-aggregation FSA module, a cross support and query function aggregation CSQFA module and a priori mask pattern PMF module which are connected in sequence.

5. The small sample semantic segmentation system based on inter-class relationship according to claim 4, characterized in that: The feature self-aggregation FSA module includes a self-attention mechanism and a position encoding unit; The input of the feature self-aggregation FSA module is the query set features and the support set features. The support set features are masked and the query set features are not processed. The query set features and the support set features are expanded into a c*k vector and added to the position code. Then, the self-aggregated features are obtained through the self-attention mechanism, where k = h×w, c, h, and w represent the channel dimension, height, and width respectively, and k represents the kth sampling point.

6. The small sample semantic segmentation system based on inter-class relationship according to claim 4, characterized in that The Cross Support and Query Feature Aggregation (CSQFA) module includes a cross attention mechanism unit.

7. The small sample semantic segmentation system based on inter-class relationship according to claim 4, characterized in that The priori mask pattern PMF module includes a calculation unit for foreground point and background point sampling and cosine similarity. The output of the cross support and query function aggregation CSQFA module is connected in series with the priori mask pattern PMF module and then connected to the decoder.

Images