Small sample image segmentation method and system based on low-rank reconstruction

Abstract

The invention belongs to the technical field of small sample image segmentation, and particularly discloses a small sample image segmentation method and system based on low-rank reconstruction, and the method comprises the following steps: obtaining a support image and a query image, constructing a small sample image segmentation model, respectively extracting support features and query features, and constructing a small sample image segmentation model; calculating a feature correlation matrix between the support features and the query features, performing non-negative decomposition on the matrix to obtain a non-negative low-rank sub-matrix, reconstructing the feature correlation matrix, and mining a potential structure of the feature correlation matrix; and jointly interpreting and generating a position map of the target area by combining the spatial position information and the relation information of the feature correlation matrix, and inputting the generated position map, the support features and the query features into a decoder for decoding to obtain final segmentation prediction. By adopting the technical scheme, the potential structure of the feature correlation matrix is mined based on low-rank reconstruction, effective transmission of label information is promoted, and the segmentation performance of a small sample image segmentation task is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of small sample image segmentation, and relates to a small sample image segmentation method and system based on low-rank reconstruction. Background Art

[0002] Few-Shot Segmentation (FSS) is one of the important tasks in computer vision, which aims to segment unseen categories in an image based on a small amount of annotated image information. Traditional image segmentation methods rely on a large amount of annotated data to train a high-performance segmentation model, but the Few-Shot Segmentation technology only needs to provide a small number of annotated training samples (usually only one or a few annotated samples for each category) to achieve the segmentation of new categories in the image.

[0003] Small sample image segmentation usually adopts a fragment-based training mechanism, which contains two sets during training and testing: a support set and a query set. In the training phase, the support set provides one or more support images containing unknown categories, and support masks about unknown categories (segmentation labels of unknown categories). The query set consists of query images containing unknown categories, and the query masks of unknown categories in the query images are known in the training phase and need to be predicted in the testing phase. There is no intersection between the training set and the test set in terms of categories. The goal of small sample image segmentation technology is to achieve segmentation of new categories in the query image by learning the small amount of image information about unknown categories provided by the support set, thereby promoting the generalization ability of the neural network model.

[0004] Small sample image segmentation is an emerging technology that can achieve efficient and accurate segmentation even with extremely limited labeled data, and has broad application prospects. Figure 1 A schematic diagram showing the few-shot image segmentation task.

[0005] Existing small sample image segmentation techniques can be mainly divided into two categories: prototype learning-based and affinity learning-based methods.

[0006] The prototype learning-based method mainly learns one or more prototypes about the target based on the known target features in the support image, and then uses the learned target prototype to locate the pixel position with similar features in the query image, thereby achieving target segmentation in the query image. This scheme inevitably loses the spatial information about the target in the support image because it involves a global average pooling operation when calculating the prototype, resulting in inaccurate correlation calculation between the target prototype in the support image and the features in the query image. If the target is noisy, poorly representative, or lacks diversity, the prototype will not be able to accurately represent the target category, resulting in poor generalization ability of the model when processing unseen categories, significantly reduced model segmentation performance, and difficulty in adapting to various complex scenarios.

[0007] Affinity learning-based methods directly calculate the correlation between the support image and the query image to achieve pixel-level feature matching, which retains the spatial information of the target compared to prototype learning-based methods. However, the pixel-level correlation matrix between features is very susceptible to noise interference, resulting in poor performance or segmentation errors when segmenting objects with complex structures, details, or large intra-class variations, that is, misclassifying irrelevant areas or backgrounds as part of the target object.

[0008] Other small sample segmentation techniques enhance the feature similarity between pixels or regions between query images and support images by introducing attention mechanisms and other methods. However, they do not solve the redundant noise caused by differences in appearance and shape within the class, and the model segmentation accuracy is disturbed. Especially when facing complex scenes and complex targets, the model segmentation robustness is low. Summary of the invention

[0009] The purpose of the present invention is to provide a small sample image segmentation method and system based on low-rank reconstruction to improve the segmentation performance of small sample image segmentation tasks.

[0010] In order to achieve the above object, the basic scheme of the present invention is: a small sample image segmentation method based on low-rank reconstruction, comprising the following steps:

[0011] Obtain support images and query images;

[0012] Based on the low-rank reconstruction network, a small sample image segmentation model is constructed;

[0013] Input the support image and the query image into the feature encoder of the small sample image segmentation model to extract the support features and the query features respectively;

[0014] The extracted features are input into the low-rank reconstruction layer of the small sample image segmentation model, the feature correlation matrix between the support features and the query features is calculated and the matrix is ​​non-negatively decomposed to obtain a non-negative low-rank sub-matrix, and then the non-negative low-rank sub-matrix is ​​used to reconstruct the correlation matrix to explore the potential structure of the feature correlation matrix;

[0015] Combine the spatial position information and relationship information of the correlation matrix, use the relationship interpreter to interpret the low-rank reconstructed feature correlation matrix and obtain a position map that can locate the target area, and the position map provides the position of the target for segmenting the target area;

[0016] The generated location map, support features, and query features are input into the decoder for decoding to obtain the final segmentation prediction.

[0017] The working principle and beneficial effects of this basic solution are as follows: This technical solution analyzes the pixel-level correlation matrix of the support image and the query image and finds that the pixel-level correlation matrix generated by the label is inherently low-rank. By exploring the potential structure of the pixel-level correlation matrix, restoring the low-rank property of the pixel-level correlation matrix through low-rank reconstruction, and interpreting the relationship of the reconstructed pixel-level correlation matrix, this method can effectively eliminate redundant relationships, realize the effective transfer of label information from the support image to the query image, and improve the segmentation performance of small sample image segmentation tasks.

[0018] Furthermore, the feature correlation matrix between the support features and the query features is calculated and the matrix is ​​non-negatively decomposed to obtain a non-negative low-rank sub-matrix, and then the non-negative low-rank sub-matrix is ​​used to reconstruct the correlation matrix. The method for mining the potential structure of the feature correlation matrix is:

[0019] According to the supported features and query features where h s 、w s and h q 、w q Represent the length and width of the supporting features and query features respectively, d represents the dimension of the feature, and the cosine similarity between the supporting features and the query features is calculated to obtain the correlation matrix

[0020] C=CosSimi(F q ,F s )

[0021] Among them, CosSimi represents cosine similarity;

[0022] Correlation Matrix Perform non-negative decomposition, assuming there is a non-negative matrix and And the noise matrix but:

[0023] C=GD T +E;

[0024] The rank of the corresponding matrix holds for the following relationship:

[0025]

[0026] Among them, Rank(.) represents the rank of the matrix;

[0027] r represents the dimension of the non-negative matrix G. By changing the size of r, the degree of low rank of the matrix can be controlled;

[0028] The objective function of the low-rank correlation matrix reconstruction layer is:

[0029]

[0030] Among them, G ij ≥0,D jk ≥0, i, j, k represent the subscripts of the matrix respectively; F represents the Frobenius norm;

[0031] Update criterion to solve the objective function:

[0032]

[0033] After n iterations, the reconstructed low-rank similarity matrix for:

[0034]

[0035] The feature correlation matrix between the supporting features and the query features is calculated and the matrix is ​​non-negatively decomposed to obtain a non-negative low-rank sub-matrix. The non-negative low-rank sub-matrix is ​​then used to reconstruct the correlation matrix to explore the potential structure of the feature correlation matrix.

[0036] Furthermore, the spatial position information and relationship information of the correlation matrix are combined, and the relational interpreter is used to interpret the low-rank reconstructed feature correlation matrix and obtain a position map that can locate the target area. The specific steps are as follows:

[0037] Import reconstructed low-rank correlation matrix and transform it into

[0038] The reconstructed low-rank correlation matrix after deformation As the input of the position decoding layer in the relation interpreter, a convolution block is used as the position decoding layer to decode its position information, where the convolution block consists of a 3×3 convolution layer, a normalization layer, and an activation layer.

[0039] The features output by the position decoding layer are used as the input of the relation decoding layer in the relation interpreter, and the relation is decoded using a convolutional block including a 1×1 convolutional layer and a normalization layer;

[0040] Output the feature map of the object location in the query image.

[0041] By integrating the spatial location information and relationship information of the correlation matrix, the target can be accurately positioned.

[0042] The present invention also provides a small sample image segmentation system based on low-rank reconstruction, comprising a data acquisition unit and a processing unit, wherein the data acquisition unit is used to acquire a support image and a query image, and an output end of the data acquisition unit is connected to an input end of the processing unit;

[0043] The processing unit executes the method of the present invention to perform small sample image segmentation.

[0044] This system is based on a small sample image segmentation network with low-rank reconstruction, which can effectively improve the accuracy of the small sample image segmentation model and enhance the robustness of the segmentation model.

[0045] Furthermore, a small sample image segmentation model is provided in the processing unit, and the small sample image segmentation model includes a feature encoder supporting image and query image sharing, multiple convolutional layers, a global average pooling layer, a low-rank reconstruction module, and a feature decoder;

[0046] The input end of the feature encoder receives the support image and the query image, and the output end of the feature encoder is connected to the convolutional layer;

[0047] One convolutional layer is connected to the feature decoder through a global average pooling layer, and the remaining convolutional layers are connected to a low-rank reconstruction module, which is connected to the feature decoder.

[0048] A small sample image segmentation framework based on a low-rank reconstruction network has a simple structure and is easy to use.

[0049] Furthermore, the relation interpreter includes a position decoding layer and a relation decoding layer, wherein the position decoding layer is a convolutional block consisting of a single 3×3 convolutional layer, a normalization layer, and an activation layer; and the relation decoding layer is a single convolutional block consisting of a 1×1 convolutional layer and a normalization layer.

[0050] The relation interpreter integrates the relationship between spatial location information and relation information for relation interpretation of low-rank correlation matrices, thereby improving segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the structure of the existing small sample image segmentation task;

[0052] Figure 2 It is a structural schematic diagram of a small sample image segmentation model of a small sample image segmentation method based on low-rank reconstruction of the present invention;

[0053] Figure 3 It is a structural schematic diagram of low-rank correlation reconstruction of a small sample image segmentation method based on low-rank reconstruction of the present invention;

[0054] Figure 4 It is a low-rank correlation reconstruction effect diagram of the small sample image segmentation method based on low-rank reconstruction of the present invention;

[0055] Figure 5 It is a structural schematic diagram of the relation interpreter of the small sample image segmentation system based on low-rank reconstruction of the present invention. DETAILED DESCRIPTION

[0056] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0057] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0058] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0059] The present invention discloses a small sample image segmentation method based on low-rank reconstruction. The main technical problem solved is to mine the potential structure of the pixel-level correlation matrix based on low-rank reconstruction, and to eliminate the redundant noise in the correlation matrix by restoring its potential structure. The potential structure of the pixel-level correlation matrix is ​​interpreted, thereby promoting the effective transmission of label information and improving the segmentation performance of the small sample image segmentation task. The small sample image segmentation method based on low-rank reconstruction includes the following steps:

[0060] Obtain support images and query images;

[0061] Based on the low-rank reconstruction network, a small sample image segmentation model is constructed, such as Figure 2 As shown in the figure, 'SharedEncoder' represents a feature encoder that supports sharing of images and query images; F* represents different features extracted; in the deep learning framework, 'Conv' represents a convolutional layer, and 'Pooling' represents a global average pooling operation. Represents the operation of corresponding pixel multiplication, 'Expansion' represents the expansion of features; 'Decoder' represents the feature decoder, 'Low-Rank Reconstruction Module' is the technical core of the present invention, representing the low-rank reconstruction module, and the entire module includes multiple low-rank reconstruction layers;

[0062] Input the support image and the query image into the feature encoder of the small sample image segmentation model to extract the support features and the query features respectively;

[0063] The extracted features are input into the low-rank reconstruction layer of the small sample image segmentation model, the feature correlation matrix between the support features and the query features is calculated and the matrix is ​​non-negatively decomposed to obtain a non-negative low-rank sub-matrix, and then the non-negative low-rank sub-matrix is ​​used to reconstruct the correlation matrix to explore the potential structure of the feature correlation matrix;

[0064] Combine the spatial position information and relationship information of the correlation matrix, use the relationship interpreter to interpret the low-rank reconstructed feature correlation matrix and obtain a position map that can locate the target area, and the position map provides the position of the target for segmenting the target area;

[0065] The generated location map, support features, and query features are input into the decoder for decoding to obtain the final segmentation prediction.

[0066] In a preferred embodiment of the present invention, Figure 3 As shown, the feature correlation matrix between the support features and the query features is calculated and the matrix is ​​non-negatively decomposed to obtain a non-negative low-rank sub-matrix, and then the non-negative low-rank sub-matrix is ​​used to reconstruct the correlation matrix. The method for mining the potential structure of the feature correlation matrix is:

[0067] According to the supported features and query features where h s 、w s and h q 、w q Represent the length and width of the supporting features and query features respectively, d represents the dimension of the feature, and the cosine similarity between the supporting features and the query features is calculated to obtain the correlation matrix

[0068] C=CosSimi(F q ,F s )

[0069] Among them, CosSimi represents cosine similarity;

[0070] Correlation Matrix Perform non-negative decomposition, assuming there is a non-negative matrix and And the noise matrix but:

[0071] C=GD T +E;

[0072] The rank of the corresponding matrix holds for the following relationship:

[0073] Rank(C)≤min(Rank(G),Rank(D))≤r<<min(Rank(h q w q ,h s w s ))

[0074] Among them, Rank(.) represents the rank of the matrix;

[0075] r represents the dimension of the non-negative matrix G. By changing the size of r, the degree of low rank of the matrix can be controlled;

[0076] The objective function of the low-rank correlation matrix reconstruction layer is:

[0077]

[0078] Among them, G ij ≥0,D jk ≥0, i, j, k represent the subscripts of the matrix respectively; F represents the Frobenius norm;

[0079] Update criterion to solve the objective function:

[0080]

[0081] After n iterations, the reconstructed low-rank similarity matrix for:

[0082] The effect of low-rank correlation reconstruction is shown in the figure Figure 4 As shown, Figure 4In the figure, 'Query' and 'Support' represent query image and support image respectively, 'Correlation Matrix' represents the correlation matrix between support image and query image, 'GT' represents the true correlation matrix generated by the labels of support image and query image, 'Baseline' represents the correlation matrix of the benchmark model, 'Ours' represents the correlation matrix reconstructed by the present invention, and 'Prediction' represents the segmentation prediction of the model. The reconstructed correlation matrix is ​​obviously closer to the true correlation matrix than the correlation matrix obtained by 'Baseline', and the corresponding segmentation result is obviously better than the segmentation result of the baseline.

[0083] Low-rank correlation matrix reconstruction can explore the potential structure of the correlation matrix through low-rank reconstruction, thereby eliminating redundant noise in the correlation matrix and improving the accuracy of small sample image segmentation models.

[0084] In a preferred embodiment of the present invention, in order to accurately decode the reconstructed low-rank correlation matrix Combine the spatial position information and relationship information of the correlation matrix, use the relationship interpreter to interpret the low-rank reconstructed feature correlation matrix and obtain a location map that can locate the target area, so as to accurately locate the target. The specific steps are as follows:

[0085] Import reconstructed low-rank correlation matrix and transform it into

[0086] The reconstructed low-rank correlation matrix after deformation As the input of the position decoding layer in the relation interpreter, a convolution block is used as the position decoding layer to decode its position information, where the convolution block consists of a 3×3 convolution layer, a normalization layer, and an activation layer.

[0087] The features output by the position decoding layer are used as the input of the relation decoding layer in the relation interpreter, and the relation is decoded using a convolutional block including a 1×1 convolutional layer and a normalization layer;

[0088] Output the feature map of the object location in the query image.

[0089] The present invention also provides a small sample image segmentation system based on low-rank reconstruction, including a data acquisition unit and a processing unit, the data acquisition unit is used to acquire a supporting image and a query image, and the output end of the data acquisition unit is electrically connected to the input end of the processing unit.

[0090] The processing unit executes the method of the present invention to perform small sample image segmentation. The system is based on a small sample image segmentation network with low rank reconstruction, which can effectively improve the accuracy of the small sample image segmentation model and enhance the robustness of the segmentation model.

[0091] In a preferred embodiment of the present invention, a small sample image segmentation model is provided in the processing unit, and the small sample image segmentation model includes a feature encoder that supports image and query image sharing, multiple convolutional layers, a global average pooling layer, a low-rank reconstruction layer, and a feature decoder.

[0092] The input end of the feature encoder receives the support image and the query image, and the output end of the feature encoder is electrically connected to the convolution layer. One convolution layer is connected to the feature decoder through a global average pooling layer, and the remaining convolution layers are connected to the low-rank reconstruction layer, which is connected to the feature decoder.

[0093] In a preferred embodiment of the present invention, Figure 5 As shown, the relation interpreter includes a position decoding layer and a relation decoding layer, wherein the position decoding layer is a convolution block consisting of a single 3×3 convolution layer, a normalization layer, and an activation layer, and the relation decoding layer is a single convolution block consisting of a 1×1 convolution layer and a normalization layer.

[0094] The relation interpreter integrates the relationship between spatial location information and relation information for relation interpretation of low-rank correlation matrices, thereby improving segmentation performance.

[0095] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0096] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A small sample image segmentation method based on low-rank reconstruction, characterized in that: The steps include: Obtain support images and query images; Based on the low-rank reconstruction network, a small sample image segmentation model is constructed; Input the support image and the query image into the feature encoder of the small sample image segmentation model to extract the support features and the query features respectively; The extracted features are input into the low-rank reconstruction layer of the small sample image segmentation model, the feature correlation matrix between the support features and the query features is calculated and the matrix is ​​non-negatively decomposed to obtain a non-negative low-rank sub-matrix, and then the non-negative low-rank sub-matrix is ​​used to reconstruct the correlation matrix to explore the potential structure of the feature correlation matrix; Combine the spatial position information and relationship information of the correlation matrix, use the relationship interpreter to interpret the low-rank reconstructed feature correlation matrix and obtain a position map that can locate the target area, and the position map provides the position of the target for segmenting the target area; The generated location map, support features, and query features are input into the decoder for decoding to obtain the final segmentation prediction.

2. The small sample image segmentation method based on low-rank reconstruction according to claim 1, characterized in that: The feature correlation matrix between the support features and the query features is calculated and the matrix is ​​non-negatively decomposed to obtain a non-negative low-rank sub-matrix. The non-negative low-rank sub-matrix is ​​then used to reconstruct the correlation matrix. The method for mining the potential structure of the feature correlation matrix is ​​as follows: According to the supported features and query features where h s 、w s and h q 、w q Represent the length and width of the supporting features and query features respectively, d represents the dimension of the feature, and the cosine similarity between the supporting features and the query features is calculated to obtain the correlation matrix C=CosSimi(F q ,F s ) Among them, CosSimi represents cosine similarity; Correlation Matrix Perform non-negative decomposition, assuming there is a non-negative matrix and And the noise matrix but: C=GD T +And; The rank of the corresponding matrix holds for the following relationship: Rank(c)≤min(Rank(G),Rank(D))≤r<<min(Rank(h q w q ,h s w s )) Among them, Rank(.) represents the rank of the matrix; r represents the dimension of the non-negative matrix G. By changing the size of r, the degree of low rank of the matrix can be controlled; The objective function of the low-rank correlation matrix reconstruction layer is: Among them, G ij ≥0, D jk ≥0, i, j, k represent the subscripts of the matrix respectively; F represents the Frobenius norm; Update criterion to solve the objective function: After n iterations, the reconstructed low-rank similarity matrix for:

3. The small sample image segmentation method based on low-rank reconstruction according to claim 1, characterized in that: Combine the spatial position information and relationship information of the correlation matrix, use the relationship interpreter to interpret the low-rank reconstructed feature correlation matrix and obtain a location map that can locate the target area. The specific steps are: Import reconstructed low-rank correlation matrix and transform it into The reconstructed low-rank correlation matrix after deformation As the input of the position decoding layer in the relation interpreter, a convolution block is used as the position decoding layer to decode its position information, where the convolution block consists of a 3×3 convolution layer, a normalization layer, and an activation layer. The features output by the position decoding layer are used as the input of the relation decoding layer in the relation interpreter, and the relation is decoded using a convolutional block including a 1×1 convolutional layer and a normalization layer; Output the feature map of the object location in the query image.

4. A small sample image segmentation system based on low-rank reconstruction, characterized in that: It includes a data acquisition unit and a processing unit, the data acquisition unit is used to acquire a support image and a query image, and the output end of the data acquisition unit is connected to the input end of the processing unit; The processing unit executes the method according to any one of claims 1 to 3 to perform small sample image segmentation.

5. The small sample image segmentation system based on low-rank reconstruction according to claim 4, characterized in that: The processing unit is provided with a small sample image segmentation model, and the small sample image segmentation model includes a feature encoder supporting image and query image sharing, multiple convolutional layers, a global average pooling layer, a low-rank reconstruction module, and a feature decoder; The input end of the feature encoder receives the support image and the query image, and the output end of the feature encoder is connected to the convolutional layer; One convolutional layer is connected to the feature decoder through a global average pooling layer, and the remaining convolutional layers are connected to a low-rank reconstruction module, which is connected to the feature decoder.

6. The small sample image segmentation system based on low-rank reconstruction according to claim 5, characterized in that: The relation interpreter includes a position decoding layer and a relation decoding layer, where the position decoding layer is a convolution block consisting of a single 3×3 convolution layer, a normalization layer, and an activation layer; the relation decoding layer is a single convolution block consisting of a 1×1 convolution layer and a normalization layer.

Images