A Few-Shot Semantic Segmentation Method and System Guided by a Similarity Matrix

Abstract

The present invention discloses a few-shot semantic segmentation method and system guided by a similarity matrix. The method first extracts the features of the support image and the query image respectively through deep learning; then calculates the similarity matrix between the query image and the target category by using the support image and its ground-truth label; then obtains the feature representation of the target category as the prototype of the target category through masked average pooling; processes the obtained multiple features through channel attention, inputs them into an atrous spatial pyramid pooling network, and then outputs the final refined segmentation result through convolution classification; finally, optimizes the prototype feature generation network and the refined semantic segmentation module network with multi-feature fusion by using training data, and performs pixel-level segmentation on the query images of new categories by using the trained model. The few-shot semantic segmentation method of the present invention has prominent advantages such as simple application and strong versatility.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a few-shot semantic segmentation method and system guided by a similarity matrix. Background Art

[0002] Semantic segmentation aims to segment different objects in an image and identify the semantic type of each pixel, which is one of the core means of visual understanding and is widely applied to fields such as scene understanding of autonomous unmanned systems and intelligent manufacturing. Driven by deep learning, the accuracy of semantic segmentation has been greatly improved in recent years. However, deep learning relies on a large number of labeled samples. Facing the problem of scarce labeled samples in practical applications, how to achieve high-precision semantic segmentation under few-shot conditions has become a difficult problem that urgently needs to be solved. The goal of few-shot segmentation is to accurately segment a query image given a few labeled support images. Currently, existing few-shot segmentation methods commonly use the meta-learning paradigm to achieve generalizable segmentation for new classes. This method samples a series of classes in the basic dataset during training to perform the segmentation task, so as to simulate the few-shot scenario of new classes during testing or inference, achieving the purpose of "learning to learn". On this basis, ideas such as double-branch models, embedded learning based on prototype networks, and prior map-guided feature learning have been proposed and applied to few-shot segmentation. The model based on double-branch guidance mainly includes a "support branch" and a "query branch". By learning and analyzing sample data and existing knowledge through the support branch, the query branch is guided to quickly learn new knowledge and achieve the segmentation of new samples. Another idea is based on prototype learning, that is, embedding samples into a low-dimensional space according to the existing knowledge of target samples, constraining the hypothesis space of the model and then learning, so as to improve the convergence speed and learning efficiency of the model. In addition, considering the correlation between support and query images, the maximum value of the cosine correlation of the high-level features of the support image and the query image is used as a prior map, which is fused with other features to guide semantic segmentation. However, the cosine correlation calculation cannot well locate the positions of target classes in the query image. In addition, the fusion module is extremely prone to the problem of the meta-learner's bias towards the classes seen during training.

[0003] In a few-shot semantic segmentation model, since the discriminative features of each category are different, how to improve the generality of the few-shot semantic segmentation model under different categories is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] The present invention proposes a few-shot semantic segmentation method and system guided by a similarity matrix, extracts a similarity matrix and prototype representations from a small number of support labeled samples, and performs multi-feature fusion, thereby realizing few-shot semantic segmentation.

[0005] The technical solutions provided by the present invention are as follows:

[0006] On the one hand, a few-shot semantic segmentation method guided by a similarity matrix includes the following steps:

[0007] Step 1: Obtain the sample data for training and testing;

[0008] The sample data for training and testing includes query images, support images with determined target categories, and the target category semantic segmentation binary ground truth masks corresponding to the images;

[0009] Step 2: Use the semantic feature matrix extraction network trained on a large-scale dataset to extract the feature matrices of the support images and query images respectively;

[0010] Step 3: Use the feature matrix of the query image, the feature matrix of the support image, and the binary ground truth mask to obtain the similarity matrix between the query image and the target category;

[0011] Step 4: Construct a few-shot semantic segmentation model guided by the similarity matrix;

[0012] The few-shot semantic segmentation model guided by the similarity matrix includes a connected prototype feature generation network and a refined semantic segmentation module with multi-feature fusion; the prototype feature generation network calculates the prototype features of the target category; the refined semantic segmentation module with multi-feature fusion outputs a refined binary semantic segmentation result;

[0013] Step 5: Train the few-shot semantic segmentation model guided by the similarity matrix;

[0014] Input the feature matrices and similarity matrix obtained from the query images, support images, and the corresponding target category semantic segmentation ground truth masks in the training sample data into the few-shot semantic segmentation model guided by the similarity matrix, output the binary semantic segmentation result of the query image, calculate the model parameter gradients using the semantic segmentation loss and update the model parameters, traverse the images in the training sample dataset, and iterate to the specified number of times to complete the training;

[0015] Step 6: For the query images, support images, and the target category semantic segmentation binary ground truth masks of new target categories in the test sample data, after performing Steps 2 - 3, input the obtained output data into the trained few-shot semantic segmentation model guided by the similarity matrix to perform semantic segmentation on the query images of the new target categories to be segmented.

[0016] The target category semantic segmentation binary ground truth masks of the support images corresponding to the images to be segmented are known;

[0017] Furthermore, the process for obtaining the similarity matrix between the query image and the target category is as follows:

[0018] Step 3-1: Based on the target category semantic segmentation binary true mask of the support image corresponding to the query image, extract the valid region of the support image;

[0019] Step 3-2: Input the feature vectors of all pixels in the feature matrix of the query image and the feature vectors of all pixels in the feature matrix of the valid region of the support image into the fully connected layer. After being processed by the fully connected layer, obtain the output data of the fully connected layer;

[0020] Step 3-3: Use a second-order function to calculate the similarity matrix between the feature matrix of the valid region of the support image and the feature matrix of the query image, and take the average value of the similarity matrices between the feature matrix of the query image and the feature matrices in all valid regions of the support images as the similarity matrix between the query image and the target category;

[0021] Using the second-order function to calculate the similarity matrix between the feature matrix of the valid region of the support image and the feature matrix of the query image means using the following calculation formula to calculate the similarity between each pixel in the query image and each pixel in the valid region of the support image in sequence, and forming the similarity matrix between the query image and the target category of the support image with the similarities between all pixels in the query image feature matrix and the target category in the support image;

[0022] d1(i, j) = (Aw i ) T (Aw i )+(Az j ) T (Az j )-(Bw i ) T (Bz j )+c T w i +c T z j

[0023] s1(i, j) = 1 - d1(i, j)

[0024]

[0025] where, w i represents the feature vector of the i-th pixel in the query image feature matrix, z j represents the feature vector of the j-th valid pixel in the support image valid region feature matrix, A and B are both n out ×n in matrices, c is an n out ×1 vector, the sizes of n out and n in are determined by the target category in the support image, Aw i and Azj , Bw i , Bz j , c T w i , c T z j are respectively the output data of the fully connected layer; d1(i, j) represents the amount of computation; s1(i, j) represents the similarity between the i-th pixel in the query image feature matrix and the j-th pixel in the effective region feature matrix of the support image, and similarity1(i) represents the similarity between the i-th pixel in the query image feature matrix and the target category in the support image.

[0026] Furthermore, the determination process of the target category in the support image corresponding to the query image for A, B, and c is as follows:

[0027] B1: Randomly assign values to A, B, and c;

[0028] B2: Sample in the support image corresponding to the query image;

[0029] In all support images and the corresponding target category semantic segmentation binary ground truth masks, collect two pairs to obtain the target support image, the auxiliary support image, the target category semantic segmentation binary ground truth mask of the target support image, and the target category semantic segmentation binary ground truth mask of the auxiliary support image;

[0030] B3: Calculate the similarity matrix of the target categories of the target support image and the auxiliary support image;

[0031] First, extract the effective region of the auxiliary support image;

[0032] Set the pixels with a value of zero in the target category semantic segmentation binary ground truth mask of the auxiliary support image as invalid pixels, and set the pixels with a value not equal to zero in the target category semantic segmentation binary ground truth mask of the auxiliary support image as valid pixels to obtain the effective region of the auxiliary support image;

[0033] Next, perform similarity operations using a second-order function;

[0034] d1(i, j) = (Ax i ) T (Ax i ) + (Ay j ) T (Ay j ) - (Bx i ) T (By j ) + c T x i + c T y j

[0035] s1(i, j) = 1 - d1(i, j)

[0036]

[0037] where x i represents the feature vector of the i-th pixel in the target support image feature matrix, y j represents the feature vector of the j-th valid pixel in the feature matrix of the valid region of the auxiliary support image, A and B are matrices of n out ×n in respectively, c is a vector of n out ×1, Ax i 、Ay j 、Bx i 、By j 、c T x i 、c T y j are the outputs of the fully connected layer respectively; s1(i, j) represents the similarity between the i-th pixel in the target support image feature matrix and the j-th pixel in the feature matrix of the valid region of the auxiliary support image, d1(i, j) represents the computational amount; similarity1(i) represents the similarity between the i-th pixel in the target support image feature matrix and the target category in the auxiliary support image, and the similarity1(i) of all pixels in the target support image feature matrix forms the similarity matrix of the target category between the target support image and the auxiliary support image;

[0038] B4: Calculate the loss function using the similarity matrix of the target category between the target support image and the auxiliary support image, and the binary segmentation ground truth mask of the target category of the target support image:

[0039]

[0040]

[0041] where H and W represent the length and width of the target support image respectively, and mask(i) represents the value of the i-th pixel of the binary segmentation ground truth mask of the target category of the target support image;

[0042] B5: Calculate the gradient of the fully connected layer parameters and update the fully connected layer parameters;

[0043]

[0044] where θ represents the fully connected layer parameters, i.e., θ = {A, B, c}, γ represents the gradient descent rate, Loss1 represents the loss function between the similarity output of the target support image and the binary segmentation ground truth mask of the target category of the target support image, and k represents the number of iterations;

[0045] B6: Repeat steps B2 - B5 until all possible combinations are traversed;

[0046] B7: Repeat steps B2 - B6 until the optimized iteration times are met.

[0047] Furthermore, the extraction of the semantic feature matrix using the semantic feature matrix extraction network means:

[0048] Step 2 - 1: Random sampling of images;

[0049] According to the target categories to be segmented in the query image, select k support images containing the target categories and the binary segmentation ground truth masks of the target categories of the support images;

[0050] Step 2 - 2: Image feature extraction;

[0051] Use the semantic feature matrix extraction network trained on the large - scale ImageNet dataset to extract features from the support images and the query image respectively, obtaining the support image feature matrix and the query image feature matrix.

[0052] The semantic feature extraction network adopts any one of resnet and vgg;

[0053] Furthermore, the prototype feature of the target category is obtained by reducing the dimension of the feature matrix of the support image obtained by the semantic feature matrix extraction network, then calculating the masked average pooling, and taking its average value;

[0054]

[0055]

[0056] Among them, represents the feature vector of the i - th pixel of the feature matrix of the support image after dimensionality reduction, represents the i - th pixel after downsampling the binary segmentation ground truth mask of the target category of the support image, p j represents the prototype feature of the j - th support image of the target category, p represents the prototype feature of the target category, and K represents the number of support images.

[0057] Furthermore, the refined semantic segmentation module for multi - feature fusion splices the feature matrix of the query image, the similarity matrix between the query image and the target category, and the prototype feature of the target category, performs channel attention processing, inputs it into the atrous spatial pyramid pooling network, and then outputs the final refined segmentation result through convolution classification.

[0058] Further, the semantic segmentation loss is obtained by calculating the cross-entropy loss pixel by pixel and then calculating the average value of the cross-entropy losses of all pixels.

[0059] Further, the parameter update of the prototype feature generation network and the refined semantic segmentation module network with multi-feature fusion is performed using the stochastic gradient descent algorithm.

[0060] On the other hand, a few-shot semantic segmentation system guided by a similarity matrix includes:

[0061] Obtain training and test sample data units: The sample data for training and testing includes query images, support images with determined target categories, and corresponding target category semantic segmentation binary ground truth masks.

[0062] Semantic feature matrix acquisition unit: Use the semantic feature matrix extraction network trained with a large-scale dataset to extract features from the support image and the query image respectively, and obtain the support image feature matrix and the query image feature matrix respectively.

[0063] Similarity matrix generation unit: Use the feature matrices of the support image and the query image obtained by the semantic feature matrix extraction network, and extract the valid region of the support image according to the target category semantic segmentation binary ground truth mask corresponding to the support image. Use the similarity calculation module to calculate the similarity matrix between the query image and each support image respectively, and take the average value as the similarity matrix between the query image and the target category.

[0064] Construct a few-shot semantic segmentation model unit guided by a similarity matrix: The few-shot semantic segmentation model guided by the similarity matrix includes a connected prototype feature generation network and a refined semantic segmentation module with multi-feature fusion; wherein, the prototype feature generation network calculates the prototype features of the target category based on the feature matrix of the support image and the corresponding target category semantic segmentation binary ground truth mask; the refined semantic segmentation module with multi-feature fusion outputs a refined binary semantic segmentation result under the guidance of the similarity matrix.

[0065] Training unit: Sample in the training data, input the support image, the ground truth, and the query image into the trained semantic feature matrix extraction network and the similarity calculation module, and then input the output results into the few-shot semantic segmentation model guided by the similarity matrix. Use the output binary segmentation result to calculate the cross-entropy loss, and perform parameter update on the few-shot semantic segmentation model guided by the similarity matrix using the stochastic gradient descent algorithm.

[0066] Segmentation unit: For the query image, support image and the target class semantic segmentation binary true mask of the new target class in the test sample data, input them into the trained semantic feature matrix extraction network and similarity calculation module, and input the obtained output data into the few-shot semantic segmentation model guided by the trained similarity matrix to perform semantic segmentation on the query image to be segmented.

[0067] In another aspect, a readable storage medium stores a computer program, and the computer program is called by a processor to implement:

[0068] The above-mentioned few-shot semantic segmentation method guided by a similarity matrix.

[0069] Beneficial effects

[0070] The technical solution of the present invention provides a few-shot semantic segmentation method and system guided by a similarity matrix. This method first extracts the features of the support image and the query image through deep learning respectively; then uses the support image and its true label to update the parameters of the similarity calculation module to obtain the similarity matrix between the query image and the target class; then obtains the feature representation of the target class through masked average pooling as the target class prototype; secondly, processes the obtained multiple features through channel attention, inputs them into the atrous spatial pyramid pooling network, and then outputs the final refined segmentation result through convolution classification; finally, optimizes the prototype feature generation network and the refined semantic segmentation module network of multi-feature fusion through training data, and uses the trained model to perform pixel-level segmentation on the query image of the new class. The few-shot semantic segmentation method of the present invention has outstanding advantages such as simple application and strong versatility.

[0071] The technical solution provided by the present invention can realize the generalization application of the semantic segmentation model on other classes with only one training.

[0072] Compared with the existing few-shot semantic segmentation methods, the advancement of the present invention is reflected in:

[0073] 1) A method for calculating the similarity matrix between the query matrix and the target class is proposed. Compared with the similarity matrix generation method without learnable parameters, it identifies the discriminative features of the target class and greatly improves the versatility.

[0074] 2) A refined semantic segmentation module of multi-feature fusion is proposed to alleviate the bias problem of the meta-learner towards the classes seen during training, perform more detailed segmentation on the rough localization of the similarity matrix, and obtain a refined semantic segmentation result. Description of the drawings

[0075] Figure 1 It is a schematic diagram of the execution process of the method described in the technical solution of the present invention;

[0076] Figure 2 This is the overall block diagram of the method according to the technical solution of the present invention. Specific embodiments

[0077] The present invention will be further described in detail below with reference to the accompanying drawings.

[0078] The data set involved in the embodiments of the present invention is the pascal data set. The target category is the category to be segmented with little annotation information. Both the support image and the query image contain the target category. The support image is an image with annotations, and the query image is the image to be segmented.

[0079] A few-shot semantic segmentation method guided by a similarity matrix, as Figure 1 - Figure 2 shown, the specific steps are as follows:

[0080] Step 1: Obtain the sample data for training and testing;

[0081] The sample data for training and testing includes the query image with a determined target category, the support image, and the binary true mask of the semantic segmentation of the target category corresponding to the image;

[0082] Divide pascal into a training data set and a testing data set; set the training and testing category sets C train 、C test , sample the image-ground truth pair Q=(I q , M q ) of the query sample, and determine the target category, sample k support samples of the target category I q represent the support image and the query image respectively, M q represent the semantic segmentation true masks of the target categories of the support and query images respectively;

[0083] Step 2: Use the semantic feature matrix extraction network trained with a large-scale data set to extract the feature matrices of the support image and the query image respectively;

[0084] The extraction of the semantic feature matrix using the semantic feature matrix extraction network means:

[0085] Step 2-1: Random sampling of images;

[0086] According to the target category to be segmented in the query image, select k support images containing the target category and the binary true mask of the semantic segmentation of the target category of the support image;

[0087] Step 2-2: Image feature extraction;

[0088] Use the semantic feature matrix extraction network trained on the large-scale ImageNet dataset to extract features from the support image and the query image respectively, obtaining the support image feature matrix and the query image feature matrix.

[0089] The semantic feature extraction network uses any one of resnet and vgg;

[0090] Step 3: Obtain the similarity matrix between the query image and the target category;

[0091] The process of obtaining the similarity matrix between the query image and the target category is as follows:

[0092] Step 3-1: Based on the target category semantic segmentation binary truth mask of the support image corresponding to the query image, extract the effective region of the support image;

[0093] Step 3-2: Input the feature vectors of all pixels in the feature matrix of the query image and the feature vectors of all pixels in the feature matrix of the effective region of the support image into the fully connected layer. After being processed by the fully connected layer, obtain the output data of the fully connected layer;

[0094] Step 3-3: Use the second-order function to calculate the similarity matrix between the feature matrix of the effective region of the support image and the feature matrix of the query image, and take the average value of the similarity matrix between the feature matrix of the query image and the feature matrices in all effective regions of the support image as the similarity matrix between the query image and the target category;

[0095] Using the second-order function to calculate the similarity matrix between the feature matrix of the effective region of the support image and the feature matrix of the query image means using the following calculation formula to calculate the similarity between each pixel in the query image and each pixel in the effective region of the support image in turn, and forming the similarity matrix between the query image and the target category of the support image by the similarities between all pixels in the query image feature matrix and the target category in the support image;

[0096] d1(i, j) = (Aw i ) T (Aw i )+(Az j ) T (Az j )-(Bw i ) T (Bz j )+c T w i +c T z j

[0097] s1(i, j) = 1 - d1(i, j)

[0098]

[0099] Among them, w i represents the feature vector of the i-th pixel in the query image feature matrix, and z j represents the feature vector of the j-th valid pixel in the effective region feature matrix of the support image. 4A and B are matrices of n out ×n in respectively, and c is a vector of n out ×1. A, B, and c are determined by the target category in the support image. Aw i , Az j , Bw i , Bz j , c T w i , c T z j are the output data of the fully connected layer respectively; d1(i, j) represents the amount of computation; s1(i, j) represents the similarity between the i-th pixel in the query image feature matrix and the j-th pixel in the effective region feature matrix of the support image, and similarity1(i) represents the similarity between the i-th pixel in the query image feature matrix and the target category in the support image.

[0100] The process of determining A, B, and c by the target category in the support image corresponding to the query image is as follows:

[0101] B1: Randomly assign values to A, B, and c;

[0102] B2: Sample in the support image corresponding to the query image;

[0103] In all support images and the corresponding target category semantic segmentation binary ground truth masks, collect two pairs to obtain the target support image, the auxiliary support image, the target category semantic segmentation binary ground truth mask of the target support image, and the target category semantic segmentation binary ground truth mask of the auxiliary support image;

[0104] B3: Calculate the similarity matrix of the target categories of the target support image and the auxiliary support image;

[0105] First, extract the effective region of the auxiliary support image;

[0106] Set the pixels with a value of zero in the target category semantic segmentation binary ground truth mask of the auxiliary support image as invalid pixels, and set the pixels with a value not equal to zero in the target category semantic segmentation binary ground truth mask of the auxiliary support image as valid pixels to obtain the effective region of the auxiliary support image;

[0107] Then, perform similarity calculation using a second-order function;

[0108] d1(i, j) = (Axi ) T (Ax i )+(Ay j ) T (Ay j )-(Bx i ) T (By j )+c T x i +c T y j

[0109] s1(i, j) = 1 - d1(i, j)

[0110]

[0111] where x i represents the feature vector of the i-th pixel in the target support image feature matrix, y j represents the feature vector of the j-th valid pixel in the feature matrix of the effective region of the auxiliary support image, A and B are matrices of n out ×n in , c is a vector of n out ×1, Ax i , Ay j , Bx i , By j , c T x i , c T y j are the outputs of the fully connected layer respectively; s1(i, j) represents the similarity between the i-th pixel in the target support image feature matrix and the j-th pixel in the feature matrix of the effective region of the auxiliary support image, d1(i, j) represents the computational amount; similarity1(i) represents the similarity between the i-th pixel in the target support image feature matrix and the target category in the auxiliary support image, and the similarity1(i) of all pixels in the target support image feature matrix forms the similarity matrix of the target category between the target support image and the auxiliary support image;

[0112] B4: Calculate the loss function using the similarity matrix of the target category between the target support image and the auxiliary support image, and the target category semantic segmentation binary ground truth mask of the target support image:

[0113]

[0114]

[0115] Wherein, H and W respectively represent the length and width of the target support image, and mask(i) represents the value of the i-th pixel of the binary ground truth mask for semantic segmentation of the target category of the target support image;

[0116] B5: Calculate the parameter gradients of the fully connected layer and update the parameters of the fully connected layer;

[0117]

[0118] Wherein, θ represents the parameters of the fully connected layer, that is, θ = {A, B, c}, γ represents the gradient descent rate, Loss1 represents the loss function of the similarity output of the target support image and the binary ground truth mask for semantic segmentation of the target category of the target support image, and k represents the number of iterations;

[0119] B6: Repeat steps B2 - B5 until all possible combinations are traversed;

[0120] B7: Repeat steps B2 - B6 until the optimization iteration number is satisfied.

[0121] Step 4: Construct a few-shot semantic segmentation model guided by a similarity matrix;

[0122] The few-shot semantic segmentation model guided by the similarity matrix includes a connected prototype feature generation network and a refined semantic segmentation module with multi-feature fusion; the prototype feature generation network calculates the prototype features of the target category; the refined semantic segmentation module with multi-feature fusion outputs a refined binary semantic segmentation result;

[0123] The prototype features of the target category are obtained by reducing the dimension of the feature matrix of the support image obtained by the semantic feature matrix extraction network, then calculating masked average pooling, and taking its average value;

[0124]

[0125]

[0126] Wherein, represents the feature vector of the i-th pixel of the feature matrix of the support image after dimensionality reduction, represents the i-th pixel after downsampling the binary ground truth mask for semantic segmentation of the target category of the support image, p j represents the prototype feature of the target category of the j-th support image, p represents the prototype feature of the target category, and K represents the number of support images.

[0127] The refined semantic segmentation module with multi-feature fusion concatenates the feature matrix of the query image, the similarity matrix between the query image and the target category, and the prototype features of the target category. After channel attention processing, it is input into the Atrous Spatial Pyramid Pooling (ASPP) network, and then through convolutional classification, the final refined segmentation result is output.

[0128] Step 5: Train the few-shot semantic segmentation model guided by the similarity matrix;

[0129] Input the feature matrix and similarity matrix obtained from the query image, support image, and the corresponding ground truth mask of the target category semantic segmentation in the training sample data into the few-shot semantic segmentation model guided by the similarity matrix, output the binary semantic segmentation result of the query image, calculate the model parameter gradient using the semantic segmentation loss and update the model parameters. Traverse the images in the training sample dataset and iterate to the specified number of times to complete the training.

[0130] Step 6: For the query image, support image, and the binary ground truth mask of the target category semantic segmentation of the support image of the new target category in the test sample data, after performing Steps 2 - 3, input the obtained output data into the trained few-shot semantic segmentation model guided by the similarity matrix to perform semantic segmentation on the query image of the new target category to be segmented.

[0131] This embodiment also provides a few-shot semantic segmentation system guided by the similarity matrix, including:

[0132] Unit for obtaining training and test sample data: The training and test sample data includes query images, support images with determined target categories, and the corresponding binary ground truth masks of the target category semantic segmentation;

[0133] Semantic feature matrix acquisition unit: Use the semantic feature matrix extraction network trained on a large-scale dataset to extract features from the support image and the query image respectively, and obtain the support image feature matrix and the query image feature matrix;

[0134] Similarity matrix generation unit: Use the feature matrices of the support image and the query image obtained by the semantic feature matrix extraction network, and extract the effective region of the support image according to the binary ground truth mask of the target category semantic segmentation corresponding to the support image. Use the similarity calculation module to calculate the similarity matrix between the query image and each support image respectively, and take the average value as the similarity matrix between the query image and the target category;

[0135] Building a Few-Shot Semantic Segmentation Model Unit Guided by Similarity Matrix: The few-shot semantic segmentation model guided by similarity matrix includes a connected prototype feature generation network and a refined semantic segmentation module with multi-feature fusion. Among them, the prototype feature generation network calculates the prototype features of the target category based on the feature matrix of the support image and the corresponding target category semantic segmentation binary ground truth mask. The refined semantic segmentation module with multi-feature fusion outputs a refined binary semantic segmentation result under the guidance of the similarity matrix.

[0136] Training Unit: Sampling in the training data, inputting the support image, ground truth, and query image into the trained semantic feature matrix extraction network and similarity calculation module, then inputting the output results into the few-shot semantic segmentation model guided by similarity matrix, calculating the cross-entropy loss using the output binary segmentation result, and using the stochastic gradient descent algorithm to update the parameters of the few-shot semantic segmentation model guided by similarity matrix.

[0137] Segmentation Unit: For the query image of the new target category in the test sample data, the support image, and the target category semantic segmentation binary ground truth mask of the support image, input them into the trained semantic feature matrix extraction network and similarity calculation module, and input the obtained output data into the trained few-shot semantic segmentation model guided by similarity matrix to perform semantic segmentation on the query image to be segmented.

[0138] For the specific implementation process of each module, please refer to the content of the above method and will not be elaborated here. It should be understood that the above division of functional modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. At the same time, the above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0139] This embodiment provides an electronic device, including:

[0140] One or more processors;

[0141] A memory storing one or more computer programs;

[0142] Wherein, the processor calls the computer program to implement: the above-mentioned few-shot semantic segmentation method guided by similarity matrix.

[0143] This embodiment also provides a readable storage medium storing a computer program, and the computer program is called by the processor to implement:

[0144] The above-mentioned few-shot semantic segmentation method guided by similarity matrix.

[0145] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the controller described in any of the foregoing embodiments, such as the hard disk or memory of the controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk equipped on the controller, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the readable storage medium may also include both the internal storage unit and the external storage device of the controller. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store the data that has been output or is to be output.

[0146] Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing readable storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0147] It should be understood that in the embodiments of the present invention, the so-called processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0148] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive. Therefore, the present invention is not limited to the examples described in the specific embodiments. Any other embodiments obtained by those skilled in the art based on the technical solution of the present invention, whether modified or replaced, as long as they do not depart from the spirit and scope of the present invention, also fall within the protection scope of the present invention.

Claims

1. A few-shot semantic segmentation method guided by a similarity matrix, characterized in that It includes the following steps: Step 1: Obtain the sample data for training and testing; The sample data for training and testing includes the query images, support images with determined target categories, and the target category semantic segmentation binary ground truth masks corresponding to the images; Step 2: Use the semantic feature matrix extraction network trained with a large-scale dataset to extract the feature matrices of the support images and query images respectively; Step 3: Use the feature matrix of the query image, the feature matrix of the support image, and the binary ground truth mask to obtain the similarity matrix between the query image and the target category; Step 4: Construct a few-shot semantic segmentation model guided by the similarity matrix; The few-shot semantic segmentation model guided by the similarity matrix includes a connected prototype feature generation network and a refined semantic segmentation module with multi-feature fusion; the prototype feature generation network calculates the prototype features of the target category; the refined semantic segmentation module with multi-feature fusion outputs the refined binary semantic segmentation result; Step 5: Train the few-shot semantic segmentation model guided by the similarity matrix; Input the feature matrices and similarity matrix obtained from the query images, support images, and the corresponding target category semantic segmentation ground truth masks in the training sample data into the few-shot semantic segmentation model guided by the similarity matrix, output the binary semantic segmentation result of the query image, calculate the model parameter gradients using the semantic segmentation loss and update the model parameters, traverse the images in the training sample dataset, and iterate to the specified number of times to complete the training; Step 6: For the query images, support images, and the binary ground truth masks of the target categories of the support images in the new target categories of the test sample data, after performing Steps 2 - 3, input the obtained output data into the trained few-shot semantic segmentation model guided by the similarity matrix to perform semantic segmentation on the query images of the new target categories to be segmented.

2. The method according to claim 1, wherein The process for obtaining the similarity matrix between the query image and the target category is as follows: Step 3 - 1: Based on the target category semantic segmentation binary ground truth mask of the support image corresponding to the query image, extract the valid region of the support image; Step 3 - 2: Input the feature vectors of all pixels in the feature matrix of the query image and the feature vectors of all pixels in the feature matrix of the valid region of the support image into the fully connected layer, and after being processed by the fully connected layer, obtain the output data of the fully connected layer; Step 3 - 3: Use the second-order function to calculate the similarity matrix between the feature matrix of the valid region of the support image and the feature matrix of the query image, and take the average of the similarity matrices between the feature matrix of the query image and the feature matrices in all valid regions of the support images as the similarity matrix between the query image and the target category; Using the second-order function to calculate the similarity matrix between the feature matrix of the valid region of the support image and the feature matrix of the query image means calculating the similarity between each pixel in the query image and each pixel in the valid region of the support image in sequence using the following calculation formula, and forming the similarity matrix between the query image and the target category of the support image with the similarities between all pixels in the query image feature matrix and the target category in the support image; d1(i, j) = (Aw i ) T (Aw i ) + (Az j ) T (Az j ) - (Bw i ) T (Bz j ) + c T w i + c T z j s1(i, j) = 1 - d1(i, j) Among them, w i represents the feature vector of the i-th pixel in the query image feature matrix, and z j represents the feature vector of the j-th valid pixel in the feature matrix of the valid region of the support image. Both A and B are matrices of n out ×n in , c is a vector of n out ×1, and n out , n in are determined by the target category in the support image. Aw i , Az j , Bw i , Bz j , c T w i , c T z j are the output data of the fully connected layer respectively; d1(i, j) represents the amount of computation; s1(i, j) represents the similarity between the i-th pixel in the query image feature matrix and the j-th pixel in the feature matrix of the valid region of the support image, and similarity1(i) represents the similarity between the i-th pixel in the query image feature matrix and the target category in the support image.

3. The method according to claim 2, wherein The determination process of A, B, and c from the target categories in the support images corresponding to the query image is as follows: B1: Randomly assign values to A, B, and c; B2: Sample in the support images corresponding to the query image; Among all the support images and the corresponding target category semantic segmentation binary ground truth masks, collect two pairs to obtain the target support image, the auxiliary support image, the target category semantic segmentation binary ground truth mask of the target support image, and the target category semantic segmentation binary ground truth mask of the auxiliary support image; B3: Calculate the similarity matrix of the target categories of the target support image and the auxiliary support image; First, extract the valid region of the auxiliary support image; Set the pixels with a value of zero in the target category semantic segmentation binary ground truth mask of the auxiliary support image as invalid pixels, and set the pixels with a value not equal to zero in the target category semantic segmentation binary ground truth mask of the auxiliary support image as valid pixels to obtain the valid region of the auxiliary support image; Next, perform similarity operations using a second-order function; d1(i, j) = (Ax i ) T (Ax i ) + (Ay j ) T (Ay j ) - (Bx i ) T (By j ) + c T x i + c T y j s1(i, j) = 1 - d1(i, j) where x i represents the feature vector of the i-th pixel in the target support image feature matrix, and y j represents the feature vector of the j-th valid pixel in the feature matrix of the effective region of the auxiliary support image; B4: Use the similarity matrix of the target categories of the target support image and the auxiliary support image, and the target category semantic segmentation binary ground truth mask of the target support image to calculate the loss function: where H and W respectively represent the length and width of the target support image, and mask(i) represents the value of the i-th pixel in the target category semantic segmentation binary ground truth mask of the target support image; B5: Calculate the parameter gradient of the fully connected layer and update the parameters of the fully connected layer; where θ represents the parameters of the fully connected layer, that is, θ = {A, B, c}, γ represents the gradient descent rate, Loss1 represents the loss function of the similarity output of the target support image and the target category semantic segmentation binary ground truth mask of the target support image, and k represents the number of iterations; B6: Repeat steps B2 - B5 until all possible combinations are traversed; B7: Repeat steps B2 - B6 until the optimization iteration number is satisfied.

4. The method according to claim 1, characterized in that The extraction of the semantic feature matrix using the semantic feature matrix extraction network refers to: Step 2 - 1: Random sampling of images; According to the target categories to be segmented in the query image, select k support images containing the target categories and the target category semantic segmentation binary ground truth masks of the support images; Step 2 - 2: Image feature extraction; Use the semantic feature matrix extraction network trained on the large-scale ImageNet dataset to extract features from the support images and the query image respectively to obtain the support image feature matrix and the query image feature matrix.

5. The method according to claim 1, characterized in that The prototype feature of the target category is obtained by reducing the dimension of the feature matrix of the support image obtained by the semantic feature matrix extraction network, then calculating the masked average pooling, and taking its average value; Among them, represents the feature vector of the i-th pixel of the feature matrix of the support image after dimensionality reduction, represents the i-th pixel after downsampling the target class semantic segmentation binary truth mask of the support image, p j represents the prototype feature of the target class of the j-th support image, p represents the prototype feature of the target class, and K represents the number of support images.

6. The method according to claim 1, wherein The refined semantic segmentation module for multi-feature fusion concatenates the feature matrix of the query image, the similarity matrix of the query image and the target category, and the prototype feature of the target category, performs channel attention processing, inputs it into the atrous spatial pyramid pooling network, and then outputs the final refined segmentation result through convolution classification.

7. The method according to claim 1, wherein The semantic segmentation loss is obtained by calculating the cross-entropy loss pixel by pixel and then calculating the average value of the cross-entropy losses of all pixels.

8. The method according to claim 1, characterized in that, The parameter update of the prototype feature generation network and the refined semantic segmentation module network with multi-feature fusion is performed using the stochastic gradient descent algorithm.

9. A few-shot semantic segmentation system guided by a similarity matrix, characterized in that, It includes: Obtaining training and test sample data units: The sample data for training and testing includes query images, support images with determined target categories, and corresponding target category semantic segmentation binary ground truth masks; Semantic feature matrix acquisition unit: Using the semantic feature matrix extraction network trained with a large-scale dataset to extract features from the support image and the query image respectively, obtaining the support image feature matrix and the query image feature matrix; Similarity matrix generation unit: Using the feature matrices of the support image and the query image obtained by the semantic feature matrix extraction network, and extracting the valid region of the support image according to the target category semantic segmentation binary ground truth mask corresponding to the support image, using the similarity calculation module to calculate the similarity matrix between the query image and each support image respectively, and taking the average value as the similarity matrix between the query image and the target category; Constructing a few-shot semantic segmentation model unit guided by the similarity matrix: The few-shot semantic segmentation model guided by the similarity matrix includes a connected prototype feature generation network and a refined semantic segmentation module with multi-feature fusion; among them, the prototype feature generation network calculates the prototype features of the target category based on the feature matrix of the support image and the corresponding target category semantic segmentation binary ground truth mask; the refined semantic segmentation module with multi-feature fusion outputs a refined binary semantic segmentation result under the guidance of the similarity matrix; Training unit: Sampling in the training data, inputting the support image, the ground truth, and the query image into the trained semantic feature matrix extraction network and the similarity calculation module, then inputting the output results into the few-shot semantic segmentation model guided by the similarity matrix, using the output binary segmentation result to calculate the cross-entropy loss, and performing parameter update on the few-shot semantic segmentation model guided by the similarity matrix using the stochastic gradient descent algorithm; Segmentation unit: For the query image, support image, and the target category semantic segmentation binary ground truth mask of the support image of the new target category in the test sample data, inputting them into the trained semantic feature matrix extraction network and the similarity calculation module, and inputting the obtained output data into the trained few-shot semantic segmentation model guided by the similarity matrix to perform semantic segmentation on the query image to be segmented.

10. A readable storage medium, characterized in that: Stores a computer program, and the computer program is called by a processor to implement: A few-shot semantic segmentation method according to any one of claims 1-8.

Images