A prototype-aware few-shot semantic segmentation method

Abstract

The present application relates to the field of computer vision, and discloses a prototype-aware few-sample semantic segmentation method, comprising the following steps: S1, collecting and preprocessing a data set including a training set and a test set; S2, dividing all images in the data set into four folds of equal size and non-intersecting according to categories; S3, extracting features of support images and query images using a pre-trained deep neural network; S4, performing cycle consistency screening based on support image features and query image features to obtain a credible query feature set; S5, iteratively optimizing foreground and background prototypes using the credible query feature set, and performing feature matching based on the optimized prototypes; S6, performing semantic segmentation of the input test image through the optimized model, and calculating the segmentation result. The present invention significantly improves the accuracy and robustness of few-sample semantic segmentation through high-confidence feature matching, cycle consistency screening, and cascade prediction optimization.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a few-sample semantic segmentation method based on prototype perception. Background Art

[0002] In the field of few-shot semantic segmentation, traditional methods usually rely on a large amount of labeled data to train the model in order to accurately segment the target area in the image. However, in many practical application scenarios, it is often very difficult to obtain sufficient labeled data, especially when the number of target categories is large or the data collection cost is high. To cope with this situation, existing few-shot semantic segmentation methods attempt to achieve segmentation of new categories by learning a small amount of labeled data, but these methods still have some challenges.

[0003] Among the existing few-shot segmentation techniques, commonly used methods include dense matching based on image features and class prototype similarity comparison. These methods can address the problem of data scarcity to a certain extent, but because they are overly dependent on the information of the support images, the accuracy of segmentation tends to drop significantly when encountering images with large intra-class diversity. In addition, the feature differences between the support images and the query images, as well as the diversity of intra-class features, will have a negative impact on the performance of the model. In this case, the model is easily disturbed by class-irrelevant information, resulting in inaccurate recognition of foreground pixels in the query image, which in turn affects the segmentation effect.

[0004] In summary, existing few-shot semantic segmentation techniques still have significant limitations when faced with problems such as data scarcity, intra-class diversity, and feature differences, and further improvements are needed to improve the accuracy and robustness of segmentation. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a prototype-aware few-sample semantic segmentation method, which significantly improves the accuracy and robustness of semantic segmentation through high-confidence feature matching and optimization under few-sample conditions.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a prototype-based few-sample semantic segmentation method, comprising the following steps:

[0007] Collect and preprocess the dataset including training and test sets;

[0008] By dividing all images in the dataset into four folds of equal size and non-intersecting according to categories, the data of each fold is further divided into a support set and a query set;

[0009] Extracting features of the support image and the query image using a pre-trained deep neural network, wherein the last block of the network is discarded and the parameters of the first two blocks are frozen;

[0010] Based on the supporting image features and query image features, cycle consistency screening is performed to obtain a credible query feature set;

[0011] Iteratively optimizing foreground and background prototypes using the credible query feature set, and performing feature matching based on the optimized prototypes;

[0012] Input the test image and perform semantic segmentation through the optimized model to calculate the segmentation result.

[0013] Preferably, the step of dividing all images in the data set into four folds of equal size and non-intersecting according to categories, and further dividing the data of each fold into a support set and a query set comprises:

[0014] Divide all images in the dataset into four folds of equal size and non-intersecting according to their categories;

[0015] The data in each fold is further divided into a support set and a query set. Query set D q ={X,Y}, where X i (X) represents the original image, Y i (Y) represents the mask corresponding to the image, K represents the shot value in the few samples, and represents the number of supporting images.

[0016] Preferably, the step of extracting features of the support image and the query image using a pre-trained deep neural network comprises:

[0017] Feature extraction of support images and query images by using pre-trained deep neural networks;

[0018] During feature extraction, the last block of the deep neural network is discarded and the parameters of the first two blocks are frozen so that no parameter updates are performed during training.

[0019] Specifically, the supported image F s and query image F q The feature extraction results are expressed as three-dimensional tensors:

[0020] {F S ,F q}∈R D×H×W

[0021] Where D represents the dimension of the feature, H and W represent the height and width of the image respectively.

[0022] Preferably, the pre-trained deep neural network is ResNet50.

[0023] Preferably, the step of performing cycle consistency screening based on supporting image features and query image features to obtain a credible query feature set comprises:

[0024] The extracted support image features F s and query image features F q Flatten into ID sequence C s and C q :

[0025] {C s ,C q}∈R D×HW

[0026] Where D represents the feature dimension, HW represents the total number of pixels after the image is flattened;

[0027] Based on the foreground mask y of the support image s Calculate the support foreground feature C s′ , the formula is:

[0028] C s′ =C s *y s

[0029] Calculate the similarity matrix A1 between the supporting foreground features and the query features, the formula is:

[0030]

[0031] in, Indicates C q Perform transposition;

[0032] In the similarity matrix A1, for each supporting foreground feature i', select the index j of the query feature that best matches it * , the formula is:

[0033]

[0034] Based on the selected query feature index j * , calculate the new low confidence query feature set C q' , the formula is:

[0035]

[0036] Calculate the support feature C again s With the new query feature set C q' The similarity matrix A2 between them is as follows:

[0037]

[0038] In the similarity matrix A2, for each query feature j * , select the index i of the supporting feature that best matches it * , the formula is:

[0039]

[0040] If the condition y is met s (i') = y s (i * ), the query feature is considered to satisfy the cycle consistency, and the pixel is added to the credible query feature set. middle.

[0041] Preferably, the step of iteratively optimizing the foreground and background prototypes using the credible query feature set and performing feature matching based on the optimized prototypes comprises:

[0042] Using the obtained trusted query feature set Calculate updated foreground and background prototypes and The formula is:

[0043]

[0044] Among them, α1 and α2 are hyperparameters, They are the foreground and background prototypes obtained after one iteration respectively;

[0045] Based on the updated foreground and background prototypes and Calculate the query image feature F q A foreground affinity graph f Affinity diagram with background A b , the formula is:

[0046]

[0047] Among them, cos(…) means calculating cosine similarity;

[0048] Affinity Diagram A f and A b Perform softmax processing to obtain the feature matching result M1, the formula is:

[0049] M1=softmax(A f ,A b )

[0050] Based on the feature matching result M1, filter out the high-confidence foreground query feature set and background query feature set The formula is:

[0051]

[0052] Among them, ε1 and ε2 are screening thresholds.

[0053] Preferably, the step of inputting the test image and performing semantic segmentation through the optimized model and calculating the segmentation result comprises:

[0054] The test image is fed into the optimized semantic segmentation model, which includes the iteratively optimized foreground and background prototypes. and

[0055] Use the optimized model to input the test image feature F q Perform segmentation prediction and calculate the foreground affinity map A of the test image f Affinity diagram with background A B ;

[0056] Affinity Diagram A f and A b Softmax processing is performed to obtain the segmented probability map M1, and the final segmentation mask is generated by thresholding M1.

[0057] The present invention also provides a prototype-aware few-sample semantic segmentation device, comprising:

[0058] Data collection module, used to collect and preprocess data sets including training sets and test sets;

[0059] A feature extraction module for extracting features of support images and query images using a pre-trained deep neural network;

[0060] A screening module, for performing cycle consistency screening based on supporting image features and query image features;

[0061] An optimization module, used to iteratively optimize foreground and background prototypes using a trusted query feature set and perform feature matching;

[0062] The segmentation module is used to input the test image and perform semantic segmentation through the optimized model.

[0063] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the above method is implemented.

[0064] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.

[0065] The present invention provides a few-sample semantic segmentation method based on prototype perception, which has the following beneficial effects:

[0066] 1. The SSPA model proposed in this invention can effectively identify and extract high-confidence query foreground pixels and their corresponding features by matching the support image features with the query image features. This matching mechanism ensures that the model can accurately capture key foreground information under the condition of few samples, thereby providing a reliable data basis for subsequent segmentation tasks.

[0067] 2. The present invention can effectively ensure the quality of query image features by using cycle consistency to filter query image features. The cycle consistency filtering mechanism can filter out irrelevant or noise features, significantly alleviating the negative impact of intra-class diversity and inter-class interference on segmentation results, thereby enhancing the robustness of the model in complex scenes.

[0068] 3. The present invention uses a cascade prediction method to iteratively optimize the preliminary prediction results and screen out high-confidence query features based on the previous prediction results. This process can further improve the quality of the query features, so that the model can more accurately distinguish the foreground and background in subsequent segmentation operations, thereby improving the overall segmentation accuracy.

[0069] 4. The present invention shows good segmentation performance in the case of few samples. Through the combination of multiple innovative methods, especially the matching and optimization of high-confidence features, the present invention can significantly improve the segmentation accuracy under the condition of limited data volume, showing strong practical application value, and is particularly suitable for the scenarios with few sample learning requirements in practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0071] Figure 2 A flowchart of a prototype-aware few-sample semantic segmentation method is provided for an embodiment of the present invention;

[0072] Figure 3 A schematic diagram of implementing cycle consistency screening in the present invention;

[0073] Figure 4 The embodiment of the present invention provides a ResNet-50 backbone based on PASCAL-5 i and COCO-20 i Comparison of 1-shot segmentation performance on the dataset;

[0074] Figure 5 The embodiment of the present invention provides a ResNet-50 backbone based on PASCAL-5 i and COCO-20 i Comparison of 5-shot segmentation performance on the dataset;

[0075] Figure 6 The embodiment of the present invention provides a ResNet-101 backbone based on PASCAL-5 i and COCO-20 i Comparison of 1-shot segmentation performance on the dataset;

[0076] Figure 7 The embodiment of the present invention provides a ResNet-101 backbone based on PASCAL-5 i and COCO-20 i Comparison of 5-shot segmentation performance on the dataset;

[0077] Figure 8 It is a schematic diagram of the structure of the device of the present invention;

[0078] Fig. 9 It is a schematic diagram of the computer device structure of the present invention.

[0079] Among them, 100, data acquisition module; 200, feature extraction module; 300, screening module; 400, optimization module; 500, segmentation module; 40, computer equipment; 41, processor; 42, memory; 43, storage medium. DETAILED DESCRIPTION

[0080] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] Please refer to the attached Figure 1 The embodiment of the present invention provides a prototype-aware few-sample semantic segmentation method, comprising the following steps:

[0082] S1. Collect and preprocess the dataset including training set and test set

[0083] First, we give a general description of the collection and preprocessing of the dataset to ensure that the model can be trained and tested in a standardized and unified data environment, thereby improving the effectiveness of the few-shot semantic segmentation task. The quality of the dataset and the preprocessing process are crucial to the performance of the model, so special attention should be paid to the diversity and consistency of the data during data collection and preprocessing to ensure that feature extraction and model optimization in subsequent steps can proceed smoothly.

[0084] In this embodiment, first, a data set including a training set and a test set is collected to ensure that the data set covers multiple categories commonly used in semantic segmentation tasks. In order to meet the needs of few-sample learning, the data set should contain diverse images of different categories so that the model can still be effectively trained under few-sample conditions.

[0085] Next, the collected data set is preprocessed. The specific preprocessing steps include:

[0086] Resize all images to a uniform size to ensure that the images fed into the neural network have the same dimensions. This is done by cropping, scaling, etc.

[0087] Normalize the image and normalize the pixel values ​​to the range of [0, 1] or [-1, 1], which can improve the stability of the model during feature extraction.

[0088] In order to improve the generalization ability of the model, the images are enhanced by data processing, such as random flipping, rotation, color adjustment, etc. Data enhancement can increase the diversity of training data and thus avoid overfitting problems.

[0089] In the preprocessing process, the data set needs to be divided into a training set and a test set. Specifically, in this embodiment:

[0090] The training set is used to train the model, and the test set is used to evaluate the performance of the model. The two must not overlap to ensure the independence and reliability of the test results.

[0091] Through the above preprocessing, we ensure that the dataset can meet the requirements of the few-sample semantic segmentation task in terms of size, pixel value range and diversity.

[0092] S2. Dataset partitioning and construction of support set and query set

[0093] In order to ensure the stability and representativeness of the training and testing of the few-shot semantic segmentation model under different sample combinations, the dataset needs to be reasonably divided. This process involves dividing the dataset into several parts and further dividing it into support sets and query sets so that the performance of the model can be effectively evaluated under few-shot conditions.

[0094] In this embodiment, first, all images in the data set are divided into four folds of equal size and non-intersecting according to categories. The specific operation steps are as follows:

[0095] Classify the dataset by category, ensuring that the samples in each category are evenly distributed into four folds. Each fold contains the same amount of data, and the images in different folds do not overlap. This division method can be used for cross-validation to ensure that the model performs stably under different data combinations.

[0096] Next, in each fold, the data is further divided into a support set and a query set. The support set and query set are constructed as follows:

[0097] Support set D s and query set D q They are defined as:

[0098]

[0099] D q ={X,Y}

[0100] Among them, X i (X) represents the original image, Y i (Y) represents the mask corresponding to the image, K represents the shot value in the few samples, and represents the number of supporting images.

[0101] Finally, the support set in each fold is used for model training, and the query set is used for model validation. The division of each fold combined with the results of training and validation can realize cross-validation of the model to fully evaluate the generalization ability of the model under few-sample conditions.

[0102] Through this data division method, the model can be trained and tested under various data combinations, which helps to improve the robustness and generalizability of the model and provides a solid data foundation for subsequent feature extraction and semantic segmentation.

[0103] S3. Extract image features using pre-trained deep neural networks

[0104] In order to extract useful features from the support image and the query image, the present invention uses a pre-trained deep neural network model. This process can not only effectively utilize the advantages of the pre-trained model and reduce the training time, but also obtain more stable and recognizable features through feature extraction, which provides a basis for subsequent cycle consistency screening and model optimization.

[0105] In this embodiment, a pre-trained ResNet50 deep neural network model is used to extract features of the support image and the query image. The specific implementation steps are as follows:

[0106] First, the last block of the ResNet50 model is discarded, and the previous convolutional layers are retained. This is because the features of the back layers of ResNet50 are relatively abstract and may not be suitable for fine feature extraction under few-sample conditions, while the features of the front layers are more general and stable.

[0107] Secondly, freeze the parameters of the first two blocks of the ResNet50 model. These parameters will not be updated during model training. The purpose of freezing parameters is to maintain the effective features learned in the pre-trained model and avoid overfitting due to insufficient training data under few-sample conditions.

[0108] Finally, the support image F is trained through the ResNet50 model. s and query image F q Perform feature extraction to obtain the features represented by the three-dimensional tensor, F s ,F q ∈R D×H×W , where D represents the dimension of the feature, H and W represent the height and width of the image respectively.

[0109] Through the above steps, the model can effectively extract key features in the support image and the query image. These features will be used for cycle consistency screening and iterative optimization of foreground and background prototypes in subsequent steps, thereby improving the performance of few-shot semantic segmentation.

[0110] S4. Cycle consistency screening based on support image features and query image features

[0111] In order to establish a more accurate match between the support image and the query image, the present invention introduces a cycle consistency screening mechanism. This process aims to ensure that the selected features have a high degree of consistency between the foreground and background, thereby improving the reliability of the features and laying the foundation for subsequent prototype optimization and feature matching.

[0112] In this embodiment, based on the supporting image features and the query image features, a cycle consistency screening is performed to obtain a credible query feature set. The specific implementation steps are as follows:

[0113] First, the supporting image features F extracted in step S3 are s and query image features F q Flatten it to get the ID sequence C s and C q , where {C s ,C q}∈R D×HW , D represents the feature dimension, and HW represents the total number of pixels after the image is flattened.

[0114] Next, we use the foreground mask y of the support images Calculate the support foreground feature C s′ , and its calculation formula is:

[0115] C ′ =C s *y s

[0116] Here, y s Represents the foreground mask of the support image, which is combined with the support image feature C s Multiply them together to get the support feature C that only contains the foreground area s′ .

[0117] Then, the similarity matrix A1 between the supporting foreground features and the query features is calculated, and its calculation formula is:

[0118]

[0119] in, Indicates C q Transpose and calculate the similarity matrix A1 through matrix multiplication, in which each element represents the similarity between the supporting foreground feature and the query feature.

[0120] In the similarity matrix A1, for each supporting foreground feature i', select the index j of the query feature that best matches it * , and its calculation formula is:

[0121]

[0122] This formula indicates that the best matching query feature index j is determined by selecting the position of the maximum value in each column (corresponding to the supporting foreground feature) in the similarity matrix. * .

[0123] Based on the selected query feature index j * , calculate the new query feature set C q' , and support feature C s With the new query feature set C q' Calculate the similarity matrix A2 again, the formula is:

[0124]

[0125] In the similarity matrix A2, for each query feature j * , select the index i of the supporting feature that best matches it * , and its calculation formula is:

[0126]

[0127] Finally, if condition y is satisfied s(i') = y s (i * ), the query feature is considered to satisfy the cycle consistency, and the pixel is added to the credible query feature set. middle.

[0128] Through the above steps, the present invention can effectively screen out query features with high consistency and representativeness, ensuring that these features have high confidence in the classification of foreground and background, which provides a reliable feature basis for subsequent foreground and background prototype optimization and feature matching.

[0129] S5. Iteratively optimize foreground and background prototypes using a trusted query feature set

[0130] In order to further improve the accuracy of few-sample semantic segmentation, the present invention uses the trusted query feature set obtained in the previous step to iteratively optimize the foreground and background prototypes, and performs feature matching based on the optimized prototypes. The purpose of this step is to make the foreground and background features better reflect the actual situation of the query image through multiple iterations, thereby improving the accuracy of segmentation.

[0131] In this embodiment, the specific implementation steps are as follows:

[0132] First, using the trusted query feature set obtained in step S4 To update the outlook and

[0133] Background prototype. Foreground prototype and background prototype The update formula is:

[0134]

[0135] Among them, α1 and α2 are weight coefficients, usually set to 0.5. and Denote the foreground and background prototypes of the support image respectively. The purpose of this step is to adjust the prototypes by using credible query features to make them closer to the actual feature distribution of the query image.

[0136] Next, based on the updated foreground and background prototypes and Calculate the query image feature F q A foreground affinity graph f Affinity diagram with background A b The calculation formula is as follows:

[0137]

[0138] Among them, cos(…) represents the cosine similarity calculation, A f and A bRepresent the query image features F q Similarity to the updated foreground and background prototypes.

[0139] Afterwards, for affinity graph A f and A b Perform softmax processing to obtain the feature matching result M1. The formula is as follows:

[0140] M1=softmax(A f ,A b )

[0141] The softmax operation can convert the affinity of the foreground and background into a probability value, which is used to indicate the probability of each pixel belonging to the foreground or background.

[0142] Finally, according to the feature matching result M1, the query image features are screened to obtain a high-confidence foreground query feature set and background query feature set The filter criteria are:

[0143]

[0144] Among them, ε1 and ε2 are preset thresholds, usually set to 0.7 and 0.6 respectively, to ensure that only those features with higher confidence are retained.

[0145] Through iterative optimization of the above steps, the prototypes of the foreground and background can gradually converge to the state that best represents the actual situation of the query image, making the feature matching results more accurate. This provides a solid foundation for the final semantic segmentation.

[0146] S6. Semantic segmentation and result calculation

[0147] In the previous steps, the iterative optimization of the foreground and background prototypes has been completed. Next, the optimized model needs to be applied to the test image to perform semantic segmentation and calculate the accuracy of the segmentation results. The purpose of this step is to verify the segmentation performance of the model under few-sample conditions and evaluate the model through accuracy indicators.

[0148] In this embodiment, the specific implementation steps are as follows:

[0149] First, the optimized model is applied to the test images. For each test image, its feature F is extracted q , and feed it into the optimized foreground and background prototypes and , calculate its foreground affinity graph A f Affinity diagram with background A b By computing the affinity graph, we can quantify the likelihood of each pixel belonging to the foreground or background.

[0150] Next, for the foreground affinity graph A f Affinity diagram with background A b Softmax processing is performed to obtain the segmented probability map M1. The value of each pixel in M1 represents its classification probability, and the highest value determines the final classification of the pixel.

[0151] Finally, the final segmentation mask is generated by thresholding M1. Specifically, a suitable threshold is selected to classify the part above the threshold as the foreground and the part below the threshold as the background, thereby generating a clear segmentation mask.

[0152] Furthermore, based on the generated segmentation mask and the real mask, the accuracy of the segmentation result is calculated. The segmentation accuracy is calculated by mIoU (mean Intersection over Union), and the specific formula is:

[0153]

[0154] Among them, k represents the total number of categories, TP is the number of correctly predicted positive samples, FP is the number of incorrectly predicted positive samples, and FN is the number of incorrectly predicted negative samples. By calculating mIoU, the average segmentation performance of the model on all categories can be quantified, thereby comprehensively evaluating the performance of the model.

[0155] Through these steps, the optimized model can be applied to actual image segmentation tasks, and the accuracy of the segmentation results can be evaluated through indicators such as mIoU. This process not only verifies the effectiveness of the model, but also provides data support for subsequent optimization and improvement.

[0156] In general, the present invention significantly improves the accuracy and stability of few-sample semantic segmentation through a series of optimization steps, including data preprocessing, feature extraction, cycle consistency screening and prototype optimization.

[0157] Example:

[0158] An embodiment of the present invention provides a new method for few-sample semantic segmentation based on prototype perception. The embodiment of the present invention is described in detail below in conjunction with the accompanying drawings to better understand the technical solution of the present invention and its advantages.

[0159] Figure 2 The flowchart of a prototype-aware few-sample semantic segmentation method provided by an embodiment of the present invention is shown. The method includes the following main steps:

[0160] S1. Collect and download the dataset required for the experiment, which includes a training set and an independent test set.

[0161] S2. Preprocess the downloaded dataset.

[0162] S3. Extract feature maps of training data using deep neural networks.

[0163] S4. Perform cycle consistency query on the extracted features to obtain a credible query feature set.

[0164] S5. Iteratively optimize the feature matching results by using the trusted query feature set and the support and query feature dense matching results.

[0165] S6. Train the model parameters, then freeze the parameters and input the test image to obtain the segmentation result.

[0166] Figure 3 Schematic diagram of cycle consistency screening in an embodiment of the present invention. In the figure, the first row shows the performance of the model of the present invention in screening the correct features, and the second row shows examples of other models failing to screen in the same scenario. It can be seen that the cycle consistency screening mechanism can effectively identify the correct features and has strong robust error correction capabilities. The model of the present invention can more accurately identify foreground and background features and improve the accuracy of segmentation through cycle consistency screening.

[0167] Figure 4 The 1-shot segmentation performance comparison chart based on the ResNet-50 backbone network on the PASCAL-5i and COCO-20i datasets is shown. In this comparison, the segmentation performance of each fold is represented by P 0-3 and C 0-3, which represent the fold0-fold3 indicators of the PASCAL-5i and COCO-20i datasets, respectively, and M represents the average performance of all folds. It can be seen that the SSPA model of the present invention is significantly higher than the other three mainstream models in terms of the average segmentation accuracy on most folds and all datasets, verifying the effectiveness and superiority of the method of the present invention.

[0168] Figure 5 The comparison of 5-shot segmentation performance based on the ResNet-50 backbone network on the PASCAL-5i and COCO-20i datasets is shown. Compared with 1-shot, the support set contains more image information in the 5-shot setting, so the segmentation accuracy is significantly improved. Figure 4 It can be seen that the SSPA model of the present invention is significantly higher than the other three mainstream models in the average segmentation accuracy of most folds and all data sets under the 5-shot setting, which further proves the excellent performance of the model of the present invention in processing rich supporting information.

[0169] Figure 6The comparison chart of 1-shot segmentation performance based on the ResNet-101 backbone network on the PASCAL-5i and COCO-20i datasets is shown. Under the more powerful ResNet-101 backbone network, the features extracted by the model have stronger representation capabilities, so better segmentation results can be achieved. Figure 5 It can be seen that the model of the present invention based on ResNet-101 is significantly higher than the other three mainstream models in the average segmentation accuracy of most folds and all data sets, showing the powerful segmentation ability of the present invention under the high-performance backbone network.

[0170] Figure 7 The 5-shot segmentation performance comparison chart based on the ResNet-101 backbone network on the PASCAL-5i and COCO-20i datasets is shown. In this context, the proposed model combines the more powerful ResNet-101 backbone network with richer 5-shot support information to achieve impressive segmentation results. Figure 6 It can be seen that the SSPA model of the present invention is significantly higher than the other three mainstream models in the average segmentation accuracy of most folds and all data sets, which further verifies the superior performance of the method of the present invention under high complexity conditions.

[0171] In summary, the prototype-aware few-shot semantic segmentation method proposed in this paper shows excellent segmentation performance in various scenarios. It not only performs well in different folds of various datasets, but also achieves excellent results under different shot settings and backbone network configurations.

[0172] The prototype-aware few-sample semantic segmentation device described below and the prototype-aware few-sample semantic segmentation method described above can refer to each other.

[0173] Please see attached Figure 8 The present invention also provides a prototype-aware few-sample semantic segmentation device, comprising:

[0174] The data collection module 100 is used to collect and preprocess the data set including the training set and the test set;

[0175] A feature extraction module 200, for extracting features of a support image and a query image using a pre-trained deep neural network;

[0176] A screening module 300, for performing cycle consistency screening based on supporting image features and query image features;

[0177] An optimization module 400, for iteratively optimizing foreground and background prototypes using a credible query feature set, and performing feature matching;

[0178] The segmentation module 500 is used to input a test image and perform semantic segmentation using the optimized model.

[0179] The device of this embodiment can be used to execute the above method embodiment, and its principles and technical effects are similar, which will not be repeated here.

[0180] Please refer to the attached Fig. 9 The present invention further provides a computer device 40, comprising: a processor 41 and a memory 42, wherein the memory 42 stores a computer program executable by the processor, and when the computer program is executed by the processor, the above method is executed.

[0181] The present invention further provides a storage medium 43 on which a computer program is stored. When the computer program is run by the processor 41, the above method is executed.

[0182] Among them, the storage medium 43 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0183] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A prototype-aware few-sample semantic segmentation method, characterized in that: The following steps are involved: Collect and preprocess the dataset including training and test sets; By dividing all images in the dataset into four folds of equal size and non-intersecting according to categories, the data of each fold is further divided into a support set and a query set; Extracting features of the support image and the query image using a pre-trained deep neural network, wherein the last block of the network is discarded and the parameters of the first two blocks are frozen; Based on the supporting image features and query image features, cycle consistency screening is performed to obtain a credible query feature set; Iteratively optimizing foreground and background prototypes using the credible query feature set, and performing feature matching based on the optimized prototypes; Input the test image and perform semantic segmentation through the optimized model to calculate the segmentation result; The step of iteratively optimizing the foreground and background prototypes using the credible query feature set and performing feature matching based on the optimized prototypes includes: Using the obtained trusted query feature set Calculate updated foreground and background prototypes and The formula is: Among them, α1 and α2 are hyperparameters. They are the foreground and background prototypes obtained after one iteration respectively; Based on the updated foreground and background prototypes and Calculate the query image feature F q A foreground affinity graph f Affinity diagram with background A b , the formula is: Among them, cos(…) means calculating cosine similarity; Affinity Diagram A f and A b Perform softmax processing to obtain the feature matching result M1, the formula is: M1=softmax(A f ,A b ) Based on the feature matching result M1, filter out the high-confidence foreground query feature set and background query feature set The formula is: Among them, ε1 and ε2 are screening thresholds.

2. The prototype-aware few-sample semantic segmentation method according to claim 1, characterized in that: The step of dividing all images in the data set into four folds of equal size and non-intersecting according to categories, and further dividing the data of each fold into a support set and a query set comprises: Divide all images in the dataset into four folds of equal size and non-intersecting according to their categories; The data in each fold is further divided into a support set and a query set. Query set D q ={X,Y}, where X i (X) represents the original image, Y i (Y) represents the mask corresponding to the image, K represents the shot value in the few samples, and represents the number of supporting images.

3. The prototype-aware few-sample semantic segmentation method according to claim 1, characterized in that: The step of extracting features of the support image and the query image using a pre-trained deep neural network comprises: Feature extraction of support images and query images by using pre-trained deep neural networks; During feature extraction, the last block of the deep neural network is discarded and the parameters of the first two blocks are frozen so that no parameter updates are performed during training. Specifically, the supported image F s The feature extraction results of the query image Fw are respectively expressed as three-dimensional tensors: {F s ,F q }∈R D×H×W Where D represents the dimension of the feature, H and W represent the height and width of the image respectively.

4. The prototype-aware few-sample semantic segmentation method according to claim 3, characterized in that: The pre-trained deep neural network is ResNet50.

5. The prototype-aware few-sample semantic segmentation method according to claim 1, characterized in that: The step of performing cycle consistency screening based on the supporting image features and the query image features to obtain a credible query feature set includes: The extracted support image features F s and query image features F q Flatten into ID sequence C s and C q : {C s ,C q }∈R D×HW Where D represents the feature dimension, HW represents the total number of pixels after the image is flattened; Based on the foreground mask y of the support image s Calculate the support foreground feature C s′ , the formula is: C s′ =C s *y s Calculate the similarity matrix A1 between the supporting foreground features and the query features, the formula is: in, Indicates C q Perform transposition; In the similarity matrix A1, for each supporting foreground feature i', select the index j of the query feature that best matches it * , the formula is: Based on the selected query feature index j * , calculate the new low confidence query feature set C q' , the formula is: Calculate the support feature C again s With the new query feature set C q' The similarity matrix A2 between them is as follows: In the similarity matrix A2, for each query feature j * , select the index i of the supporting feature that best matches it * , the formula is: If the condition y is met s (i') = y s (i * ), the query feature is considered to satisfy the cycle consistency, and the pixel is added to the credible query feature set. middle.

6. The prototype-aware few-sample semantic segmentation method according to claim 1, characterized in that: The step of inputting a test image and performing semantic segmentation on the optimized model and calculating the segmentation result comprises: The test image is fed into the optimized semantic segmentation model, which includes the iteratively optimized foreground and background prototypes. and Use the optimized model to input the test image feature F q Perform segmentation prediction and calculate the foreground affinity map A of the test image f Affinity diagram with background A b ; Affinity Diagram A f and A b Softmax processing is performed to obtain the segmented probability map M1, and the final segmentation mask is generated by thresholding M1.

7. A prototype-aware few-sample semantic segmentation device, used to implement the prototype-aware few-sample semantic segmentation method according to any one of claims 1 to 6, characterized in that: include: Data collection module, used to collect and preprocess data sets including training sets and test sets; A feature extraction module for extracting features of support images and query images using a pre-trained deep neural network; A screening module, for performing cycle consistency screening based on supporting image features and query image features; An optimization module, used to iteratively optimize foreground and background prototypes using a trusted query feature set and perform feature matching; The segmentation module is used to input the test image and perform semantic segmentation through the optimized model.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images