A small sample semantic segmentation method, system, electronic device and storage medium without labeling auxiliary guidance

Abstract

The present invention relates to the field of semantic segmentation in computer vision, and in particular to a small sample semantic segmentation method with unlabeled auxiliary guidance, the method comprising: firstly, extracting the features of the unlabeled image while extracting the features of the support image and the query image using a feature extraction network; then, extracting the auxiliary guidance information in the unlabeled image using the pixel-level similarity between the unlabeled features and the support features, and optimizing the extraction of the auxiliary guidance information using a loss function based on contrastive learning; then, using an attention mechanism to aggregate the auxiliary guidance information and the support information, and finally using a convolutional network to obtain the final segmentation result. The present invention utilizes the easy accessibility and low cost of unlabeled images, makes up for the problem of insufficient support image information, greatly alleviates the problem of limited segmentation accuracy caused by intra-class differences, improves the semantic segmentation accuracy of small sample targets, and can be implanted as a plug-in into a variety of small sample semantic segmentation methods, and is very practical.

Description

Technical Field

[0001] The present invention relates to the field of semantic segmentation in computer vision, and in particular to a small sample semantic segmentation method, system, electronic device and storage medium without labeling auxiliary guidance. Background Art

[0002] The introduction and vigorous development of deep learning has greatly improved the performance of semantic segmentation. The semantic segmentation model based on convolutional neural network requires a large amount of data with pixel-level annotations. However, it is time-consuming and laborious to obtain a large amount of high-quality pixel-level annotation data, which greatly hinders the application of semantic segmentation in many fields. In order to reduce the dependence of semantic segmentation models on a large amount of annotated data, small sample semantic segmentation uses a small number of annotated samples to complete the semantic segmentation task of unknown categories.

[0003] The current small sample semantic segmentation method obtains information about the target category from the support set image to guide the segmentation of the query image. However, according to the small sample semantic segmentation training paradigm, the support set of each segmentation task only contains 1 or 5 support images, and the guidance information that can be provided is limited; at the same time, the existence of the intra-class difference problem also causes the support set to be unable to provide sufficient guidance information, thereby limiting the accuracy improvement of the small sample semantic segmentation image. In the prior art, a Chinese patent with publication number CN117726809 A discloses a small sample semantic segmentation method based on information interaction enhancement, but this method only supports information interaction between the image and the query image semantic information, and the information involved in the guidance still needs to be expanded. Therefore, there is an urgent need for a small sample semantic segmentation method that can mine more guidance information. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a small sample semantic segmentation method, system, electronic device and storage medium without labeling auxiliary guidance.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] The present invention provides a small sample semantic segmentation method without label-assisted guidance, comprising the following steps:

[0007] Step 1, extracting the middle-level features and high-level features of the support image, the query image and the unlabeled image through the feature extraction network, and obtaining the middle-level support features, the middle-level query features, the middle-level unlabeled features and the high-level support features, the high-level query features and the high-level unlabeled features;

[0008] Step 2: performing a mask average pooling operation on the middle-level support features and the corresponding support masks to obtain the original support prototype;

[0009] Step 3, generating auxiliary guidance information, including: calculating the pixel-by-pixel similarity between the high-level unlabeled features and the high-level supporting features, taking the maximum value to obtain a similarity map, setting a threshold, if the position is greater than the threshold, the position is 1, and if it is lower than the threshold, the position is 0, selecting the similarity map by the threshold to obtain an unlabeled mask, performing a mask average pooling operation on the middle-level unlabeled features and the corresponding unlabeled masks, and obtaining an unlabeled prototype;

[0010] Step 4, auxiliary information aggregation, including: using the original support prototype obtained in step 2 as an attention query to select all the unlabeled prototypes obtained in step 3, using the attention mechanism to add the unlabeled information to the support information to obtain the aggregated prototype;

[0011] Step 5, calculating the pixel-level similarity between the high-level query feature and the high-level support feature and taking the maximum value to obtain a support-query similarity graph;

[0012] In step 6, the aggregate prototype obtained in step 4, the support-query similarity graph obtained in step 5, and the middle-level query features obtained in step 1 are expanded to the same size and connected in series, and then segmented through a convolutional network to obtain the final segmentation result of the target object.

[0013] The feature extraction network includes ResNet-50.

[0014] The middle-level support features and the high-level support features are multiplied by the downsampled support mask to filter out non-target category information.

[0015] The value range of the similarity graph is between 0 and 1, and the value of the threshold is selected as 0.9.

[0016] A loss function based on contrastive learning is used to optimize the extraction of auxiliary guidance information.

[0017] Furthermore, the loss function based on contrastive learning is expressed by the following formula:

[0018]

[0019] Among them, N un is the number of unlabeled images, p s represents the original supporting prototype in step 2, represents the i-th unlabeled prototype, Represents the unlabeled background prototype obtained by mask average pooling of the complement of the j-th unlabeled mask and the corresponding unlabeled features.

[0020] The structure of the convolutional network is obtained by connecting multiple layers of convolutional networks with convolution kernels of 3x3 and 1x1 in series.

[0021] In a second aspect, the present invention provides a small sample semantic segmentation system without label-assisted guidance, comprising:

[0022] A feature extraction module is used to extract middle-level features and high-level features of the support image, the query image and the unlabeled image through a feature extraction network, and obtain middle-level support features, middle-level query features, middle-level unlabeled features and high-level support features, high-level query features and high-level unlabeled features;

[0023] A support prototype acquisition module, used for performing a mask average pooling operation on the middle-level support features and the corresponding support masks to obtain the original support prototype;

[0024] The auxiliary information generation module is used to generate auxiliary guidance information, including: calculating the pixel-by-pixel similarity between the high-level unlabeled features and the high-level support features, taking the maximum value to obtain a similarity map, setting a threshold, if the position is greater than the threshold, the position is 1, and if it is lower than the threshold, the position is 0, selecting the similarity map by the threshold to obtain an unlabeled mask, performing a mask average pooling operation on the middle-level unlabeled features and the corresponding unlabeled masks, and obtaining an unlabeled prototype;

[0025] An auxiliary information aggregation module, used for auxiliary information aggregation, including: using the original support prototype as an attention query to select all unlabeled prototypes, using an attention mechanism to add the unlabeled information to the support information, and obtaining an aggregated prototype;

[0026] A support-query similarity graph calculation module, used to calculate the pixel-level similarity between the high-level query feature and the high-level support feature and take the maximum value to obtain a support-query similarity graph;

[0027] The segmentation module is used to expand the aggregate prototype, the support-query similarity graph and the middle-level query feature to the same size and connect them in series, and then segment them through a convolutional network to obtain a final segmentation result about the target object.

[0028] In a third aspect, the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory, wherein the processor implements any of the above-described methods when executing the program.

[0029] In a fourth aspect, the present invention provides a storage medium having a program stored thereon, wherein the program implements any of the above-described methods when executed.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] 1. The small sample semantic segmentation algorithm with no annotation auxiliary guidance proposed in the present invention extracts the information of the target object from a small number of easy-to-obtain, low-cost images without pixel-level annotations to expand the limited guidance information contained in the original support set images, and uses a specially designed loss function to ensure the effectiveness of the auxiliary guidance information. The present invention can simply and efficiently mine more target category information to overcome the intra-class difference problem, greatly improving the segmentation accuracy of the target object. At the same time, the present invention can be used as a flexible plug-in to implant a variety of small sample segmentation methods, which is very practical.

[0032] 2. The present invention utilizes the similarity between images with objects of the same category and extracts the available guidance information in the unlabeled images by calculating the pixel-level similarity, further verifying the importance of similarity measurement and making efforts to get rid of the heavy pixel-level labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart for realizing the method of the present invention;

[0034] Figure 2 Schematic diagram of segmentation results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0036] The hardware environment used for implementation is: Intel Core i7 processor, NVIDIA GeForce RTX4090 graphics card, and the programming language is python. The pytorch framework is used to implement the method proposed in the invention. The training and test images used in the experiment are from PASCAL VOC 2012: http: / / host.robots.ox.ac.uk / pascal / VOC / voc2012 / . The training set contains 20 categories, of which 15 categories are used for training and the other 5 categories are used for testing.

[0037] Reference Figure 1 The present embodiment provides a method for semantic segmentation of small samples without labeling auxiliary guidance, which is specifically implemented as follows:

[0038] Step 1: First, use the pre-trained feature extraction network to extract the support image I S , query image I Q and unlabeled image I un Extract the middle-level support features f s , middle-level query feature f q , middle-level unlabeled features Fun and high-level support features High-level query features High-level unlabeled features The middle-level support feature f s and high-level support features They are multiplied by the support mask to filter out non-target category information.

[0039] Step 2: Support features f at the middle level s and the corresponding support mask M s Perform a masked average pooling operation to obtain the original support prototype p s .

[0040] Step 3: High-level unlabeled features With high-level support features Calculate the pixel-by-pixel similarity and take the maximum value to obtain a similarity map. The value range of the similarity map is between 0 and 1. In this embodiment, the threshold is set to 0.9. If it is greater than the threshold, the position is 1, and if it is lower than the threshold, it is 0. In this way, the unlabeled mask M is obtained. un .

[0041] Step 4: For the unlabeled features F in the middle layer un and the corresponding unlabeled mask M un Perform masked average pooling to obtain the unlabeled prototype P un , i.e. auxiliary guidance information;

[0042] Preferably, this embodiment uses a loss function based on contrastive learning to optimize the extraction of auxiliary guidance information. The expression of the loss function is as follows:

[0043]

[0044] Among them, N un is the number of unlabeled images, p s represents the original supporting prototype in step 2, represents the i-th unlabeled prototype, Represents the unlabeled background prototype obtained by mask average pooling of the complement of the j-th unlabeled mask and the corresponding unlabeled features.

[0045] Step 5: The original support prototype p obtained in step 2 s As attention query, select all unlabeled prototypes P obtained in step 4 un , the attention mechanism is used to add the unlabeled information to the supporting information to obtain the aggregate prototype

[0046] Step 6: Calculate high-level query features and high-level support features The pixel-level similarity of the query is calculated and the maximum value is taken to obtain the support-query similarity graph m. p .

[0047] Step 7: The aggregate prototype obtained in step 5 and the support-query similarity graph m obtained in step 6 p And the middle-level query feature f obtained in step 1 q Expand to the same size and connect them in series, that is, after multi-source feature fusion, segmentation is performed through the convolutional network to obtain the final segmentation result Y of the target object. Q The segmentation results are as follows: Figure 2 As shown in the figure, the structure of the convolutional network can be obtained by connecting multiple layers of convolutional networks with convolutional kernels of 3x3 and 1x1 in series.

[0048] The present invention can be applied as a plug-in in existing small sample semantic segmentation methods to improve the segmentation accuracy achieved by existing methods. The present invention achieves 67.7% mIoU (intersection over union ratio of true value and predicted value) on the PASCAL VOC2012 dataset with ResNet-50 as the feature extraction network under the setting of 1 supporting sample, verifying the effectiveness of the method.

[0049] In the second aspect, this embodiment provides a small sample semantic segmentation system without label-assisted guidance. Based on the system provided by this embodiment, the method mentioned in the above embodiment can be implemented. The system includes: a feature extraction module, a support prototype acquisition module, an auxiliary information generation module, an auxiliary information aggregation module, a support-query similarity graph calculation module and a segmentation module.

[0050] Specifically, the steps implemented by this module are as follows:

[0051] A feature extraction module, used to implement step 1 of this embodiment;

[0052] Supporting a prototype acquisition module, used to implement step 2 of this embodiment;

[0053] An auxiliary information generation module, used to implement steps 3 and 4 of this embodiment;

[0054] An auxiliary information aggregation module, used to implement step 5 of this embodiment;

[0055] Support - query similarity graph calculation module, used to implement step 6 of this embodiment;

[0056] The segmentation module implements step 7 of this embodiment.

[0057] In a third aspect, this embodiment provides an electronic device, including a memory, a processor, and a program stored in the memory, wherein the processor implements the method mentioned in the above embodiment when executing the program.

[0058] In a fourth aspect, this embodiment provides a storage medium on which a program is stored, and when the program is executed, the method mentioned in the above embodiment is implemented. The storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0059] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A small sample semantic segmentation method without label-assisted guidance, characterized in that: The following steps are involved: Step 1, extracting the middle-level features and high-level features of the support image, the query image and the unlabeled image through the feature extraction network, and obtaining the middle-level support features, the middle-level query features, the middle-level unlabeled features and the high-level support features, the high-level query features and the high-level unlabeled features; Step 2: performing a mask average pooling operation on the middle-level support features and the corresponding support masks to obtain the original support prototype; Step 3, generating auxiliary guidance information, including: calculating the pixel-by-pixel similarity between the high-level unlabeled features and the high-level supporting features, taking the maximum value to obtain a similarity map, setting a threshold, if the position is greater than the threshold, the position is 1, and if it is lower than the threshold, the position is 0, selecting the similarity map by the threshold to obtain an unlabeled mask, performing a mask average pooling operation on the middle-level unlabeled features and the corresponding unlabeled masks, and obtaining an unlabeled prototype; Step 4, auxiliary information aggregation, including: using the original support prototype obtained in step 2 as an attention query to select all the unlabeled prototypes obtained in step 3, using the attention mechanism to add the unlabeled information to the support information to obtain the aggregated prototype; Step 5, calculating the pixel-level similarity between the high-level query feature and the high-level support feature and taking the maximum value to obtain a support-query similarity graph; In step 6, the aggregate prototype obtained in step 4, the support-query similarity graph obtained in step 5, and the middle-level query features obtained in step 1 are expanded to the same size and connected in series, and then segmented through a convolutional network to obtain the final segmentation result of the target object.

2. The small sample semantic segmentation method without labeling auxiliary guidance according to claim 1, characterized in that: The feature extraction network includes ResNet-50.

3. The small sample semantic segmentation method without labeling auxiliary guidance according to claim 1, characterized in that: The middle-level support features and the high-level support features are multiplied by the downsampled support mask to filter out non-target category information.

4. The small sample semantic segmentation method without labeling auxiliary guidance according to claim 1, characterized in that: The value range of the similarity graph is between 0 and 1, and the value of the threshold is selected as 0.

9.

5. The small sample semantic segmentation method without labeling auxiliary guidance according to claim 1, characterized in that: A loss function based on contrastive learning is used to optimize the extraction of auxiliary guidance information.

6. The small sample semantic segmentation method without labeling assistance guidance according to claim 5, characterized in that: The loss function based on contrastive learning is expressed by the following formula: Among them, N un is the number of unlabeled images, p s Indicates the original support prototype, represents the i-th unlabeled prototype, Represents the unlabeled background prototype obtained by mask average pooling of the complement of the j-th unlabeled mask and the corresponding unlabeled features.

7. The small sample semantic segmentation method without labeling auxiliary guidance according to claim 1, characterized in that: The structure of the convolutional network is obtained by connecting multiple layers of convolutional networks with convolution kernels of 3x3 and 1x1 in series.

8. A small sample semantic segmentation system without label-assisted guidance, characterized in that: include: A feature extraction module is used to extract middle-level features and high-level features of the support image, the query image and the unlabeled image through a feature extraction network, and obtain middle-level support features, middle-level query features, middle-level unlabeled features and high-level support features, high-level query features and high-level unlabeled features; A support prototype acquisition module, used for performing a mask average pooling operation on the middle-level support features and the corresponding support masks to obtain the original support prototype; The auxiliary information generation module is used to generate auxiliary guidance information, including: calculating the pixel-by-pixel similarity between the high-level unlabeled features and the high-level support features, taking the maximum value to obtain a similarity map, setting a threshold, if the position is greater than the threshold, the position is 1, and if it is lower than the threshold, the position is 0, selecting the similarity map by the threshold to obtain an unlabeled mask, performing a mask average pooling operation on the middle-level unlabeled features and the corresponding unlabeled masks, and obtaining an unlabeled prototype; An auxiliary information aggregation module, used for auxiliary information aggregation, including: using the original support prototype as an attention query to select all unlabeled prototypes, using an attention mechanism to add the unlabeled information to the support information, and obtaining an aggregated prototype; A support-query similarity graph calculation module, used to calculate the pixel-level similarity between the high-level query feature and the high-level support feature and take the maximum value to obtain a support-query similarity graph; The segmentation module is used to expand the aggregate prototype, the support-query similarity graph and the middle-level query feature to the same size and connect them in series, and then segment them through a convolutional network to obtain a final segmentation result about the target object.

9. An electronic device comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 7 is implemented.

Images