A semantic segmentation method for local security with few labeled samples

Abstract

The present invention discloses a semantic segmentation method under the condition of few labeled samples for on-site security. The method comprises: obtaining a first aerial image and a second aerial image, wherein the first aerial image is an image with a category label at the pixel level for a part of objects of the same category in a plurality of aerial images, and the second aerial image is an unlabeled image corresponding to the first aerial image; inputting the first aerial image and the second aerial image into a feature extractor to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image; based on the category label, performing feature extraction on the first feature map and the second feature map to obtain a plurality of pixel-level sample features of the first aerial image, the true label of each pixel-level sample feature of the first aerial image, and a plurality of unlabeled pixel-level sample features of the second aerial image. The present invention solves the technical problems in the prior art that the semantic segmentation model requires a large amount of labeled data, does not constrain intra-class differences, leads to semantic aliasing, and has low multi-classification segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a semantic segmentation method under conditions of few labeled samples for on-site security. Background Art

[0002] In the low-altitude security mission of on-site security, it is of great significance for drones to obtain low-altitude RGB data and use image segmentation technology to perform semantic analysis on the data to help make airspace security and defense decisions. Image semantic segmentation technology is a dense pixel-level semantic classification of scene images. The semantic segmentation method based on deep learning relies on a large number of densely annotated images, but due to reasons such as time cost and difficulty in sample collection, it is difficult to obtain sufficient annotated samples in practical applications. The semantic segmentation with a small number of annotated samples involved in the present invention aims to achieve pixel-level classification of images by relying on a small number of annotated images. This technology has important applications in scenarios such as urban planning, agricultural management, and military security.

[0003] Existing semantic segmentation techniques for a small number of annotated images have made great progress through the combination of meta-learning and metric learning. However, they usually require a large base dataset and only focus on distinguishing between a single category and the background in the image, which to some extent limits the application of this technology in practical scenarios. Dong N. et al. proposed in the paper “Dong N, Xing EP. Few-shot semantic segmentation with prototype learning[C] / / BMVC.2018,3(4):4.” to use a prototype learner to extract prototypes representing semantic categories from annotated images for pixel-level feature measurement. Wang K. et al. proposed in the paper “Wang K, Liew JH, Zou Y, et al. Panet: Few-shot image semantic segmentation with prototype alignment[C] / / proceedings of the IEEE / CVFinternational conference on computer vision.2019:9197-9206.” to use query images to calculate prototype alignment to improve the quality of prototype generation. Fan Q. et al. proposed in the document “Fan Q, Pei W, Tai YW, et al. Self-support few-shot semantic segmentation [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 701-719.” to generate a self-support prototype from the query image to guide self-semantic segmentation to alleviate intra-class feature differences.

[0004] While these methods have made certain contributions, they all have limitations. First, they require a large amount of base training data, which many application scenarios cannot provide. Second, these methods only distinguish between a single category and the background, which greatly limits their practicality. Summary of the Invention

[0005] An embodiment of the present invention provides a semantic segmentation method under conditions of few labeled samples for on-site security, so as to at least solve the technical problems in the prior art that the semantic segmentation model requires a large amount of labeled data, has no constraints on intra-class differences, leads to semantic aliasing, and has low multi-classification segmentation accuracy.

[0006] According to one aspect of an embodiment of the present invention, a semantic segmentation method under the condition of few labeled samples for on-site security is provided. The method may include: obtaining a first aerial image and a second aerial image, wherein the first aerial image is an image with a category label at the pixel level for a portion of objects of the same category in multiple aerial images, and the second aerial image is an unlabeled image corresponding to the first aerial image; inputting the first aerial image and the second aerial image into a feature extractor to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image; based on the category label, performing feature extraction on the first feature map and the second feature map to obtain multiple pixel-level sample features of the first aerial image, the true label of each pixel-level sample feature of the first aerial image, and multiple unlabeled pixel-level sample features of the second aerial image; based on the multiple pixel-level sample features of the first aerial image, obtaining a prototype of each category in the multiple pixel-level sample features of the first aerial image; based on the prototype of each category, obtaining a prototype classifier; based on the prototype classifier, each pixel-level sample feature of the first aerial image, and each pixel-level sample feature The real label is used to obtain the classification constraint loss value; the unsupervised constraint loss value is obtained based on the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image and the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image, wherein the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image are determined based on the prototype classifier and the multiple unlabeled pixel-level sample features of the second aerial image; the prototype classifier and each pixel-level sample feature of the first aerial image are input into the supervised prototype contrast learning module to obtain the supervised contrast learning loss value; the target loss value is obtained based on the classification constraint loss value, the unsupervised constraint loss value and the supervised contrast learning loss value; the prototype of each category in the multiple pixel-level sample features of the first aerial image is updated using the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image; the aerial image to be detected is obtained and the semantic segmentation result of the aerial image to be detected is determined.

[0007] Optionally, based on multiple pixel-level sample features of the first aerial image, a prototype of each category in the multiple pixel-level sample features of the first aerial image is obtained, including: performing mean calculation on the multiple pixel-level sample features of the first aerial image to obtain the prototype of each category in the multiple pixel-level sample features of the first aerial image.

[0008] Optionally, the mean of the multiple pixel-level sample features of the first aerial image is calculated to obtain the prototype expression of each category in the multiple pixel-level sample features of the first aerial image:

[0009]

[0010] in, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, C i represents the number of categories of the first aerial image, x i Represents one of the multiple pixel-level sample features of the first aerial image, y i represents the label of the i-th category in each pixel-level sample feature of the first aerial image, and Num represents the number of samples of the i-th category in the first aerial image.

[0011] Optionally, a classification constraint loss value is obtained based on the prototype classifier, each pixel-level sample feature of the first aerial image, and the true label of each pixel-level sample feature, including: performing similarity measurement prediction based on the prototype classifier and each pixel-level sample feature of the first aerial image to obtain the predicted probability of each pixel-level sample feature of the first aerial image; performing cross-entropy calculation on the predicted probability of each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature to obtain the classification constraint loss value.

[0012] Optionally, an unsupervised constrained loss value is obtained based on the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image, and pseudo-labels of multiple unlabeled pixel-level sample features of the second aerial image, including: performing similarity measurement prediction based on the prototype classifier and multiple unlabeled pixel-level sample features of the second aerial image to obtain the predicted probability of each unlabeled pixel-level sample feature of the second aerial image; performing cross-entropy calculation on the predicted probability of each unlabeled pixel-level sample feature of the second aerial image and the pseudo-label of each unlabeled pixel-level sample feature to obtain the unsupervised constrained loss value.

[0013] Optionally, the prototype classifier and each pixel-level sample feature of the first aerial image are input into a supervised prototype contrastive learning module, and the expression of the supervised contrastive learning loss value is obtained as follows:

[0014]

[0015] Among them, C i represents the i-th category number of the first aerial image, Indicates that in C i The supervised contrastive learning loss value under Indicates that the first aerial image is not of category C i The number of samples, x k Indicates that the first aerial image contains all the images except category C i Other pixel-level sample features, x i Represents each pixel-level sample feature of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, N foreIndicates the number of samples of all categories in the first aerial image.

[0016] Optionally, based on the classification constraint loss value, the unsupervised constraint loss value, and the supervised contrastive learning loss value, the expression for the target loss value is:

[0017] L total =L cls +α·L SPCL +β·L us (3)

[0018] Among them, L total is the target loss value, L cls is the classification constraint loss value, L SPCL is the supervised contrastive learning loss value, L us is the unsupervised constraint loss value, α is the first weight parameter, and β is the second weight parameter.

[0019] Optionally, updating the prototype of each category in the multiple pixel-level sample features of the first aerial image using the pseudo labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image includes:

[0020]

[0021] in, is the update item of the prototype of the i-th category in the multiple pixel-level sample features of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, x s represents a single pixel-level sample feature in the unlabeled pixel-level sample features of the remaining images except the image of the i-th category in the first aerial image, x j represents the unlabeled pixel-level sample features in the remaining images of the first aerial image except the image of the i-th category, represents the pseudo label of a single pixel-level sample feature in the unlabeled pixel-level sample features in the remaining images except the image of the i-th category in the first aerial image, represents the pseudo labels of the unlabeled pixel-level sample features in the remaining images of the first aerial image except the image of the i-th category, is the prototype of the i-th category in the multiple pixel-level sample features of the updated first aerial image, and ε is the update item weight.

[0022] Beneficial effects of the present invention:

[0023] The present invention proposes a semantic segmentation method under the condition of few labeled samples for on-site security, which processes the first aerial image and the second aerial image to obtain the true label of each pixel-level sample feature of the first aerial image and multiple unlabeled pixel-level sample features of the second aerial image, obtain the prototype of each category in the multiple pixel-level sample features of the first aerial image according to the multiple pixel-level sample features of the first aerial image, splice the prototypes of each category to obtain a prototype classifier, obtain the prototype classifier according to the prototype classifier, each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature, obtain the prototype classifier according to the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image and pseudo labels of multiple unlabeled pixel-level sample features of the second aerial image, obtain the unsupervised constraint loss value, and input the prototype classifier and each pixel-level sample feature of the first aerial image into the supervised prototype classifier. The contrastive learning module obtains a supervised contrastive learning loss value, obtains a target loss value according to the classification constraint loss value, the unsupervised constraint loss value and the supervised contrastive learning loss value, and updates the prototype of each category in the multiple pixel-level sample features of the first aerial image using the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image. The above steps obtain a semantic segmentation model of the aerial image, input the aerial image to be detected into the semantic segmentation model, and obtain the semantic segmentation result of the aerial image to be detected. This solves the technical problems in the prior art that the semantic segmentation model requires a large amount of labeled data, does not constrain intra-class differences, leads to semantic aliasing, and has low multi-classification segmentation accuracy. It achieves the technical effect of not requiring a large amount of labeled data to train the semantic segmentation model, constraining intra-class differences, reducing semantic aliasing, and improving multi-classification segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 This is a flowchart of a semantic segmentation method for local security under conditions of few labeled samples according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or precedence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products or devices.

[0028] Example 1

[0029] According to an embodiment of the present invention, a semantic segmentation method under conditions of few labeled samples for on-site security is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system containing at least one set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0030] Figure 1 is a flowchart of a semantic segmentation method for local security under the condition of few labeled samples according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0031] Step S101: Acquire a first aerial image and a second aerial image, wherein the first aerial image is an image in which a portion of objects of the same category in multiple aerial images are annotated with category labels at the pixel level, and the second aerial image is an unlabeled image corresponding to the first aerial image.

[0032] In the technical solution provided in the above step S101 of the present invention, a first aerial image and a second aerial image are obtained. For example, there are 10 images, each of which contains three types of images, namely, cars, pedestrians and buildings. The car in each image is labeled at the pixel level, and the pedestrians and buildings in each image are not labeled at the pixel level. The first aerial image is 10 images with the car labeled at the pixel level, and the second aerial image is 10 images with the pedestrians and buildings not labeled at the pixel level, or the pedestrians in each image are labeled at the pixel level, and the cars and buildings in each image are not labeled at the pixel level.

[0033] In step S102 , the first aerial image and the second aerial image are input into a feature extractor to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image.

[0034] In the technical solution provided in the above step S102 of the present invention, the feature extractor processes the first aerial image and the second aerial image to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image.

[0035] In step S103, based on the category label, feature extraction is performed on the first feature map and the second feature map to obtain multiple pixel-level sample features of the first aerial image, the true label of each pixel-level sample feature of the first aerial image, and multiple unlabeled pixel-level sample features of the second aerial image.

[0036] In the technical solution provided in step S103 of the present invention, for example, the car is labeled, and according to the category label of the pixel-level features of the car, the first feature map and the second feature map are subjected to feature extraction to obtain multiple pixel-level sample features x of the first aerial image. i , the true label y of each pixel-level sample feature i and multiple unlabeled pixel-level sample features x of the second aerial image j .

[0037] Step S104 : obtaining a prototype of each category in the plurality of pixel-level sample features of the first aerial image based on the plurality of pixel-level sample features of the first aerial image.

[0038] In the technical solution provided in the above step S104 of the present invention, a prototype of each category in the multiple pixel-level sample features of the first aerial image is obtained according to the multiple pixel-level sample features of the first aerial image.

[0039] Step S105: obtaining a prototype classifier based on the prototype of each category.

[0040] In the technical solution provided in step S105 of the present invention, each category prototype is spliced ​​to obtain the expression of the prototype classifier:

[0041]

[0042] Among them, W is the prototype classifier, is the prototype of each category, and the value of i ranges from 1 to n.

[0043] Step S106 , obtaining a classification constraint loss value based on the prototype classifier, each pixel-level sample feature of the first aerial image, and a true label of each pixel-level sample feature.

[0044] In the technical solution provided in the above step S106 of the present invention, a classification constraint loss value is obtained according to the prototype classifier, each pixel-level sample feature of the first aerial image, and the true label of each pixel-level sample feature.

[0045] Step S107, obtaining an unsupervised constrained loss value based on the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image, and pseudo-labels of multiple unlabeled pixel-level sample features of the second aerial image, wherein the pseudo-labels of multiple unlabeled pixel-level sample features of the second aerial image are determined based on the prototype classifier and multiple unlabeled pixel-level sample features of the second aerial image.

[0046] In the technical solution provided in the above step S107 of the present invention, an unsupervised constraint loss value is obtained based on the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image, and pseudo labels of multiple unlabeled pixel-level sample features of the second aerial image.

[0047] Step S108 : Input the prototype classifier and each pixel-level sample feature of the first aerial image into a supervised prototype contrastive learning module to obtain a supervised contrastive learning loss value.

[0048] In the technical solution provided in the above step S108 of the present invention, the supervised prototype contrastive learning module processes each pixel-level sample feature of the prototype classifier and the first aerial image to obtain a supervised contrastive learning loss value.

[0049] Step S109: Obtain a target loss value based on the classification constraint loss value, the unsupervised constraint loss value, and the supervised contrastive learning loss value.

[0050] In the technical solution provided in the above step S109 of the present invention, the target loss value is obtained according to the classification constraint loss value, the unsupervised constraint loss value and the supervised contrastive learning loss value.

[0051] Step S110 , using the pseudo labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image, updates the prototype of each category in the multiple pixel-level sample features of the first aerial image.

[0052] In the technical solution provided in the above step S110 of the present invention, the prototype of each category in the multiple pixel-level sample features of the first aerial image is updated by using the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image.

[0053] Step S111: Acquire the aerial image to be detected and determine the semantic segmentation result of the aerial image to be detected.

[0054] In the technical solution provided by step S111 of the present invention, through steps S101 to S111, a semantic segmentation model is obtained, an aerial image to be detected is obtained, and the aerial image to be detected is processed by the semantic segmentation model to obtain a semantic segmentation result of the aerial image to be detected, wherein the aerial image to be detected is input into a feature extractor to obtain a feature map, which is then input into a segmentation model for fully automatic segmentation to obtain a segmentation map of unknown category. The specific formula is as follows:

[0055]

[0056] Among them, I is the aerial image to be detected, SAMAutoSeg is the segmentation model, is a segmentation map of unknown category, the value range of l is 1~N, and the training sample prototype is adaptively updated with the same formula (4) using the feature map of the aerial image to be detected. Then, the updated prototype is used to densely predict the feature map, and the prediction result is used to classify the segmentation map of unknown category to obtain the final prediction result. The specific formula is as follows:

[0057]

[0058] in, For statistical operations, represents the mth test image, and p is the pixel-level prediction category result.

[0059] The above method of this embodiment is further introduced below.

[0060] As an optional implementation method, step S102 obtains the prototype of each category in the multiple pixel-level sample features of the first aerial image based on the multiple pixel-level sample features of the first aerial image, including: calculating the mean of the multiple pixel-level sample features of the first aerial image to obtain the prototype of each category in the multiple pixel-level sample features of the first aerial image.

[0061] In this embodiment, a mean value calculation is performed on a plurality of pixel-level sample features of the first aerial image to obtain a prototype of each category in the plurality of pixel-level sample features of the first aerial image.

[0062] As an optional embodiment, in step S104, the mean of the multiple pixel-level sample features of the first aerial image is calculated to obtain the prototype expression of each category in the multiple pixel-level sample features of the first aerial image:

[0063]

[0064] in, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, C i represents the number of categories of the first aerial image, x i Represents one of the multiple pixel-level sample features of the first aerial image, y i represents the label of the i-th category in each pixel-level sample feature of the first aerial image, and Num represents the number of samples of the i-th category in the first aerial image.

[0065] As an optional implementation method, step S106 obtains a classification constraint loss value based on the prototype classifier, each pixel-level sample feature of the first aerial image, and the true label of each pixel-level sample feature, including: performing similarity measurement prediction based on the prototype classifier and each pixel-level sample feature of the first aerial image to obtain the predicted probability of each pixel-level sample feature of the first aerial image; performing cross-entropy calculation on the predicted probability of each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature to obtain the classification constraint loss value.

[0066] In this embodiment, a similarity metric prediction is performed between the prototype classifier and each pixel-level sample feature of the first aerial image to obtain a predicted probability of each pixel-level sample feature of the first aerial image. A cross entropy calculation is performed between the predicted probability of each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature to obtain an expression for the classification constraint loss value:

[0067]

[0068] Among them, L cls is the classification constraint loss value, L CE (·) is the cross entropy loss function, N fore Represents the number of pixel-level sample features of the first aerial image, y i The true label of each pixel-level sample feature of the first aerial image, W is the prototype classifier, x iis the pixel-level sample feature of the first aerial image, W·x i is the predicted probability of each pixel-level sample feature of the first aerial image.

[0069] As an optional implementation method, step S107, based on the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image and the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image, obtains an unsupervised constrained loss value, including: performing similarity measurement prediction based on the prototype classifier and the multiple unlabeled pixel-level sample features of the second aerial image to obtain the predicted probability of each unlabeled pixel-level sample feature of the second aerial image; performing cross-entropy calculation on the predicted probability of each unlabeled pixel-level sample feature of the second aerial image and the pseudo-label of each unlabeled pixel-level sample feature to obtain the unsupervised constrained loss value.

[0070] In this embodiment, a similarity metric prediction is performed based on the prototype classifier and multiple unlabeled pixel-level sample features of the second aerial image to obtain the predicted probability of each unlabeled pixel-level sample feature of the second aerial image; the predicted probability of each unlabeled pixel-level sample feature of the second aerial image and the pseudo label of each unlabeled pixel-level sample feature are cross-entropy calculated to obtain the expression of the unsupervised constraint loss value:

[0071]

[0072] Among them, L us is the unsupervised constraint loss value, N back is the number of unlabeled pixel-level sample features of the second aerial image, x j are multiple unlabeled pixel-level sample features of the second aerial image, Pseudo labels for each pixel-level sample feature of the second aerial image.

[0073] As an optional embodiment, in step S108, the prototype classifier and each pixel-level sample feature of the first aerial image are input into the supervised prototype contrastive learning module, and the expression of the supervised contrastive learning loss value is obtained as follows:

[0074]

[0075] Among them, C i represents the i-th category number of the first aerial image, Indicates that in C i The supervised contrastive learning loss value under Indicates that the first aerial image is not of category C i The number of samples, x k Indicates that the first aerial image contains all the images except category C i Other pixel-level sample features, xj Represents each pixel-level sample feature of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, N fore Indicates the number of pixel-level sample features of the first aerial image.

[0076] As an optional embodiment, in step S109, based on the classification constraint loss value, the unsupervised constraint loss value and the supervised contrastive learning loss value, the expression for the target loss value is obtained as follows:

[0077] L total =L cls +α·L SPCL +β·L us (3)

[0078] Among them, L total is the target loss value, L cls is the classification constraint loss value, L SPCL is the supervised contrastive learning loss value, L us is the unsupervised constraint loss value, α is the first weight parameter, and β is the second weight parameter.

[0079] As an optional embodiment, step S110, using the pseudo labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image, updates the prototype of each category in the multiple pixel-level sample features of the first aerial image, including:

[0080]

[0081] in, is the update item of the prototype of the i-th category in the multiple pixel-level sample features of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, x s represents a single pixel-level sample feature in the unlabeled pixel-level sample features of the remaining images except the image of the i-th category in the first aerial image, x j represents the unlabeled pixel-level sample features in the remaining images of the first aerial image except the image of the i-th category, represents the pseudo label of a single pixel-level sample feature in the unlabeled pixel-level sample features in the remaining images except the image of the i-th category in the first aerial image, represents the pseudo labels of the unlabeled pixel-level sample features in the remaining images of the first aerial image except the image of the i-th category, is the prototype of the i-th category in the multiple pixel-level sample features of the updated first aerial image, and ε is the update item weight.

[0082] Experimental process of the embodiment of the present invention:

[0083] Simulation conditions:

[0084] The present invention is a simulation performed using Pytorch on an Intel(R) Xeon(R) silver 4214R CPU @ 2.40GHz, 128G memory, and a Linux operating system.

[0085] The data used in the simulation are public datasets.

[0086] Simulation content:

[0087] The simulation data is the UAV aerial photography datasets (UDD) and VDD, which contain 6 and 7 categories, respectively. We cropped the images from both datasets to 256 × 256 pixels. For each category, we extracted 10 image patches and their corresponding binary labels as training data, and the remaining image patches were used for testing.

[0088] In order to prove the effectiveness of the method, the present invention is compared with the traditional semantic segmentation methods Unet and Deeplab V3

[0089] We compared the proposed method with PCFNet, a common small-sample semantic segmentation method. To ensure a fair comparison, the traditional semantic segmentation methods were provided with identical image samples and fully annotated labels. For the common small-sample semantic segmentation method PCFNet, both single-category and full-category metrics were calculated, with the results denoted as PCFNet and PCFNet*, respectively. The proposed method is MPSAM, using mIoU as the evaluation metric. The comparison results are shown in Table 1:

[0090] Table 1 Comparison results

[0091] method UDD VDD Unet 12.92％ 9.87％ DeeplabV3 24.92％ 14.99％ DeeplabV3+ 24.24％ 22.76％ PSPNet 26.84％ 26.19％ DANet 24.32％ 31.1％ PCFNet 18.35％ 23.25％ <![CDATA[PCFNet * ]]> 14.81％ 17.72％ MPSAM (technology of the present invention) 36.73％ 45.2％

[0092] As can be seen from Table 1, the present invention is superior to other algorithms in terms of mIoU indicator.

[0093] In an embodiment of the present invention, a first aerial image and a second aerial image are obtained, wherein the first aerial image is an image with a category label at the pixel level for a portion of objects of the same category in a plurality of aerial images, and the second aerial image is an unlabeled image corresponding to the first aerial image; the first aerial image and the second aerial image are input into a feature extractor to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image; based on the category label, feature extraction is performed on the first feature map and the second feature map to obtain a plurality of pixel-level sample features of the first aerial image, a true label of each pixel-level sample feature of the first aerial image, and a plurality of unlabeled pixel-level sample features of the second aerial image; based on the plurality of pixel-level sample features of the first aerial image, a prototype of each category in the plurality of pixel-level sample features of the first aerial image is obtained; based on the prototype of each category, a prototype classifier is obtained; based on the prototype classifier, each pixel-level sample feature of the first aerial image, and the true label of each pixel-level sample feature, a classification constraint loss value is obtained; based on the prototype classifier, the plurality of unlabeled pixel-level sample features of the second aerial image, and the plurality of unlabeled pixel-level sample features of the second aerial image, a classification constraint loss value is obtained. The invention discloses a method for detecting a plurality of unlabeled pixel-level sample features of an image, and obtains an unsupervised constrained loss value by pseudo-labeling the plurality of unlabeled pixel-level sample features of the second aerial image, wherein the pseudo-labels of the plurality of unlabeled pixel-level sample features of the second aerial image are determined based on the prototype classifier and the plurality of unlabeled pixel-level sample features of the second aerial image; the prototype classifier and each pixel-level sample feature of the first aerial image are input into the supervised prototype contrast learning module to obtain a supervised contrast learning loss value; the target loss value is obtained based on the classification constraint loss value, the unsupervised constraint loss value and the supervised contrast learning loss value; the prototype of each category in the plurality of pixel-level sample features of the first aerial image is updated by using the pseudo-labels of the plurality of unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the plurality of pixel-level sample features of the first aerial image; the aerial image to be detected is obtained, and the semantic segmentation result of the aerial image to be detected is determined, which solves the technical problems in the prior art that the semantic segmentation model requires a large amount of labeled data, does not constrain the intra-class differences, leads to semantic aliasing, and has low multi-classification segmentation accuracy, and achieves the technical effect of not requiring a large amount of labeled data for training the semantic segmentation model, constraining the intra-class differences, reducing semantic aliasing, and improving the multi-classification segmentation accuracy.

[0094] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0095] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0097] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0098] In addition, the functional units in various embodiments of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0099] The above are only preferred embodiments of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A semantic segmentation method for local security with few labeled samples, characterized by: include: Acquire a first aerial image and a second aerial image, wherein the first aerial image is an image in which a portion of objects of the same category in a plurality of aerial images are annotated with category labels at the pixel level, and the second aerial image is an unlabeled image corresponding to the first aerial image; Inputting the first aerial image and the second aerial image into a feature extractor to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image; Based on the category label, feature extraction is performed on the first feature map and the second feature map to obtain multiple pixel-level sample features of the first aerial image, the true label of each pixel-level sample feature of the first aerial image, and multiple unlabeled pixel-level sample features of the second aerial image; Based on the multiple pixel-level sample features of the first aerial image, obtaining a prototype of each category in the multiple pixel-level sample features of the first aerial image; Based on the prototype of each category, a prototype classifier is obtained; Obtaining a classification constraint loss value based on the prototype classifier, each pixel-level sample feature of the first aerial image, and a true label of each pixel-level sample feature; obtaining an unsupervised constrained loss value based on the prototype classifier, the plurality of unlabeled pixel-level sample features of the second aerial image, and the pseudo-labels of the plurality of unlabeled pixel-level sample features of the second aerial image, wherein the pseudo-labels of the plurality of unlabeled pixel-level sample features of the second aerial image are determined based on the prototype classifier and the plurality of unlabeled pixel-level sample features of the second aerial image; Input the prototype classifier and each pixel-level sample feature of the first aerial image into the supervised prototype contrastive learning module to obtain the supervised contrastive learning loss value; Based on the classification constraint loss value, the unsupervised constraint loss value and the supervised contrastive learning loss value, the target loss value is obtained; Updating the prototype of each category in the multiple pixel-level sample features of the first aerial image using the pseudo labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image; Obtain the aerial image to be detected and determine the semantic segmentation result of the aerial image to be detected.

2. The method according to claim 1, characterized in that Based on the multiple pixel-level sample features of the first aerial image, a prototype of each category in the multiple pixel-level sample features of the first aerial image is obtained, including: The multiple pixel-level sample features of the first aerial image are averaged to obtain a prototype of each category in the multiple pixel-level sample features of the first aerial image.

3. The method according to claim 2, characterized in that The mean of multiple pixel-level sample features of the first aerial image is calculated to obtain the prototype expression of each category in the multiple pixel-level sample features of the first aerial image: in, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, C i represents the number of categories of the first aerial image, x i Represents one of the multiple pixel-level sample features of the first aerial image, y i represents the label of the i-th category in each pixel-level sample feature of the first aerial image, and Num represents the number of samples of the i-th category in the first aerial image.

4. The method according to claim 1, wherein Based on the prototype classifier, each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature, the classification constraint loss value is obtained, including: Performing similarity metric prediction based on the prototype classifier and each pixel-level sample feature of the first aerial image to obtain a prediction probability of each pixel-level sample feature of the first aerial image; The predicted probability of each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature are cross-entropy calculated to obtain the classification constraint loss value.

5. The method according to claim 1, wherein Based on the prototype classifier, the plurality of unlabeled pixel-level sample features of the second aerial image, and the pseudo labels of the plurality of unlabeled pixel-level sample features of the second aerial image, an unsupervised constraint loss value is obtained, including: Perform similarity metric prediction based on the prototype classifier and multiple unlabeled pixel-level sample features of the second aerial image to obtain a prediction probability of each unlabeled pixel-level sample feature of the second aerial image; The predicted probability of each unlabeled pixel-level sample feature of the second aerial image and the pseudo label of each unlabeled pixel-level sample feature are cross-entropy calculated to obtain the unsupervised constraint loss value.

6. The method according to claim 1, characterized in that Input the prototype classifier and each pixel-level sample feature of the first aerial image into the supervised prototype contrastive learning module, and the expression of the supervised contrastive learning loss value is obtained as follows: Among them, C i represents the i-th category number of the first aerial image, Indicates that in C i The supervised contrastive learning loss value under Indicates that the first aerial image is not of category C i The number of samples, x k Indicates that the first aerial image contains all the images except category C i Other pixel-level sample features, x i Represents each pixel-level sample feature of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, N fore Indicates the number of samples of all categories in the first aerial image.

7. The method according to claim 1, characterized in that Based on the classification constraint loss value, unsupervised constraint loss value and supervised contrastive learning loss value, the expression of the target loss value is: L total =L cls +α·L SPCL +β·L us (3) Among them, L total is the target loss value, L cls is the classification constraint loss value, L SPCL is the supervised contrastive learning loss value, L us is the unsupervised constraint loss value, α is the first weight parameter, and β is the second weight parameter.

8. The method according to claim 1, characterized in that Updating the prototype of each category in the multiple pixel-level sample features of the first aerial image using the pseudo labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image, including: in, is the update item of the prototype of the i-th category in the multiple pixel-level sample features of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, x s represents a single pixel-level sample feature in the unlabeled pixel-level sample features of the remaining images except the image of the i-th category in the first aerial image, x j represents the unlabeled pixel-level sample features in the remaining images of the first aerial image except the image of the i-th category, represents the pseudo label of a single pixel-level sample feature in the unlabeled pixel-level sample features in the remaining images except the image of the i-th category in the first aerial image, represents the pseudo labels of the unlabeled pixel-level sample features in the remaining images of the first aerial image except the image of the i-th category, is the prototype of the i-th category in the multiple pixel-level sample features of the updated first aerial image, and ε is the update item weight.

9. A processor, characterized in that: The processor is configured to run a program, wherein the program executes the method according to any one of claims 1 to 8 when running.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8.

Images