Semantic segmentation method for near-field security and protection under few-label sample condition
By using feature extraction and prototype classifiers in the semantic segmentation model, combined with supervised and unsupervised comparison learning, the problem of unconstrained in-class differences in semantic segmentation models under the condition of few labeling samples is solved, and the accuracy of multi-classification segmentation is improved.
Patent Information
- Application Number
- CN202411046330.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-01
AI Technical Summary
The prior art semantic segmentation model requires a large amount of label data under the condition of few labeling samples, resulting in unconstrained in-class differences, resulting in low semantic aliasing and multi-class segmentation accuracy.
By obtaining a part of the annotated image and the unlabeled image, the pixel-level features are extracted using the feature extractor, a prototype classifier is generated, and the prototype is updated through supervised and unsupervised comparison learning, and the target loss value is generated to reduce the dependence on in-class differences.
When training semantic segmentation models, there is no need for a large amount of label data, which reduces semantic aliasing and improves multi-classification segmentation accuracy.
Smart Images

Figure CN119027662A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a semantic segmentation method under conditions of few labeled samples for on-site security. Background Art
[0002] In the low-altitude security mission of on-site security, it is of great significance for drones to obtain low-altitude RGB data and use image segmentation technology to perform semantic analysis on the data to help make airspace security and defense decisions. Image semantic segmentation technology is a dense pixel-level semantic classification of scene images. The semantic segmentation method based on deep learning relies on a large number of densely annotated images, but due to time costs and sample collection difficulties, it is difficult to obtain sufficient annotated samples in practical applications. The semantic segmentation with a small number of annotated samples involved in the present invention aims to achieve pixel-level classification of images by relying on a small number of annotated images. This technology has important applications in urban planning, agricultural management, military security and other scenarios.
[0003] Existing semantic segmentation techniques with a small number of annotated images have made good progress through the combination of meta-learning and metric learning methods, but they usually require a large base data set and only focus on distinguishing a single category from the background in the image, which to some extent limits the application of this technology in practical scenarios. Dong N. et al. proposed in the paper “Dong N, Xing EP. Few-shot semantic segmentation with prototype learning [C] / / BMVC. 2018, 3 (4): 4” to use a prototype learner to extract prototypes representing semantic categories from annotated images for pixel-level feature measurement. Wang K. et al. proposed in the paper “Wang K, Liew JH, Zou Y, et al. Panet: Few-shot image semantic segmentation with prototype alignment [C] / / proceedings of the IEEE / CVF international conference on computer vision. 2019: 9197-9206” to use query images to calculate prototype alignment to improve the quality of prototype generation. Fan Q. et al. proposed in the document “Fan Q, Pei W, Tai YW, et al. Self-support few-shot semantic segmentation [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 701-719.” to generate a self-support prototype from a query image to guide self-semantic segmentation to alleviate intra-class feature differences.
[0004] Although these methods have made certain contributions, they all have their limitations. First, they require a large amount of basic training data, but many application scenarios cannot provide sufficient basic data; second, these methods only distinguish between a single category and the background, which greatly limits their practicality. Summary of the invention
[0005] The embodiment of the present invention provides a semantic segmentation method under the condition of few labeled samples for local security, so as to at least solve the technical problems that the semantic segmentation model in the prior art requires a large amount of labeled data, has no constraints on intra-class differences, leads to semantic aliasing, and has low multi-classification segmentation accuracy.
[0006] According to one aspect of an embodiment of the present invention, a semantic segmentation method under the condition of few labeled samples for on-site security is provided. The method may include: obtaining a first aerial image and a second aerial image, wherein the first aerial image is an image with a category label annotated at the pixel level for a portion of objects of the same category in multiple aerial images, and the second aerial image is an unlabeled image corresponding to the first aerial image; inputting the first aerial image and the second aerial image into a feature extractor to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image; based on the category label, performing feature extraction on the first feature map and the second feature map to obtain multiple pixel-level sample features of the first aerial image, the true label of each pixel-level sample feature of the first aerial image, and multiple unlabeled pixel-level sample features of the second aerial image; based on the multiple pixel-level sample features of the first aerial image, obtaining a prototype of each category in the multiple pixel-level sample features of the first aerial image; based on the prototype of each category, obtaining a prototype classifier; based on the prototype classifier, each pixel-level sample feature of the first aerial image, and each pixel-level sample feature The real label is obtained to obtain the classification constraint loss value; based on the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image and the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image, an unsupervised constraint loss value is obtained, wherein the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image are determined based on the prototype classifier and the multiple unlabeled pixel-level sample features of the second aerial image; the prototype classifier and each pixel-level sample feature of the first aerial image are input into the supervised prototype contrast learning module to obtain the supervised contrast learning loss value; based on the classification constraint loss value, the unsupervised constraint loss value and the supervised contrast learning loss value, a target loss value is obtained; using the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototypes of each category in the multiple pixel-level sample features of the first aerial image, the prototypes of each category in the multiple pixel-level sample features of the first aerial image are updated; the aerial image to be detected is obtained, and the semantic segmentation result of the aerial image to be detected is determined.
[0007] Optionally, based on multiple pixel-level sample features of the first aerial image, a prototype of each category in the multiple pixel-level sample features of the first aerial image is obtained, including: calculating the mean of the multiple pixel-level sample features of the first aerial image to obtain the prototype of each category in the multiple pixel-level sample features of the first aerial image.
[0008] Optionally, the mean of multiple pixel-level sample features of the first aerial image is calculated to obtain the expression of the prototype of each category in the multiple pixel-level sample features of the first aerial image:
[0009]
[0010] in, Represents the prototype of the i-th category among multiple pixel-level sample features of the first aerial image, C i represents the number of the i-th category of the first aerial image, x i Represents one of the multiple pixel-level sample features of the first aerial image, y i represents the label of the i-th category in each pixel-level sample feature of the first aerial image, and Num represents the number of samples of the i-th category of the first aerial image.
[0011] Optionally, a classification constraint loss value is obtained based on the prototype classifier, each pixel-level sample feature of the first aerial image, and the true label of each pixel-level sample feature, including: performing similarity metric prediction based on the prototype classifier and each pixel-level sample feature of the first aerial image to obtain the predicted probability of each pixel-level sample feature of the first aerial image; performing cross entropy calculation on the predicted probability of each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature to obtain the classification constraint loss value.
[0012] Optionally, an unsupervised constrained loss value is obtained based on a prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image, and pseudo-labels of multiple unlabeled pixel-level sample features of the second aerial image, including: performing similarity metric prediction based on the prototype classifier and multiple unlabeled pixel-level sample features of the second aerial image to obtain a predicted probability of each unlabeled pixel-level sample feature of the second aerial image; performing cross entropy calculation on the predicted probability of each unlabeled pixel-level sample feature of the second aerial image and the pseudo-label of each unlabeled pixel-level sample feature to obtain an unsupervised constrained loss value.
[0013] Optionally, the prototype classifier and each pixel-level sample feature of the first aerial image are input into a supervised prototype contrastive learning module, and the expression of the supervised contrastive learning loss value is obtained as follows:
[0014]
[0015] Among them, C i represents the i-th category number of the first aerial image, Indicates that in C i The supervised contrastive learning loss value under , Indicates that the first aerial image is not of category C i The number of samples, x k Indicates that the first aerial image contains all the images except for category C i Other pixel-level sample features, x i Represents each pixel-level sample feature of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, N foreRepresents the number of samples of all categories in the first aerial image.
[0016] Optionally, based on the classification constraint loss value, the unsupervised constraint loss value, and the supervised contrastive learning loss value, an expression for the target loss value is obtained as follows:
[0017] L total =L cls +α·L SPCL +β·L us (3)
[0018] Among them, L total is the target loss value, L cls is the classification constraint loss value, L SPCL is the supervised contrastive learning loss value, L us is the unsupervised constraint loss value, α is the first weight parameter, and β is the second weight parameter.
[0019] Optionally, using the pseudo labels of the plurality of unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the plurality of pixel-level sample features of the first aerial image, updating the prototype of each category in the plurality of pixel-level sample features of the first aerial image comprises:
[0020]
[0021] in, is the update item of the prototype of the i-th category in the multiple pixel-level sample features of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, x s represents a single pixel-level sample feature among the unlabeled pixel-level sample features in the remaining images except the image of the i-th category in the first aerial image, x j represents the unlabeled pixel-level sample features in the remaining images of the first aerial image except the image of the i-th category, represents a pseudo label of a single pixel-level sample feature among the unlabeled pixel-level sample features in the remaining images except the image of the i-th category in the first aerial image, represents the pseudo labels of the unlabeled pixel-level sample features in the remaining images of the first aerial image except the images of the i-th category, is the prototype of the i-th category in the multiple pixel-level sample features of the updated first aerial image, and ε is the update item weight.
[0022] Beneficial effects of the present invention:
[0023] The present invention proposes a semantic segmentation method under the condition of few labeled samples for on-site security. By processing a first aerial image and a second aerial image, a true label of each pixel-level sample feature of the first aerial image and multiple unlabeled pixel-level sample features of the second aerial image are obtained. According to the multiple pixel-level sample features of the first aerial image, a prototype of each category in the multiple pixel-level sample features of the first aerial image is obtained, and the prototypes of each category are spliced to obtain a prototype classifier. According to the prototype classifier, each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature, a prototype classifier is obtained. According to the prototype classifier, the multiple unlabeled pixel-level sample features of the second aerial image and the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image, an unsupervised constraint loss value is obtained. The prototype classifier and each pixel-level sample feature of the first aerial image are input into a supervised prototype classifier. A contrastive learning module is used to obtain a supervised contrastive learning loss value. A target loss value is obtained according to the classification constraint loss value, the unsupervised constraint loss value and the supervised contrastive learning loss value. The prototype of each category in the multiple pixel-level sample features of the first aerial image is updated by using the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image. The above steps obtain a semantic segmentation model of the aerial image, and the aerial image to be detected is input into the semantic segmentation model to obtain the semantic segmentation result of the aerial image to be detected. The technical problems that the semantic segmentation model in the prior art requires a large amount of labeled data, does not constrain the intra-class differences, leads to semantic aliasing, and has low multi-classification segmentation accuracy are solved. The technical effect of not requiring a large amount of labeled data to train the semantic segmentation model, constraining the intra-class differences, reducing semantic aliasing, and improving the multi-classification segmentation accuracy is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0025] Figure 1 It is a flowchart of a semantic segmentation method under the condition of few labeled samples for local security according to an embodiment of the present invention. DETAILED DESCRIPTION
[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] Example 1
[0029] According to an embodiment of the present invention, a semantic segmentation method for local security with few labeled samples is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.
[0030] Figure 1 is a flowchart of a semantic segmentation method for local security under the condition of few labeled samples according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:
[0031] Step S101, obtaining a first aerial image and a second aerial image, wherein the first aerial image is an image in which a portion of objects of the same category in multiple aerial images are annotated with category labels at the pixel level, and the second aerial image is an unlabeled image corresponding to the first aerial image.
[0032] In the technical solution provided in the above step S101 of the present invention, a first aerial image and a second aerial image are obtained. For example, there are 10 images, each of which contains three types of objects, namely, cars, pedestrians and buildings. The car in each image is labeled at the pixel level, and the pedestrians and buildings in each image are not labeled at the pixel level. The first aerial image is 10 images in which the car is labeled at the pixel level, and the second aerial image is an image in which the pedestrians and buildings in the 10 images are not labeled at the pixel level, or the pedestrians in each image are labeled at the pixel level, and the cars and buildings in each image are not labeled at the pixel level.
[0033] Step S102: input the first aerial image and the second aerial image into a feature extractor to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image.
[0034] In the technical solution provided in the above step S102 of the present invention, the feature extractor processes the first aerial image and the second aerial image to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image.
[0035] Step S103: Based on the category label, feature extraction is performed on the first feature map and the second feature map to obtain multiple pixel-level sample features of the first aerial image, the true label of each pixel-level sample feature of the first aerial image, and multiple unlabeled pixel-level sample features of the second aerial image.
[0036] In the technical solution provided in step S103 of the present invention, for example, the vehicle is labeled, and according to the category label of the pixel-level feature of the vehicle, the first feature map and the second feature map are subjected to feature extraction to obtain a plurality of pixel-level sample features x of the first aerial image. i , the true label y of each pixel-level sample feature i and multiple unlabeled pixel-level sample features x of the second aerial image j .
[0037] Step S104: obtaining a prototype of each category in the multiple pixel-level sample features of the first aerial image based on the multiple pixel-level sample features of the first aerial image.
[0038] In the technical solution provided in the above step S104 of the present invention, a prototype of each category in the multiple pixel-level sample features of the first aerial image is obtained according to the multiple pixel-level sample features of the first aerial image.
[0039] Step S105, obtaining a prototype classifier based on the prototype of each category.
[0040] In the technical solution provided in step S105 of the present invention, each category prototype is concatenated to obtain the expression of the prototype classifier:
[0041]
[0042] Among them, W is the prototype classifier, For the prototype of each category, the value of i ranges from 1 to n.
[0043] Step S106, obtaining a classification constraint loss value based on the prototype classifier, each pixel-level sample feature of the first aerial image, and a true label of each pixel-level sample feature.
[0044] In the technical solution provided in the above step S106 of the present invention, a classification constraint loss value is obtained according to the prototype classifier, each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature.
[0045] Step S107, obtaining an unsupervised constrained loss value based on the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image, and pseudo-labels of multiple unlabeled pixel-level sample features of the second aerial image, wherein the pseudo-labels of multiple unlabeled pixel-level sample features of the second aerial image are determined based on the prototype classifier and multiple unlabeled pixel-level sample features of the second aerial image.
[0046] In the technical solution provided in the above step S107 of the present invention, an unsupervised constraint loss value is obtained based on the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image, and pseudo-labels of multiple unlabeled pixel-level sample features of the second aerial image.
[0047] Step S108, inputting the prototype classifier and each pixel-level sample feature of the first aerial image into a supervised prototype contrast learning module to obtain a supervised contrast learning loss value.
[0048] In the technical solution provided in the above step S108 of the present invention, the supervised prototype contrastive learning module processes each pixel-level sample feature of the prototype classifier and the first aerial image to obtain a supervised contrastive learning loss value.
[0049] Step S109, obtaining a target loss value based on the classification constraint loss value, the unsupervised constraint loss value and the supervised contrastive learning loss value.
[0050] In the technical solution provided in the above step S109 of the present invention, the target loss value is obtained according to the classification constraint loss value, the unsupervised constraint loss value and the supervised contrastive learning loss value.
[0051] Step S110, using the pseudo labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image, updates the prototype of each category in the multiple pixel-level sample features of the first aerial image.
[0052] In the technical solution provided by the above step S110 of the present invention, the prototype of each category in the multiple pixel-level sample features of the first aerial image is updated by using the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image.
[0053] Step S111, obtaining the aerial image to be detected, and determining the semantic segmentation result of the aerial image to be detected.
[0054] In the technical solution provided by the above step S111 of the present invention, through steps S101 to S111, a semantic segmentation model is obtained, and an aerial image to be detected is obtained. The aerial image to be detected is processed by the semantic segmentation model to obtain a semantic segmentation result of the aerial image to be detected, wherein the aerial image to be detected is input into a feature extractor to obtain a feature map, which is input into a segmentation model for full-automatic segmentation to obtain a segmentation map of unknown category. The specific formula is as follows:
[0055]
[0056] Among them, I is the aerial image to be detected, SAMAutoSeg is the segmentation model, is a segmentation map of unknown category, the value range of l is 1~N, the feature map of the aerial image to be detected is used to adaptively update the training sample prototype with formula (4), and then the updated prototype is used to densely predict the feature map, and the prediction result is used to classify the segmentation map of unknown category to obtain the final prediction result. The specific formula is as follows:
[0057]
[0058] in, For statistical operations, represents the mth test image, and p is the pixel-level prediction category result.
[0059] The above method of this embodiment is further introduced below.
[0060] As an optional implementation method, step S102, based on multiple pixel-level sample features of the first aerial image, obtains the prototype of each category in the multiple pixel-level sample features of the first aerial image, including: calculating the mean of the multiple pixel-level sample features of the first aerial image to obtain the prototype of each category in the multiple pixel-level sample features of the first aerial image.
[0061] In this embodiment, a mean value calculation is performed on a plurality of pixel-level sample features of the first aerial image to obtain a prototype of each category in the plurality of pixel-level sample features of the first aerial image.
[0062] As an optional implementation manner, in step S104, the mean of multiple pixel-level sample features of the first aerial image is calculated to obtain the expression of the prototype of each category in the multiple pixel-level sample features of the first aerial image:
[0063]
[0064] in, Represents the prototype of the i-th category among multiple pixel-level sample features of the first aerial image, C i represents the number of the i-th category of the first aerial image, x i Represents one of the multiple pixel-level sample features of the first aerial image, y i represents the label of the i-th category in each pixel-level sample feature of the first aerial image, and Num represents the number of samples of the i-th category of the first aerial image.
[0065] As an optional implementation method, step S106 obtains a classification constraint loss value based on the prototype classifier, each pixel-level sample feature of the first aerial image, and the true label of each pixel-level sample feature, including: performing a similarity metric prediction based on the prototype classifier and each pixel-level sample feature of the first aerial image to obtain a predicted probability of each pixel-level sample feature of the first aerial image; performing a cross entropy calculation on the predicted probability of each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature to obtain a classification constraint loss value.
[0066] In this embodiment, the prototype classifier and each pixel-level sample feature of the first aerial image are predicted by similarity measurement to obtain the predicted probability of each pixel-level sample feature of the first aerial image, and the predicted probability of each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature are cross-entropy calculated to obtain the expression of the classification constraint loss value:
[0067]
[0068] Among them, L cls is the classification constraint loss value, L CE (·) is the cross entropy loss function, N fore Represents the number of pixel-level sample features of the first aerial image, y i The true label of each pixel-level sample feature of the first aerial image, W is the prototype classifier, x iis the pixel-level sample feature of the first aerial image, W·x i is the predicted probability of each pixel-level sample feature of the first aerial image.
[0069] As an optional implementation method, step S107, based on the prototype classifier, multiple unlabeled pixel-level sample features of the second aerial image and pseudo-labels of multiple unlabeled pixel-level sample features of the second aerial image, obtains an unsupervised constrained loss value, including: performing similarity metric prediction based on the prototype classifier and multiple unlabeled pixel-level sample features of the second aerial image to obtain the predicted probability of each unlabeled pixel-level sample feature of the second aerial image; performing cross entropy calculation on the predicted probability of each unlabeled pixel-level sample feature of the second aerial image and the pseudo-label of each unlabeled pixel-level sample feature to obtain the unsupervised constrained loss value.
[0070] In this embodiment, a similarity metric prediction is performed based on the prototype classifier and multiple unlabeled pixel-level sample features of the second aerial image to obtain the predicted probability of each unlabeled pixel-level sample feature of the second aerial image; the predicted probability of each unlabeled pixel-level sample feature of the second aerial image and the pseudo label of each unlabeled pixel-level sample feature are cross-entropy calculated to obtain the expression of the unsupervised constraint loss value:
[0071]
[0072] Among them, L us is the unsupervised constraint loss value, N back is the number of unlabeled pixel-level sample features of the second aerial image, x j are multiple unlabeled pixel-level sample features of the second aerial image, Pseudo labels for each pixel-level sample feature of the second aerial image.
[0073] As an optional implementation manner, in step S108, the prototype classifier and each pixel-level sample feature of the first aerial image are input into the supervised prototype contrast learning module, and the expression of the supervised contrast learning loss value is obtained as follows:
[0074]
[0075] Among them, C i represents the i-th category number of the first aerial image, Indicates that in C i The supervised contrastive learning loss value under , Indicates that the first aerial image is not of category C i The number of samples, x k Indicates that the first aerial image contains all the images except for category C i Other pixel-level sample features, xj Represents each pixel-level sample feature of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, N fore Represents the number of pixel-level sample features of the first aerial image.
[0076] As an optional implementation manner, in step S109, based on the classification constraint loss value, the unsupervised constraint loss value and the supervised contrastive learning loss value, an expression for obtaining the target loss value is:
[0077] L total =L cls +α·L SPCL +β·L us (3)
[0078] Among them, L total is the target loss value, L cls is the classification constraint loss value, L SPCL is the supervised contrastive learning loss value, L us is the unsupervised constraint loss value, α is the first weight parameter, and β is the second weight parameter.
[0079] As an optional implementation manner, step S110, using the pseudo labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image, updates the prototype of each category in the multiple pixel-level sample features of the first aerial image, including:
[0080]
[0081] in, is the update item of the prototype of the i-th category in the multiple pixel-level sample features of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, x s represents a single pixel-level sample feature among the unlabeled pixel-level sample features in the remaining images except the image of the i-th category in the first aerial image, x j represents the unlabeled pixel-level sample features in the remaining images of the first aerial image except the image of the i-th category, represents a pseudo label of a single pixel-level sample feature among the unlabeled pixel-level sample features in the remaining images except the image of the i-th category in the first aerial image, represents the pseudo labels of the unlabeled pixel-level sample features in the remaining images of the first aerial image except the images of the i-th category, is the prototype of the i-th category in the multiple pixel-level sample features of the updated first aerial image, and ε is the update item weight.
[0082] Experimental process of the embodiment of the present invention:
[0083] Simulation conditions:
[0084] The present invention is a simulation performed by using Pytorch on a central processing unit of Intel(R) Xeon(R) silver 4214R CPU@2.40GHz, a memory of 128G, and a Linux operating system.
[0085] The data used in the simulation are public datasets.
[0086] Simulation content:
[0087] The simulation data are UDD and VDD, which contain 6 and 7 categories respectively. We cut the images of the two datasets into 256 × 256 size. For each category, 10 image blocks and corresponding binary labels are extracted as training data, and the remaining image blocks are used for testing.
[0088] In order to prove the effectiveness of the method, this paper compares it with traditional semantic segmentation methods Unet and Deeplab V3
[0089] etc. and the general small sample semantic segmentation method PCFNet were compared. In order to make the comparison as fair as possible, the traditional semantic segmentation method provides the same image samples and full-category annotated labels for it. For the general small sample semantic segmentation method PCFNet, the single category index and full category index are calculated respectively, and the results are recorded as PCFNet and PCFNet* respectively. The method proposed in this invention is MPSAM, and mIoU is the evaluation index. The comparison results are shown in Table 1:
[0090] Table 1 Comparison results
[0091] method UDD VDD Unet 12.92% 9.87% DeeplabV3 24.92% 14.99% DeeplabV3+ 24.24% 22.76% PSPNet 26.84% 26.19% DANet 24.32% 31.1% PCFNet 18.35% 23.25% <![CDATA[PCFNet * ]]> 14.81% 17.72% MPSAM (Technology of the Invention) 36.73% 45.2%
[0092] It can be seen from Table 1 that the present invention is superior to other algorithms in terms of mIoU indicator.
[0093] In an embodiment of the present invention, the first aerial image and the second aerial image are obtained by acquiring a first aerial image and a second aerial image, wherein the first aerial image is an image with a category label at the pixel level for a portion of objects of the same category in multiple aerial images, and the second aerial image is an unlabeled image corresponding to the first aerial image; the first aerial image and the second aerial image are input into a feature extractor to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image; based on the category label, feature extraction is performed on the first feature map and the second feature map to obtain multiple pixel-level sample features of the first aerial image, the true label of each pixel-level sample feature of the first aerial image, and multiple unlabeled pixel-level sample features of the second aerial image; based on the multiple pixel-level sample features of the first aerial image, a prototype of each category in the multiple pixel-level sample features of the first aerial image is obtained; based on the prototype of each category, a prototype classifier is obtained; based on the prototype classifier, each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature, a classification constraint loss value is obtained; based on the prototype classifier, the multiple unlabeled pixel-level sample features of the second aerial image and the The invention discloses a method for detecting a plurality of unlabeled pixel-level sample features of an image, and obtains an unsupervised constrained loss value by using pseudo labels of the plurality of unlabeled pixel-level sample features of the second aerial image, wherein the pseudo labels of the plurality of unlabeled pixel-level sample features of the second aerial image are determined based on the prototype classifier and the plurality of unlabeled pixel-level sample features of the second aerial image; the prototype classifier and each pixel-level sample feature of the first aerial image are input into the supervised prototype contrast learning module to obtain a supervised contrast learning loss value; the target loss value is obtained based on the classification constraint loss value, the unsupervised constraint loss value and the supervised contrast learning loss value; the prototype of each category in the plurality of pixel-level sample features of the first aerial image is updated by using the pseudo labels of the plurality of unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the plurality of pixel-level sample features of the first aerial image; the aerial image to be detected is obtained, and the semantic segmentation result of the aerial image to be detected is determined, so as to solve the technical problems that the semantic segmentation model in the prior art requires a large amount of label data, does not constrain the intra-class differences, leads to semantic aliasing, and has low multi-classification segmentation accuracy, and achieves the technical effect that a large amount of label data is not required for training the semantic segmentation model, the intra-class differences are constrained, the semantic aliasing is reduced, and the multi-classification segmentation accuracy is improved.
[0094] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0095] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0096] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0097] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0098] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0099] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A semantic segmentation method for local security with few labeled samples. It is characterized in that include: Acquire a first aerial image and a second aerial image, wherein the first aerial image is an image in which a portion of objects of the same category in a plurality of aerial images are annotated with category labels at the pixel level, and the second aerial image is an unlabeled image corresponding to the first aerial image; Inputting the first aerial image and the second aerial image into a feature extractor to obtain a first feature map corresponding to the first aerial image and a second feature map corresponding to the second aerial image; Based on the category label, feature extraction is performed on the first feature map and the second feature map to obtain multiple pixel-level sample features of the first aerial image, the true label of each pixel-level sample feature of the first aerial image, and multiple unlabeled pixel-level sample features of the second aerial image; Based on a plurality of pixel-level sample features of the first aerial image, a prototype of each category in the plurality of pixel-level sample features of the first aerial image is obtained; Based on the prototype of each category, a prototype classifier is obtained; Obtaining a classification constraint loss value based on the prototype classifier, each pixel-level sample feature of the first aerial image, and a true label of each pixel-level sample feature; Based on the prototype classifier, the multiple unlabeled pixel-level sample features of the second aerial image, and the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image, an unsupervised constraint loss value is obtained, wherein the pseudo-labels of the multiple unlabeled pixel-level sample features of the second aerial image are determined based on the prototype classifier and the multiple unlabeled pixel-level sample features of the second aerial image; Inputting the prototype classifier and each pixel-level sample feature of the first aerial image into a supervised prototype contrastive learning module to obtain a supervised contrastive learning loss value; Based on the classification constraint loss value, the unsupervised constraint loss value and the supervised contrastive learning loss value, the target loss value is obtained; Using the pseudo labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image, the prototype of each category in the multiple pixel-level sample features of the first aerial image is updated; Obtain the aerial image to be detected, and determine the semantic segmentation result of the aerial image to be detected.
2. The method according to claim 1, It is characterized in that Based on the multiple pixel-level sample features of the first aerial image, a prototype of each category in the multiple pixel-level sample features of the first aerial image is obtained, including: The multiple pixel-level sample features of the first aerial image are averaged to obtain a prototype of each category in the multiple pixel-level sample features of the first aerial image.
3. The method according to claim 2, It is characterized in that The mean of multiple pixel-level sample features of the first aerial image is calculated to obtain the expression of the prototype of each category in the multiple pixel-level sample features of the first aerial image: in, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, C i represents the number of the i-th category of the first aerial image, x i Represents one of the multiple pixel-level sample features of the first aerial image, y i represents the label of the i-th category in each pixel-level sample feature of the first aerial image, and Num represents the number of samples of the i-th category of the first aerial image.
4. The method according to claim 1, It is characterized in that Based on the prototype classifier, each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature, the classification constraint loss value is obtained, including: Perform similarity metric prediction based on the prototype classifier and each pixel-level sample feature of the first aerial image to obtain a prediction probability of each pixel-level sample feature of the first aerial image; The predicted probability of each pixel-level sample feature of the first aerial image and the true label of each pixel-level sample feature are cross-entropy calculated to obtain the classification constraint loss value.
5. The method according to claim 1, It is characterized in that Based on the prototype classifier, the plurality of unlabeled pixel-level sample features of the second aerial image and the pseudo labels of the plurality of unlabeled pixel-level sample features of the second aerial image, an unsupervised constraint loss value is obtained, including: Perform similarity metric prediction based on the prototype classifier and multiple unlabeled pixel-level sample features of the second aerial image to obtain a prediction probability of each unlabeled pixel-level sample feature of the second aerial image; The predicted probability of each unlabeled pixel-level sample feature of the second aerial image and the pseudo label of each unlabeled pixel-level sample feature are cross-entropy calculated to obtain an unsupervised constraint loss value.
6. The method according to claim 1, It is characterized in that The prototype classifier and each pixel-level sample feature of the first aerial image are input into the supervised prototype contrastive learning module, and the expression of the supervised contrastive learning loss value is obtained as follows: Among them, C i represents the i-th category number of the first aerial image, Indicates that in C i The supervised contrastive learning loss value under , Indicates that the first aerial image is not of category C i The number of samples, x k Indicates that the first aerial image contains all the images except for category C i Other pixel-level sample features, x i Represents each pixel-level sample feature of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, N fore Represents the number of samples of all categories in the first aerial image.
7. The method according to claim 1, It is characterized in that Based on the classification constraint loss value, unsupervised constraint loss value and supervised contrastive learning loss value, the expression of the target loss value is: L total =L cls +α·L SPCL +β·L us (3) Among them, L total is the target loss value, L cls is the classification constraint loss value, L SPCL is the supervised contrastive learning loss value, L us is the unsupervised constraint loss value, α is the first weight parameter, and β is the second weight parameter.
8. The method according to claim 1, It is characterized in that Using the pseudo labels of the multiple unlabeled pixel-level sample features of the second aerial image and the prototype of each category in the multiple pixel-level sample features of the first aerial image, updating the prototype of each category in the multiple pixel-level sample features of the first aerial image, including: in, is the update item of the prototype of the i-th category in the multiple pixel-level sample features of the first aerial image, Represents the prototype of the i-th category in multiple pixel-level sample features of the first aerial image, x s represents a single pixel-level sample feature among the unlabeled pixel-level sample features in the remaining images except the image of the i-th category in the first aerial image, x j represents the unlabeled pixel-level sample features in the remaining images of the first aerial image except the image of the i-th category, represents a pseudo label of a single pixel-level sample feature among the unlabeled pixel-level sample features in the remaining images except the image of the i-th category in the first aerial image, represents the pseudo labels of the unlabeled pixel-level sample features in the remaining images of the first aerial image except the images of the i-th category, is the prototype of the i-th category in the multiple pixel-level sample features of the updated first aerial image, and ε is the update item weight.
9. A processor, It is characterized in that The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 8 when running.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Class-specified multi-mode joint representation method for large-scene remote sensing image classification
CN113850216A
Picture semantic segmentation method and device
CN113902913A
Semi-supervised image semantic segmentation method and system, computer equipment and medium
CN117333666A
Medical image segmentation method based on weak supervision
CN117830332A
KR20240035066A
