An active learning method for semantic segmentation based on superpixel feature representation learning

Through superpixel feature characterization learning and comparison learning, the samples of the semantic segmentation model are dynamically selected for annotation, which solves the problem of unoptimized feature selection methods in the prior art, and realizes high-performance semantic segmentation under small labeling costs.

CN117173701BActive Publication Date: 2024-11-05NORTHWESTERN POLYTECHNICAL UNIV

Patent Information

Application Number
CN202311029793.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-14
Publication Date
2024-11-05
Estimated Expiration
2043-08-14

AI Technical Summary

Technical Problem

The existing semantic segmentation active learning methods are difficult to update dynamically while maintaining data information, resulting in insufficient optimization of feature selection methods and affecting model performance.

Method used

Using a method based on superpixel feature representation learning, the feature vectors output from the backbone network are extracted through the feature extraction header, and the feature repository and UMAP algorithm are used for dimensionality reduction and clustering. Combined with comparative learning, the feature representation is optimized, and the sample is dynamically selected for labeling.

Benefits of technology

Achieve better semantic segmentation performance under smaller annotation costs, ensure sample diversity and information volume, and improve the classification performance and feature representation capabilities of the network.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117173701B_ABST
    Figure CN117173701B_ABST
Patent Text Reader

Abstract

The present invention discloses a semantic segmentation active learning method based on superpixel feature representation learning, designs a feature extraction head to extract the features output by the backbone network, outputs pixel-by-pixel feature vectors, stores feature vectors of historical categories through a feature repository, selects positive and negative samples in contrastive learning through difficult sample mining technology, and applies INFONCE loss to make feature vectors of the same category close and vectors of different categories far apart. In terms of sample selection, the UMAP algorithm is trained through the vectors in the feature repository, and then the feature vectors of the unlabeled image area segmented by superpixel are reduced in dimension, and then clustered through the Kmeans algorithm, and the superpixel blocks that need to be labeled are selected in each cluster according to the entropy value, so as to achieve the best segmentation performance with the least labeled data. Experiments show that 95% of the performance of the fully labeled data can be achieved with only 8% of the labeled data, which is ahead of the existing active learning methods applied to semantic segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and in particular relates to an active learning method for semantic segmentation. Background Art

[0002] Semantic segmentation is an important task in the field of computer vision. Its goal is to achieve fine classification of images through pixel-level image recognition, that is, to label the object category to which each pixel in the image belongs. This technology has a wide range of applications in autonomous driving, environmental understanding, biomedical analysis, remote sensing, and robotic manipulation. With the rise of deep learning methods, the semantic segmentation task has made significant progress. However, the success of semantic segmentation models based on deep networks largely depends on the availability of large-scale training datasets with pixel-level annotations. But the process of obtaining pixel-by-pixel annotations is very time-consuming and costly, and usually requires manual annotation by experts.

[0003] To address this problem, active learning has been proposed as an effective solution. Active learning is a machine learning algorithm that actively selects samples with the highest value for annotation, thereby using as few but high-quality samples as possible to achieve the best possible performance of the model. In other words, active learning methods can improve the efficiency of sample annotation and maximize the performance of the model under a limited annotation budget. It is a data efficiency improvement solution based on samples, and is particularly suitable for tasks with high annotation costs and difficulties, such as medical images, unmanned driving, anomaly detection, and related problems based on Internet big data.

[0004] In the active learning framework, the design of the query strategy is crucial, that is, it is used to determine which samples are worth being labeled by experts. The value of a sample is not fixed. It is closely related to the characteristics of the sample itself, as well as factors such as the task being solved and the network model used. To give a simple example, in the cat-dog binary classification problem, a photo of a dog that looks like a cat often has a higher value for training the classification model because it is difficult to distinguish correctly. However, if the same photo is applied to the animal and plant binary classification problem, its importance is reduced because the model can relatively easily distinguish between cats and dogs and plants. Therefore, the design of the query strategy is not simple and fixed, but needs to be set according to the specific environment, problem, and needs.

[0005] This has led to the emergence of various query strategies. In the sample selection process, diversity and uncertainty are two main criteria. In addition, some researchers combine multiple query strategies and use hybrid strategies for querying. This can take into account both the amount of information and diversity of the selected samples, thereby better selecting the samples required by the model for annotation.

[0006] Active learning for semantic segmentation is challenging because it involves a large number of pixels and complex correlations between these pixels. Huang et al. proposed the idea of ​​using consistency regularization in the semi-supervised field to improve the learning of the network in the active learning process in "Semi-supervised active learning with temporal output discrepancy [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 3447-3456.", and judged the importance of the image by the difference between the teacher and student network before and after, and used this as a criterion for sample selection. Golestaneh et al. proposed using self-consistency based on rectangular regions to determine whether it is worth labeling in "Importance of self-consistency in active learning for semantic segmentation [J]. arXiv preprintarXiv: 2008.01860, 2020." Specifically, the sum of the pixel entropy before and after the horizontal flip of the same image is calculated as the criterion for selecting samples, and the mean square error loss before and after the flip is further used as an auxiliary loss to optimize the network model during network training. Cai et al. demonstrated the effectiveness of different active learning methods for semantic segmentation based on superpixel level under the condition of using realistic annotation costs, that is, the number of clicks required to label data, rather than the number of pixels, in "Revisiting superpixels for active learning in semantic segmentation with realistic annotation costs [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 10988-10997." And by assigning a primary label (the class label of most pixels in the region) to each region to obtain an estimate of the class distribution, and using this to assign weights to superpixel blocks, a class-balanced acquisition function was proposed to improve the performance of active learning.

[0007] There are still some problems with current active learning methods for semantic segmentation. In order to achieve diversity, previous strategies either directly use feature distance as the criterion or use non-learnable dimensionality reduction techniques before applying clustering. The former strategy calculates the distance between high-dimensional features, but the redundant information in the high-dimensional representation may affect the accuracy of the distance metric. The latter strategy is static and non-learnable. However, the network state and corresponding features will change with the participation of more annotated data, so the data selection method should be adapted accordingly. Neither of these two strategies is optimal. How to dynamically update the information of the data itself while removing a large amount of data redundancy has become an urgent problem to be solved. Summary of the invention

[0008] In order to overcome the shortcomings of the prior art, the present invention provides a semantic segmentation active learning method based on superpixel feature representation learning. In view of the problem that semantic segmentation requires a large number of labeled images, in terms of model architecture, a feature extraction head is designed to extract the features output by the backbone network, output pixel-by-pixel feature vectors, store feature vectors of historical categories through a feature repository, select positive and negative samples in contrastive learning through difficult sample mining technology, and apply INFONCE loss to make feature vectors of the same category close and vectors of different categories far away. In terms of sample selection, the UMAP algorithm is trained through the vectors in the feature repository, and then the feature vectors of the unlabeled image area segmented by superpixel are reduced in dimension, and then clustered by the Kmeans algorithm, and the superpixel blocks that need to be labeled are selected in each cluster according to the entropy value, so as to achieve the best segmentation performance with the least labeled data. Experiments show that only 8% of the labeled data can achieve 95% of the performance of the full labeled data, which is ahead of the existing active learning methods applied to semantic segmentation.

[0009] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0010] Step 1: Perform superpixel segmentation on the dataset D, that is, divide each image into multiple superpixel blocks; then randomly select a portion of the superpixel blocks from the segmented superpixel blocks and mark them as the initial labeled dataset D L , and the rest is taken as the unlabeled dataset D U ;

[0011] The network model adopts a teacher-student network framework, where the student network θ s and the teacher network θ t They have the same network structure, which consists of a backbone network and two parallel modules; the backbone network adopts the FCN+Resnet50 structure, and the two parallel modules are linear classification modules θ seg and feature extraction module θ proj ,θseg and θ proj They are two different modules that do not share parameters and follow the backbone network; Represent the student network θ s The corresponding linear classification module and feature extraction module, Represent the teacher network θ t The corresponding linear classification module and feature extraction module; both parallel modules are composed of two-dimensional convolution, batch normalization function, Relu activation function and Dropout; the initial feature storage library M is set to empty;

[0012] Step 2: Using the teacher network θ t For the unlabeled dataset D U Make predictions through the linear classification module and feature extraction module Obtain the classification probability and feature vector of each pixel point; for all pixels in the same superpixel block, take the average of their feature vectors to obtain the feature vector representation f of the superpixel block R ; The information entropy value of the pixel is obtained according to the classification probability of the pixel, and the information entropy values ​​of all pixels in the same superpixel block are averaged to obtain the information entropy of the corresponding superpixel area;

[0013] Step 3: When the feature repository M is empty, the nonlinear dimensionality reduction method UMAP is randomly initialized; when M is not empty, the feature vectors in M ​​are used to train UMAP to learn a mapping function; the feature vectors of the superpixel blocks are reduced to two-dimensional space by the UMAP method; then the Kmeans clustering method is applied to cluster the reduced feature vectors, each clustering cluster contains a set of feature vectors, representing a group of similar superpixel blocks, and the superpixel block with the largest average information entropy value in step 2 is obtained in each cluster as the sample to be marked;

[0014] Step 4: After the superpixel block set selected in step 3 is labeled by expert O, it is added to the labeled dataset D. L , and in the unlabeled dataset D U Remove the corresponding super pixel block;

[0015] Step 5: Use the updated D L After weak data enhancement, they are input into the student network and the teacher network respectively, and the corresponding classification probability and feature representation are obtained through their respective linear classification modules and feature extraction modules to calculate the main loss L respectively. CE and contrast loss L Contra ;

[0016] The main loss uses cross entropy loss, and the linear classification module of the student network Get the predicted value of each pixel, and then calculate the cross entropy loss based on the true label value;

[0017] For contrast loss, InfoNCE is used. The calculation requires three different samples: anchor samples, positive samples, and negative samples. The selection of anchor samples is based on the linear classification module of the teacher network. Get the predicted value of each pixel, calculate the entropy value of each pixel, and select the corresponding pixel according to the size to pass through the student network feature extraction module Output features; positive samples are the average of feature vectors of the same category in the current feature repository; negative samples are the features with the highest similarity between vectors of different categories in the current feature repository and the anchor sample;

[0018] Step 6: Update the feature repository M;

[0019] Linear classification module based on teacher network The probability value of the output is used to calculate the information entropy of each pixel and the feature extraction module The output feature vectors are sorted according to the entropy values ​​of the corresponding pixels. For each category, the first M e Small feature vectors are added to the feature repository, which stores the feature vectors of each category. The feature repository is dynamically updated by the feature vectors generated by each training. If the length of added features exceeds the preset length, the first feature to be queued is removed from the queue.

[0020] Step 7: The total loss is the weighted sum of the cross entropy loss and the contrast loss. The network is trained using the Adam optimizer to update the parameters of the student network, and then the parameters of the teacher network are updated using the exponential moving average.

[0021] Step 8: Repeat steps 2 to 7 until the preset budget or specified performance is reached.

[0022] Preferably, the structure of the linear classification module is, first, a convolution layer, the number of convolution kernels of which is one quarter of the number of input channels, the convolution kernel size is 3×3, and the padding is set to 1; then a batch normalization operation is performed, and then a Relu activation function is applied, followed by a Dropout operation with a probability set to 0.1, and finally a convolution layer, the number of convolution kernels is the number of categories to be classified, and the convolution kernel size is 1×1; the structure of the feature extraction module is set to be consistent with the linear classification module, the difference is that in the last convolution layer of the feature extraction module, the number of convolution kernels is the dimension size of the feature vector, which is set to 128, that is, the feature vector dimension is set to 128.

[0023] Preferably, the calculation formula of information entropy in step 2 is as follows:

[0024]

[0025] Among them, H(i) represents the entropy value of the i-th point, C represents the number of categories, Represents the predicted probability value of the i-th point in category c.

[0026] Preferably, the cross entropy loss for:

[0027]

[0028] Among them, N L Represents the pixel set of all labeled data, y ic Indicates whether the i-th pixel belongs to category c.

[0029] Preferably, the contrast loss L Contra for:

[0030]

[0031] Among them, M a is the number of anchor samples, z ci is the feature vector of the i-th anchor sample in category c, is the feature vector of the jth negative sample corresponding to the i-th anchor sample in category c, is the corresponding positive sample of the i-th anchor sample in category c. Each anchor sample has a corresponding positive sample and M n negative samples; τ is a temperature coefficient, set to 0.5.

[0032] Preferably, the total loss is a weighted sum of the cross entropy loss and the contrast loss, expressed as:

[0033] L total =L CE +0.1*L contra .

[0034] The beneficial effects of the present invention are as follows:

[0035] 1) An active learning method based on exploring superpixel features for semantic segmentation is proposed, which achieves better semantic segmentation performance with lower data annotation cost;

[0036] 2) By selecting superpixel regions, the boundary information of the object is maintained as much as possible, and the same superpixel region belongs to the same category as much as possible, so that the obtained superpixel features are more specific and reliable; the superpixel blocks are reduced in dimension and clustered by the learnable nonlinear dimensionality reduction method UMAP and the clustering method Kmeans, and then the selected superpixel blocks are determined by information entropy, so that the selected samples can ensure diversity while containing the maximum amount of information, ensuring the best training effect at the lowest annotation cost.

[0037] 3) Optimize the feature representation ability of pixels through contrastive learning, so that the pixel features of the same category are close to each other, and the pixel features of different categories are far away from each other. Combined with the feature repository to store historical feature vectors, the feature vectors used for contrastive learning training and sample selection are more robust, effectively reducing the impact of noise, thereby improving the network's pixel-by-pixel classification performance during the sample selection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a network training framework diagram of the method of the present invention. DETAILED DESCRIPTION

[0039] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0040] The present invention aims to solve the problem that a large amount of labeled data is required for network training in semantic segmentation, and proposes a semantic segmentation active learning method based on superpixel feature learnable clustering and contrastive learning. The method aims to filter out as few representative and high-information samples as possible from a data set by constructing an active learning framework based on superpixel features, combining a learnable nonlinear dimensionality reduction method and contrastive learning, so as to reduce the annotation cost and achieve better semantic segmentation performance. The active learning method includes two parts. First, in order to ensure the diversity and reliability of the selected samples, the image is divided into superpixel blocks by a traditional superpixel segmentation method, and a feature repository is designed to store the representative feature vectors of each category. Then, the learnable nonlinear dimensionality reduction method UMAP is trained using the feature repository, and the feature vectors of the superpixel blocks are reduced in dimension by the trained UMAP. Next, the feature vectors after dimensionality reduction are clustered using the clustering method Kmeans to obtain different clusters. Finally, the information entropy of the superpixel blocks in each cluster is calculated to determine which superpixel blocks are selected as samples; second, in order to improve the representation ability of the feature vector, the present invention introduces a contrastive learning method in the network training process. According to the characteristics of active learning, by comparing and learning the feature vectors generated by the backbone network, the feature vectors of samples of the same type are made as close as possible, and the feature vectors of samples of different types are made as separate as possible. This step can effectively improve the discriminative ability of feature vectors, while enhancing the representation ability of feature storage libraries on feature space, thereby improving the accuracy of semantic segmentation. Through this method, a small number of representative and high-information samples can be effectively selected for annotation while maintaining data diversity and reliability, reducing annotation costs and improving the performance of semantic segmentation.

[0041] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0042] A semantic segmentation active learning method based on superpixel feature learnable clustering and contrastive learning, specifically comprising the following steps:

[0043] S1, first perform superpixel segmentation on the dataset D, that is, divide each image into multiple superpixel blocks. Then randomly select a small number of superpixel blocks from the segmented superpixel blocks and mark them as the initial labeled dataset D L , and the rest is taken as the unlabeled dataset D U The network model adopts a teacher-student network framework, where the student network θ s and the teacher network θ t They have the same network structure, which consists of a backbone network and two modules. The backbone network adopts FCN+Resnet50 structure. The two modules are linear classification module θ seg and feature extraction module θ proj ,θseg and θ proj They are two different modules that do not share parameters and are placed right behind the backbone network. and Represent the student network θ s and the teacher network θ t The corresponding linear classification module and feature extraction module. Both modules are composed of two-dimensional convolution, batch normalization function, Relu activation function and Dropout. Next, the labeled dataset D L After weak data enhancement, it is input into the student network θ s In the above example, the vectors output by the backbone network are respectively passed through the linear classification module and feature extraction module The classification probability and feature vector of each pixel are obtained to complete the initialization training of the student and teacher networks, and the specific steps are shown in S5. The initial feature storage library M is set to empty.

[0044] S2, using the teacher network θ t For the unlabeled dataset D U Make predictions through the linear classification module and feature extraction module Obtain the classification probability and feature vector of each pixel. For all pixels in the same superpixel area, take the average of their feature vectors to obtain the feature vector representation f of the superpixel area. R The information entropy value of the pixel is obtained according to the classification probability of the pixel, and the information entropy of all pixels in the same superpixel area is averaged to obtain the information entropy of the corresponding superpixel area.

[0045] S3, when the feature repository M is empty, the nonlinear dimensionality reduction method UMAP is randomly initialized; when M is not empty, the feature vectors in M ​​are used to train UMAP to learn a mapping function that can reduce the dimensionality of the features while maintaining their high-dimensional spatial structure. The feature vectors of the superpixel blocks are reduced to two-dimensional space by the UMAP method. Next, the Kmeans clustering method is applied to cluster the reduced feature vectors. Each cluster will contain a set of feature vectors representing a group of similar superpixel blocks. In each cluster, the superpixel block with the largest average entropy value in step S2 is obtained as the sample to be marked.

[0046] S4: After the superpixel block set selected by S3 is labeled by expert O, it is added to the labeled dataset D. L , and in the unlabeled dataset D U Remove the corresponding superpixel blocks.

[0047] S5, with updated DL After weak data enhancement, they are input into the student network and the teacher network respectively, and the corresponding classification probability and feature representation are obtained through their respective linear classification modules and feature extraction modules to calculate the main loss L respectively. CE and auxiliary loss L Contra For the main loss, cross entropy loss is used, and the linear classification module of the student network is used. Get the predicted value of each pixel, and then calculate the cross entropy loss based on the true label value. InfoNCE is used for auxiliary loss, and three different samples, anchor samples, positive samples, and negative samples, are required for calculation. The selection of anchor samples is based on the linear classification module of the teacher network. Get the predicted value of each pixel, calculate the entropy value of each pixel, and select the corresponding pixel according to the size to pass through the student network feature extraction module Output features; positive samples are the average of feature vectors of the same category in the current feature repository; negative samples are the features with the highest similarity between vectors of different categories in the current feature repository and the anchor sample.

[0048] S6, update the feature repository M. According to the teacher network linear classification module The probability value of the output is used to calculate the information entropy of each pixel and the feature extraction module The output feature vectors are sorted according to the entropy values ​​of the corresponding pixels. For each category, the first M e Small feature vectors are added to the feature repository, which stores a certain number of feature vectors for each category. The feature repository is dynamically updated through the feature vectors generated by each training. If the length of added features exceeds the preset length, the first feature to be queued will be removed from the queue.

[0049] S7, the total loss is the weighted sum of the cross entropy loss and the contrast loss. The network is trained by the Adam optimizer, the parameters of the student network are updated, and then the parameters of the teacher network are updated by exponential moving average.

[0050] S8, repeat the above steps S2-S7 until the preset budget or specified performance is reached, which can be referred to Table 1, or can be set as needed.

[0051] Example:

[0052] like Figure 1 As shown in the figure, in order to solve the problem that semantic segmentation requires a large amount of labeled data, this paper invented a semantic segmentation active learning method based on superpixel feature learnable clustering and contrastive learning. Figure 1 It is the model structure diagram of the proposed method, which specifically includes the following steps:

[0053] S1, for a given dataset D, the Seeds superpixel segmentation method is used, where the number of segmentations for each image is set to 40, to perform superpixel segmentation preprocessing on all images to obtain a set of superpixel blocks. 100 superpixel blocks are randomly selected from all superpixel blocks and marked as the initial labeled dataset D L , and the rest is taken as the unlabeled dataset D U .

[0054] The network model adopts a teacher-student network framework, where the student network θ s and the teacher network θ t They have the same network structure, which consists of a backbone network and two modules. The backbone network adopts the FCN+Resnet50 structure, and the two modules are linear classification module θ seg and feature extraction module θ proj , the former is used to predict the classification probability value of the pixel point, and the latter is used to extract the high-dimensional feature representation of the pixel point. and Represent the student network θ s and the teacher network θ t The corresponding linear classification module and feature extraction module. The structure of the linear classification module is as follows: first, a convolution layer with a convolution kernel of one-fourth the number of input channels, a convolution kernel size of 3×3, and padding set to 1; then a batch normalization operation is performed, and then a Relu activation function is applied, followed by a Dropout operation with a probability set to 0.1, and finally a convolution layer with a convolution kernel of 1×1, the number of categories to be classified. The structure of the feature extraction module is basically the same as that of the linear classification module, except that in the last convolution layer, the number of convolution kernels is the dimension of the feature vector, which is set to 128 here, that is, the feature vector dimension is set to 128. The outputs of these two modules are subjected to an upsampling operation to obtain the classification probability of the pixel point and the corresponding feature vector.

[0055] The labeled dataset D L After weak data enhancement, it is input into the student network θ s In the above example, the vectors output by the backbone network are respectively passed through the linear classification module and feature extraction module The classification probability and feature vector of each pixel are obtained to complete the initialization training of the student and teacher networks. The specific steps are shown in S5. The initialization feature storage library M is empty, with a total of C queues, where C is the number of categories in the data set and the size of each queue is 1024. The number of superpixels to be labeled to be selected for each active learning iteration is S, which is determined according to different data sets. The specific settings can be referred to Table 1, or they can be set as needed.

[0056] S2, in active learning, all superpixel blocks R of unlabeled data U As network input, according to the teacher network θ t Linear classification module and feature extraction module Get the classification probability and feature vector of each pixel. Calculate the corresponding entropy value according to the classification probability of pixel i. The larger the entropy, the more information the pixel contains.

[0057]

[0058] Among them, H(i) represents the entropy value of the i-th point, C represents the number of categories, Represents the predicted probability value of the i-th point in category c.

[0059] Then, for each superpixel region, the feature vectors of the pixels contained in it are averaged to obtain the feature vector representing the superpixel block. At the same time, the entropy values ​​of all pixels in the region are calculated and averaged to obtain the entropy value representation of the region. At this point, the feature vector and entropy value of each superpixel block are obtained as its representation.

[0060] S3, when the feature repository M is empty, the nonlinear dimensionality reduction method UMAP is randomly initialized; when M is not empty, the UMAP algorithm is trained with the feature vectors in the feature repository to learn the structural information of the feature space. UMAP is a learnable dimensionality reduction algorithm that can construct a high-dimensional graph representation of the data and optimize the low-dimensional graph so that it is as similar to the high-dimensional graph as possible in structure. In this process, UMAP constructs a weighted graph based on the similarity of the data, in which the strength of the edge represents the proximity of two points. Then, UMAP projects this graph into a low-dimensional space. In the method, the target dimension of dimensionality reduction is set to two dimensions for the convenience of visualization and debugging.

[0061] Use the trained UMAP algorithm to reduce the dimension of the feature vector of the superpixel block and map it to a two-dimensional space. Next, use the K-means algorithm to cluster the feature vector after dimension reduction. In each iteration, the number of clusters is equal to the number of superpixel blocks S to be selected. The specific number depends on the size of the data set. Please refer to the experimental results Table 1. This strategy ensures the diversity of the selected samples. For each cluster cluster, the feature vectors contained therein have similar characteristics. This means that they are close to each other in the space after dimension reduction. In each cluster, according to the entropy value calculated in step S2, the superpixel block with the maximum entropy value is selected as the sample to be marked. Through such a selection strategy, it is ensured that the selected samples have a high amount of information. This method can dynamically select samples with diversity and information.

[0062] S4: Update the labeled dataset D according to the set of superpixel blocks to be labeled selected in step S3. L and the unlabeled dataset D U The selected superpixel blocks are handed over to expert O for annotation and added to the labeled dataset D L , and the unlabeled dataset D U Remove this data.

[0063] S5, the labeled dataset D updated according to step S4 L , train the network. In each iteration of active learning, the number of rounds of network training is set to E, which is set to 3 here, that is, three rounds of iterative training are performed on the labeled data set to improve network performance.

[0064] Specifically, the input image is randomly flipped as weak data enhancement. The enhanced image is input into the student network and the teacher network respectively, and the cross entropy loss is calculated by comparing the probability value output by the linear classification module of the student network with the true label of the image.

[0065]

[0066] Among them, N L Represents the pixel set of all labeled data, y ic Indicates whether the i-th pixel belongs to category c.

[0067] For contrast loss, the InfoNCE loss function is adopted Update and optimize the feature vector. This loss function uses the cosine similarity between anchor samples, positive samples, and negative samples to calculate the loss through normalized cross entropy. The goal is to increase the similarity between anchor samples and positive samples and reduce the similarity with negative samples to optimize the representation of the feature vector.

[0068]

[0069] Among them, M a is the number of anchor samples, z ci is the feature vector of the i-th anchor sample in category c, is the feature vector of the jth negative sample corresponding to the i-th anchor sample in category c, is the corresponding positive sample of the i-th anchor sample in category c. Each anchor sample has a corresponding positive sample and M n negative samples. τ is a temperature coefficient, set to 0.5.

[0070] For the output of the feature extraction module of the student network, select M for each category according to the size of the entropy value. a(set to 50) feature vectors as anchor samples. The selection strategy is divided into two parts: one part is the feature vector with the largest entropy value and the other part is the randomly selected feature vector. The formula of the anchor sample selection strategy is as follows:

[0071]

[0072] Among them, F s Represents the feature set output by the feature extraction module of the student network. i c Represents the feature of the i-th pixel of the c-th class. Indicates that in F s Random Selection The feature vector of pixels, represents the entropy value of the i-th pixel of the c-th class predicted correctly by the teacher network, Indicates according to After sorting by size Large value.

[0073] Then select the corresponding positive and negative samples in the feature repository. The positive sample is the average value of all feature vectors of the same category in the current feature repository. The negative sample is the first M cosine distances between each category in the current feature repository and the selected anchor sample. n In the experiment, M is set n is 50, and the negative sample selection strategy formula is as follows:

[0074]

[0075] Among them, F m represents the feature set of the feature repository, cosine(f i c ,z cj ) represents the cosine distance between the i-th feature and the j-th anchor sample in the c-th category, cosine j Indicates that the cosine distance between the anchor sample j and the feature repository is sorted, and the Mth n The value corresponding to the larger value.

[0076] S6, the feature repository is used to store the feature vectors of each category during the training process. If the feature vectors of all pixels are stored, a large amount of memory will be occupied and there will be redundant features. Through the dynamically updated feature repository, that is, during the training process, only some of the filtered feature vectors are stored. As the training progresses, the feature repository will be continuously updated to maintain an appropriate size, so that contrastive learning can be effectively performed under limited GPU memory, solving the problem that the number of samples in contrastive learning is limited by GPU memory. The feature repository contains C queues. Each queue is set to a fixed length of 1024. Each category will be filtered into the queue M at each network iteration. e The length of the feature vectors is set to 128. When the length exceeds the fixed value, the earliest feature vector that enters the queue will be dequeued, so as to achieve the effect of dynamically updating the feature storage library.

[0077] According to the output of the linear classification module of the teacher network, the pixels are predicted and the pixels predicted correctly are sorted according to their entropy values. For each category, the top M e The feature vector corresponding to each pixel is added to the feature storage library M, which is used as the update method of the feature storage library. Select a specific feature F enqueue The formula for updating the feature repository is as follows:

[0078]

[0079] Among them, F t represents the feature set output by the feature extraction module of the teacher network, Indicates the c-th sample, sorted by entropy value, the M-th e The value corresponding to the larger value.

[0080] S7, the total loss is the weighted sum of the cross entropy loss and the contrast loss.

[0081] L total =L CE +0.1*L contra

[0082] L CE represents the cross entropy loss, L contra Denotes contrast loss. The student network is optimized and trained using the Adam optimizer, where Adam's parameters β1 and β2 are set to 0.9 and 0.999, and the initial learning rate and weight decay are set to 5e-4 and 2e-4, respectively. The teacher network parameters are obtained using the following formula:

[0083]

[0084] in, is the teacher network θ t The parameters at the kth iteration, For student network θ s The parameters at the kth iteration, α is the set hyperparameter, which is set to 0.99 here.

[0085] S8, so far is an iterative process of active learning. Repeat the above steps S2-S7 to sample, label, and train the network until the preset budget or specified performance is reached. You can refer to Table 1 or set it yourself as needed.

[0086] Table 1

[0087]

[0088] In order to verify the effectiveness of the method, this embodiment uses multiple public data sets for experiments and compares them with other active learning methods. The NWPU-YRCC data set contains 524 training data, 180 verification data, and 183 test data. The size of each image is 1024×512. The CamVid data set is a data set of urban road scenes publicly released by the University of Cambridge, containing 367 training data, 101 verification data, and 233 test data. The size of each image is 480×360 pixels. Cityscapes is a widely used urban semantic segmentation data set that contains high-quality images from various cities. The data set includes 5,000 finely annotated images of size 1024×2048 pixels, covering 19 semantic categories. The data set is divided into three subsets, with 2,975 images in the training set, 500 images in the verification set, and 1,524 images in the test set. Due to the large data set, the images were resized to 512×1024 pixels during the training process.

[0089] The algorithm proposed in this experiment is compared with several other algorithms, including full data (training using the entire training data set), Random, Equal (Importance of self-consistency in active learning for semantic segmentation), TOD (Temporal Output Discrepancy for Active Learning), and MEAL (MEAL: Manifold Embedding-based Active Learning). The evaluation index is MIOU (mean intersection over union), which is used to evaluate the prediction accuracy. The larger the MIOU, the better the prediction result. Budget represents the proportion of the amount of data used to the total data. The performance of the method in this embodiment (ours) is the best among the above methods. When achieving 95% of the full data training performance, the method in this embodiment only uses 8% of the data, while the Equal method requires 12% of the data and the TOD method requires 14% of the data. It can also be seen that the method in this embodiment maintains the same performance on different data sets and has good robustness.

[0090] The present invention proposes an active learning framework, the core idea of ​​which is to reduce the dimension of the features of the superpixel region through a learnable dimensionality reduction method, and then use a clustering method to ensure that the selected samples are diverse, and use the entropy value to ensure that these samples have rich information. At the same time, we also combine contrastive learning to improve the representation ability of feature vectors. It is worth noting that this method is not limited to the use of UMAP algorithm, Kmeans clustering and specific contrastive learning methods, but can flexibly apply other learnable dimensionality reduction methods, clustering algorithms and different contrastive learning methods.

Claims

1. A semantic segmentation active learning method based on superpixel feature representation learning, characterized in that: The steps include: Step 1: Perform superpixel segmentation on the dataset D, that is, divide each image into multiple superpixel blocks; then randomly select a portion of the superpixel blocks from the segmented superpixel blocks and mark them as the initial labeled dataset D L , and the rest is taken as the unlabeled dataset D U ; The network model adopts a teacher-student network framework, where the student network θ s and the teacher network θ t They have the same network structure, which consists of a backbone network and two parallel modules; the backbone network adopts the FCN+Resnet50 structure, and the two parallel modules are linear classification modules θ seg and feature extraction module θ proj ,θ seg and θ proj They are two different modules that do not share parameters and follow the backbone network; Represent the student network θ s The corresponding linear classification module and feature extraction module, Represent the teacher network θ t The corresponding linear classification module and feature extraction module; both parallel modules are composed of two-dimensional convolution, batch normalization function, Relu activation function and Dropout; the initial feature storage library M is set to empty; Step 2: Using the teacher network θ t For the unlabeled dataset D U Make predictions through the linear classification module and feature extraction module Obtain the classification probability and feature vector of each pixel point; for all pixels in the same superpixel block, take the average of their feature vectors to obtain the feature vector representation f of the superpixel block R ; The information entropy value of the pixel is obtained according to the classification probability of the pixel, and the information entropy values ​​of all pixels in the same superpixel block are averaged to obtain the information entropy of the corresponding superpixel area; Step 3: When the feature repository M is empty, the nonlinear dimensionality reduction method UMAP is randomly initialized; when M is not empty, the feature vectors in M ​​are used to train UMAP to learn a mapping function; the feature vectors of the superpixel blocks are reduced to two-dimensional space by the UMAP method; then the Kmeans clustering method is applied to cluster the reduced feature vectors, each clustering cluster contains a set of feature vectors, representing a group of similar superpixel blocks, and the superpixel block with the largest average information entropy value in step 2 is obtained in each cluster as the sample to be marked; Step 4: After the superpixel block set selected in step 3 is labeled by expert O, it is added to the labeled dataset D. L , and in the unlabeled dataset D U Remove the corresponding super pixel block; Step 5: Use the updated D L After weak data enhancement, they are input into the student network and the teacher network respectively, and the corresponding classification probability and feature representation are obtained through their respective linear classification modules and feature extraction modules to calculate the main loss L respectively. CE and contrast loss L Contra ; The main loss uses cross entropy loss, and the linear classification module of the student network Get the predicted value of each pixel, and then calculate the cross entropy loss based on the true label value; For contrast loss, InfoNCE is used. The calculation requires three different samples: anchor samples, positive samples, and negative samples. The selection of anchor samples is based on the linear classification module of the teacher network. Get the predicted value of each pixel, calculate the entropy value of each pixel, and select the corresponding pixel according to the size to pass through the student network feature extraction module Output characteristics; The positive sample is the average value of the feature vectors of the same category in the current feature repository; the negative sample is the feature with the highest similarity between the vectors of different categories in the current feature repository and the anchor sample; Select M for each category according to the size of the entropy value a The feature vector is used as the anchor sample, and the selection strategy is divided into two parts: one part is the feature vector with the largest entropy value and the other part is the randomly selected feature vector; the anchor sample selection strategy formula is as follows: Among them, M a is the number of anchor samples, F s represents the feature set output by the feature extraction module of the student network, Represents the feature of the i-th pixel of the c-th category; Indicates that in F s Random Selection The feature vector of pixels, represents the entropy value of the i-th pixel of the c-th class predicted correctly by the teacher network, Indicates according to After sorting by size Large value; Step 6: Update the feature repository M; According to the output of the linear classification module of the teacher network, the pixels are predicted and the pixels predicted correctly are sorted according to their entropy values. For each category, the top M e The feature vector corresponding to the pixel point is added to the feature storage library M, which is used as the update method of the feature storage library; select a specific feature F enqueue The formula for updating the feature repository is as follows: Among them, F t represents the feature set output by the feature extraction module of the teacher network, Indicates the c-th sample, sorted by entropy value, the M-th e The value corresponding to the larger value; Step 7: The total loss is the weighted sum of the cross entropy loss and the contrast loss. The network is trained using the Adam optimizer to update the parameters of the student network, and then the parameters of the teacher network are updated using the exponential moving average. Step 8: Repeat steps 2 to 7 until the preset budget or specified performance is reached.

2. The method for semantic segmentation active learning based on superpixel feature representation learning according to claim 1, characterized in that: The structure of the linear classification module is as follows: first, a convolution layer, the number of convolution kernels of which is one quarter of the number of input channels, the convolution kernel size is 3×3, and the padding is set to 1; then a batch normalization operation is performed, and then a Relu activation function is applied, followed by a Dropout operation with a probability set to 0.1, and finally a convolution layer, the number of convolution kernels of which is the number of categories to be classified, and the convolution kernel size is 1×1; the structure of the feature extraction module is set to be consistent with the linear classification module, the difference is that in the last convolution layer of the feature extraction module, the number of convolution kernels is the dimension size of the feature vector, which is set to 128, that is, the dimension of the feature vector is set to 128.

3. The method for semantic segmentation active learning based on superpixel feature representation learning according to claim 1, characterized in that: The calculation formula of information entropy in step 2 is as follows: Among them, H(i) represents the entropy value of the i-th point, C represents the number of categories, Represents the predicted probability value of the i-th point in category c.

4. The method for semantic segmentation active learning based on superpixel feature representation learning according to claim 1, characterized in that: The cross entropy loss for: Among them, N L Represents the pixel set of all labeled data, y ic Indicates whether the i-th pixel belongs to category c.

5. The method for semantic segmentation active learning based on superpixel feature representation learning according to claim 1, characterized in that: The contrast loss L Contra for: Among them, M a is the number of anchor samples, z ci is the feature vector of the i-th anchor sample in category c, is the feature vector of the jth negative sample corresponding to the i-th anchor sample in category c, is the corresponding positive sample of the i-th anchor sample in category c. Each anchor sample has a corresponding positive sample and M n negative samples; τ is a temperature coefficient, set to 0.

5.

6. The method for semantic segmentation active learning based on superpixel feature representation learning according to claim 1, characterized in that: The total loss is the weighted sum of the cross entropy loss and the contrast loss, expressed as: L total =L CE +0.1*L contra .

Citation Information

Patent Citations

  • Saliency detection method based on active learning

    CN110443257A

  • Yellow River ice semantic segmentation method based on multi-attention mechanism double-flow fusion network

    CN111160311A

  • Image defect segmentation method based on super-pixel active learning and semi-supervised learning strategy

    CN114758133A

  • Small target feature enhancement image segmentation method and system

    CN114937148A

  • Training Neural Networks for Object Detection

    US20230196749A1

Cited By

  • Active learning method and system for medical image data annotation with combination of uncertainty and representativeness

    CN120236134A