Image refinement segmentation method based on presentation consistent contrast learning

Abstract

The invention discloses an image fine segmentation method based on presentation consistent contrast learning. The method comprises the following steps: step 1, constructing a basic segmentation module, and obtaining a feature map after a picture passes through the basic segmentation module; 2, constructing a superpixel-guided representation consistency comparison learning module, and taking the feature map as the input of the superpixel-guided representation consistency comparison learning module; 3, constructing a local style guided representation consistency comparison learning module; taking the feature map as the input of a local style guided representation consistency comparison learning module; inter-block similarity loss calculation and pixel-level comparison loss calculation are carried out in the two modules; and step 4, combining the pixel-level contrast learning loss and the inter-block similarity loss to carry out combined training, and minimizing the loss. According to the method, the influence of the error tag on the segmentation model is effectively reduced, a refined segmentation result is realized, and the image can be segmented more finely.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a method for fine-grained image segmentation based on image consistency contrast learning. Background Art

[0002] Image segmentation is a basic computer vision task that aims to assign a semantic category to each pixel and has attracted great attention in the past few years. Semantic segmentation usually adopts deep learning methods. Starting from the pioneering fully convolutional network (FCN) and the Deep Lab series, CNN-based models have become the main design choice for dense prediction tasks. With the great success of natural language processing, there has been a surge in interest in introducing Transformer into visual tasks. Segformer, SETR, etc. represented by Transformer have shown great potential. Semantic segmentation models have undergone major changes in network architecture. With the rapid development of deep learning and natural language processing, the performance boundaries of semantic segmentation have been greatly pushed.

[0003] In the era of deep learning, the architecture of the segmentation model can be roughly divided into two parts: encoder and decoder. In the encoder stage, convolution and pooling operations are mainly performed to extract high-dimensional features containing semantic information. The encoder usually adopts popular classification networks, mainly VGG, ResNet, and DenseNet. The self-attention mechanism is also commonly used in the encoder network. The attention mechanism is used to represent the correlation between different regions (pixels) or different channels in the feature map. It calculates the weighted sum of the pairwise correlations of a single sample at all positions to update the features at each position.

[0004] In the decoder stage, the multi-layer high-dimensional feature vector extracted by the encoder is usually mapped back to the original image using operations such as interpolation, deconvolution, or de-pooling. For the decoder, it is usually used in conjunction with the encoder to achieve better results. There are different types of decoders for different goals, including achieving multi-scale receptive fields, collecting multi-scale semantics, expanding receptive fields, strengthening edge features, capturing global context, etc.

[0005] Contrastive learning is an unsupervised learning method for learning representations of data without label information. Its goal is to learn discriminative representations by comparing similarities and differences. In contrastive learning, data is organized into positive and negative pairs. Positive pairs contain similar data samples, while negative pairs contain dissimilar data samples. By comparing the differences between positive and negative pairs, the model can learn feature representations of the data such that similar samples are closer in the representation space, while dissimilar samples are more dispersed.

[0006] Although the classic segmentation network has produced impressive performance on classic datasets, the existing datasets generally use coarse-grained semantic labeling, which results in the labels of the training data containing noise, limiting the effectiveness of the model. In addition, some face sketches and face portraits do not have real labels, and the pseudo labels produced have errors in the boundaries, leading to the limitations of the model. Therefore, it is imperative to design and implement efficient and refined image segmentation methods, realize refined segmentation of images, face images, and medical images, and reduce the segmentation error rate.

[0007] The patent application with application publication number CN118053051A and name “Hyperspectral Remote Sensing Image Classification Method Based on Superpixel Self-Attention Mechanism” discloses a hyperspectral remote sensing image classification method based on superpixel self-attention mechanism. The shortcoming of this method is that when extracting superpixel feature points, the connection between superpixel points is not deeply explored. The simple use of superpixels cannot enable the network to learn more representative features. However, by using superpixels and contrast learning, by bringing the superpixel features of the same area closer, tight clusters can be formed in the feature space, and the boundaries between different categories can be made clearer.

[0008] SETR, represented by Transformer, uses ViT as the backbone and combines multiple CNN decoders to expand feature resolution. Despite its good performance, there are some limitations. ViT outputs single-scale low-resolution features instead of multi-scale features. The computational cost is high on large images. SegNeXt, represented by CNN, uses convolutional attention operations to capture multi-scale features, and uses a lightweight decoder to obtain multi-scale context from local to global, achieving adaptability in spatial and channel dimensions and information aggregation from low-level to high-level. Although SegNeXt reduces the number of parameters to a certain extent, it usually requires a large amount of data. Although classic segmentation networks have produced impressive performance on classic datasets, existing datasets generally use coarse-grained semantic labeling, which results in noisy labels in the training data, limiting the effectiveness of the model. In addition, some artistic portraits of human faces do not have real labels, and the pseudo labels produced have certain errors, leading to limitations of the model. Summary of the invention

[0009] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a method for fine-grained image segmentation based on consistent contrast learning, which aims to learn a refined image segmentation model from coarse-grained labeled data. By using contrast learning to constrain the distance between superpixels with the same label, the segmentation features of the same region are shortened, the differences in the segmentation features of different regions are enhanced, and the impact of wrong labels on the segmentation model is effectively reduced, thus achieving a refined segmentation result. Secondly, by using style consistency to constrain the same segmented regions to have the same style, the misclassification is reduced as much as possible, and the image can be segmented more finely.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] The image fine segmentation method based on appearance consistent contrast learning includes the following steps:

[0012] Step 1: Build a basic segmentation module and pass the image through the basic segmentation module to obtain a feature map;

[0013] Step 2: construct a superpixel-guided image consistency contrast learning module, and use the feature map as the input of the superpixel-guided image consistency contrast learning module;

[0014] Step 3: Construct a local style guided representation consistency contrast learning module; use the feature map as the input of the local style guided representation consistency contrast learning module;

[0015] The inter-block similarity loss and pixel-level contrast loss are calculated in two modules, which helps to improve the accuracy and fineness of segmentation.

[0016] Step 4: Combine the pixel-level contrastive learning loss and the inter-patch similarity loss for joint training to minimize the loss.

[0017] The step 1 is specifically as follows:

[0018] The basic segmentation module adopts an encoder-decoder structure, with SegFormer as the basic segmentation module. The encoder adopts the Transformer architecture and the decoder adopts the Uper structure. The encoder is used to extract the feature representation of the image, and the decoder is used to map these features back to the semantic segmentation results at the pixel level.

[0019] Furthermore, the encoder introduces the self-attention mechanism of the Transformer module, which enables the model to model the relationship between pixels on a global scale, so as to better capture contextual information and improve the accuracy of semantic segmentation. The decoder is used to receive multi-layer feature information of the encoder and input it into the Pyramid Pooling Module (PPM).

[0020] The structure of the PPM module is used to aggregate context information of different scales to obtain global context information; it effectively increases the receptive field and enables better subsequent segmentation.

[0021] The specific operation is as follows: through convolution and upsampling operations, multiple layers of features are fused and features suitable for segmentation are generated. Finally, the results obtained by PPM are mapped to the features of the target number of channels through the segmentation head to obtain the segmentation map and label image, and the loss function is calculated.

[0022] Furthermore, the input of the basic segmentation module is a picture of size H×W×3;

[0023] First, the image is divided into blocks of size 4×4, and the blocks are passed as input to the multi-layer Transformer encoder to obtain multi-layer features of the original image size {1 / 4, 1 / 8, 1 / 16, 1 / 32}, where i∈{1,2,3,4} and the number of channels C i+1 Greater than C i , i represents the corresponding different image scale features. Next, the multi-layer features are input into the decoder part; the highest layer features are input into the pyramid pooling module (PPM) to obtain the pooled features;

[0024] Through upsampling and convolution operations, the features of other layers are fused with the features after pooling to obtain a size of Segmentation of high-dimensional features;

[0025] Finally, through a segmentation convolution head, the segmentation high-dimensional features are mapped to a size of The segmentation result of cls Indicates the number of segmentation types.

[0026] The step 2 is specifically as follows:

[0027] Input an image of size H×W×3. After being processed by the basic segmentation module, the algorithm obtains a Segmentation of high-dimensional features;

[0028] Among them, one branch inputs the segmentation high-dimensional features into the segmentation head to generate a size of The segmentation map output;

[0029] The other branch inputs the segmented high-dimensional features into the feature mapping head and maps them into a size of characteristics, and conduct L 2 Regularization, this feature will be used in the subsequent calculation of contrastive learning loss;

[0030] L2 Regularization:

[0031]

[0032] x i represents the i-th element of vector x, and y represents the normalized vector;

[0033] Will The feature input superpixel guided appearance consistency contrast learning module calculates the pixel-level contrast learning loss. The pixel-level contrast learning loss is for a pixel i and its corresponding semantic label. Positive samples belong to the category Other pixels of , negative samples are pixels belonging to other categories, and the pixel-level contrast loss is defined as:

[0034]

[0035] Where P i and N i Represent the embedding sets of positive and negative samples of pixel i respectively, and the positive and negative samples and anchor point i are not limited to come from the same image.

[0036] The step 3 is specifically as follows:

[0037] The local style guided representation consistency contrast learning module includes a segmentation feature extraction skeleton, a style feature extraction network, and an inter-block feature similarity calculation module;

[0038] The segmentation feature extraction skeleton adopts the same feature extraction method as the basic segmentation module, and uses the feature mapping head to obtain the image segmentation features and perform block operations on the image segmentation features;

[0039] The style feature extraction network extracts style features from images divided into blocks to calculate the style similarity between blocks. The inter-block feature similarity calculation module obtains the segmentation feature similarity matrix and the style feature similarity matrix between image blocks by calculating the Gram matrix. The Gram matrix obtains the correlation between different blocks by performing the inner product of the blocks. Then, the style feature similarity matrix is ​​used to constrain the distribution of the segmentation feature similarity matrix.

[0040] The specific process is as follows: an image of size H×W×3 is input. After being processed by the basic segmentation module, the algorithm obtains a Segmentation of high-dimensional features;

[0041] Then, the segmented high-dimensional features are processed by the feature mapping head to obtain a size of The image scale segmentation feature is L 2 Regularize and divide the segmentation feature into blocks;

[0042] L 2 Regularization:

[0043]

[0044] x i represents the i-th element of vector x, and y represents the normalized vector. At the same time, the algorithm crops the input image of size H×W×3 to obtain H×W×3×P image blocks, where P represents the number of blocks. Then, the corresponding style feature block is obtained from the image block to obtain a feature representation of size H×W×C×P. Similar to the segmentation feature, the style feature block also needs to go through L 2 Regularization processing,Next, the algorithm matches the image segmentation feature blocks obtained in the segmentation stage with the style feature blocks, and obtains the inter-block similarity matrix (segmentation feature similarity matrix and style feature similarity matrix) by calculating the Gram matrix, with a size of P×P×C. Finally, the algorithm constrains the consistency of the two similarity matrices through the inter-block similarity loss.

[0045] The inter-block similarity loss aligns the style feature block with the segmentation feature block to achieve inter-block similarity constraint face segmentation. The inter-block similarity loss spatially segments the image on scale s, and defines scale s as 32. This size can better obtain style features and is more suitable for matching segmentation blocks and style blocks. Divide it into M blocks Right now Then, the Gram matrix from the segmentation patch and the style patch is calculated, and then the L2 distance between the segmentation matrix and the style matrix is ​​minimized as follows:

[0046]

[0047] Among them, φ seg ((·)) represents the activation map after the segmentation skeleton network feature extraction and activation by the ReLU layer, φ vgg ((·)) represents the style activation map after style feature extraction and activation by the ReLU layer. Represents the inter-block Gram matrix, capturing the negative correlation of channel scales. Based on the inter-block similarity loss of the style patch, the L2 distance of the Gram matrix between the style patch and the segmentation feature patch is calculated to measure the segmentation consistency effect of the local pattern.

[0048] In step 4, the pixel-level contrast learning loss and the inter-block similarity loss are combined, and the most basic cross entropy loss in the segmentation task is added to train the entire segmentation model, jointly guiding the optimization of the model parameters.

[0049] Beneficial effects of the present invention:

[0050] The present invention proposes a new image segmentation method based on consistent contrast learning. It uses a self-attention mechanism and an efficient feature fusion method to achieve the most advanced results in the field of face image segmentation, and can effectively solve the problem of low segmentation accuracy caused by model training on coarse-grained labels. At the same time, the method uses a superpixel-guided consistent contrast learning module to construct an embedding space that makes the same categories closer and different categories farther. The consistent contrast learning module guided by the local style can capture the details and style types of the local pattern. Combining the two modules can obtain more accurate and detailed segmentation results.

[0051] (1) The present invention proposes a local style-guided representation consistency contrast learning facial image refinement segmentation strategy, which can perform more refined segmentation of facial images and effectively reduce the impact of incorrect labels on the segmentation model. In addition, the strategy uses a loss function and a branch model to adjust the segmentation model, which does not increase the computational overhead of the inference stage.

[0052] (2) The present invention proposes a strategy for realizing fine segmentation of face images through superpixel-guided consistent contrast learning, which can also perform more fine segmentation of face images and effectively reduce the impact of wrong labels on the segmentation model. In addition, the strategy uses loss functions and branch models to adjust the segmentation model, which does not increase the computational overhead of the inference stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a diagram of the pyramid pooling module of the present invention.

[0054] Figure 2 This is a basic segmentation module diagram of the present invention.

[0055] Figure 3 This is a diagram of the superpixel-guided representation consistency contrast learning module of the present invention.

[0056] Figure 4 This is a diagram of the local style guided representation consistency comparison learning module of the present invention.

[0057] Figure 5 This is an example diagram of the segmentation results of the present invention on the CelebAMask dataset.

[0058] Figure 6 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0060] like Figure 6 As shown in FIG. 1 , the image fine-grained segmentation method based on image consistency contrast learning includes:

[0061] Step 1: Build the basic segmentation module:

[0062] The basic segmentation module adopts an encoder-decoder structure. The basic structure of most encoders is to stack N feature extraction modules. Input image X, after passing through the feature extraction modules in sequence, the image size is reduced in sequence, and the number of channels is increased in sequence, and a segmentation feature map of H×W×C is obtained. The basic segmentation module can be any segmentation network, such as U-Net, SegNeXt, ConvNeXt based on CNN or SegFormer, SETR based on Transformer. Most decoders obtain the multi-scale features generated by the encoder, fuse these features and restore the resolution of the feature map through operations such as deconvolution or upsampling to generate the predicted segmentation result.

[0063] The present invention chooses SegFormer as the basic segmentation module. The encoder adopts the Transformer architecture and the decoder adopts the Uper structure, where the encoder is used to extract the feature representation of the image, and the decoder is used to map these features back to the pixel-level semantic segmentation results. The encoder part introduces the self-attention mechanism of the Transformer module, which enables the model to model the relationship between pixels on a global scale, thereby better capturing contextual information and improving the accuracy of semantic segmentation. The decoder is used to receive the multi-layer feature information of the encoder and input it into the Pyramid Pooling Module (PPM).

[0064] Figure 1 The structure of the PPM module is shown. After that, through convolution and upsampling operations, multi-layer features are fused and features suitable for segmentation are generated. Finally, the features are mapped to features of the target number of channels through the segmentation head to obtain the segmentation map and label image, and the loss function is calculated.

[0065] The input of the basic segmentation module is an image of size H×W×3. First, the image is segmented into blocks of size 4×4. These blocks are then passed as input to the multi-layer Transformer encoder to obtain multi-layer features of the original image size {1 / 4, 1 / 8, 1 / 16, 1 / 32}, where i∈{1,2,3,4} and the number of channels C i+1 Greater than C i Next, these multi-layer features are fed into the decoder part.

[0066] First, the top-level features are input into the pyramid pooling module (PPM) to obtain the pooled features. Then, the features of the other layers are fused with the pooled features through upsampling and convolution operations to obtain a size of Finally, through a segmentation convolution head, the segmentation high-dimensional features are mapped to a size of The segmentation result of cls Indicates the number of segmentation types. The schematic diagram of this module is as follows Figure 2 As shown in the figure, for the input image, the overlapping blocks are first embedded to divide the image into multiple smaller patches, and then multi-level features of different scales are obtained through different transformation blocks. The multi-level features are obtained through PPM and decoder. Segmentation diagram.

[0067] Step 2: Construct a superpixel-guided image consistency contrast learning module:

[0068] The purpose of constructing a superpixel-guided consistent contrastive learning module is to use the segmentation features obtained by the basic segmentation module through the mapping head for the calculation of contrastive learning loss. By using the idea of ​​contrastive learning, the segmentation features of the same region are brought closer and the differences of the segmentation features of different regions are enhanced to achieve refined segmentation.

[0069] Input an image of size H×W×3. After being processed by the basic segmentation module, the algorithm obtains a The segmentation high-dimensional features. One branch inputs the feature into the segmentation head and generates a size of The other branch inputs the feature into the feature mapping head and maps it into a segmentation map of size characteristics, and conduct L 2 Regularization. This feature will be used in the subsequent calculation of contrastive learning loss. It should be noted that the feature map head is only used in the training phase and will be removed in the prediction phase, so it will not cause any changes to the original segmentation network and will not increase additional computational overhead. The structure of this module is as follows Figure 3 For a batch of training data, first pass through the skeleton network of the basic segmentation module to obtain The segmentation head maps the segmentation high-dimensional features into The mapping head maps the segmented high-dimensional features into The feature map is further used to calculate the pixel-level contrastive learning loss, and the feature distance in the same area is shortened by the contrastive learning loss.

[0070] Pixel-level contrastive learning loss is a loss proposed by this module to achieve robust face segmentation and eliminate label noise. This loss achieves refined segmentation by bringing superpixel features in the same area closer. This can form tight clusters in the feature space and make the boundaries between different categories clearer. Pixel-level contrastive learning loss is used to compare a pixel i with its corresponding semantic label. Positive samples belong to the category The negative samples are pixels belonging to other categories. The pixel-level contrast loss is defined as:

[0071]

[0072] Where P i and N i Represent the embedding set of positive and negative samples of pixel i respectively. Moreover, positive and negative samples and anchor point i are not limited to come from the same image. This pixel contrast-based loss learns an embedding space by pulling pixel samples of the same category closer and pushing samples of different categories apart.

[0073] Step 3: Construct a local style-guided representational consistency contrast learning module:

[0074] Since the basic segmentation module and the superpixel-guided appearance consistency contrast learning module have certain limitations and cannot fully meet the needs of fine-grained segmentation, the local style-guided appearance consistency contrast learning module makes up for these shortcomings. This module divides the input image into blocks and extracts the style features of each block. At the same time, it divides the segmentation features into blocks and uses similarity constraints to maintain the consistency of the style block and the segmentation feature block. This constraint helps to improve the accuracy and delicacy of segmentation.

[0075] The structure of this module is as follows Figure 4 As shown, it includes a segmentation feature extraction skeleton, a style feature extraction network and an inter-block feature similarity calculation module. The same feature extraction method as the basic segmentation module is used, and the feature mapping head is used to obtain the image segmentation features and perform block operations on the features. The style feature extraction network extracts style features from the image divided by blocks to calculate the style similarity between blocks. The inter-block feature similarity calculation module obtains the segmentation feature similarity matrix and the style feature similarity matrix between image blocks by calculating the Gram matrix. The Gram matrix obtains the correlation between different blocks by performing the inner product of the blocks. Afterwards, the style feature similarity matrix is ​​used to constrain the distribution of the segmentation feature similarity matrix.

[0076] The specific process is as follows: an image of size H×W×3 is input. After being processed by the basic segmentation module, the algorithm obtains a Then, this feature is processed by the feature mapping head to obtain a high-dimensional feature of size The algorithm performs L segmentation on this feature to improve stability and training efficiency. 2Regularization. At the same time, the algorithm uses another branch to randomly crop the image to obtain H×W×3×P image blocks, where P represents the number of blocks. Then, the algorithm uses the pre-trained style feature extraction network to obtain the corresponding style feature blocks from these image blocks and obtain a feature representation of size H×W×C×P. Similar to the segmentation features, these style feature blocks also need to go through L 2 Regularization processing. Next, the algorithm matches the image segmentation feature blocks obtained in the segmentation stage with the style feature blocks. By calculating the Gram matrix, the inter-block similarity matrix is ​​obtained, with a size of P×P×C. Finally, the algorithm constrains the consistency of the two similarity matrices through the loss function.

[0077] Inter-block similarity loss is a loss proposed by this module to achieve robust face segmentation and eliminate image label noise. This loss uses the consistency of style between blocks in the same segmentation area to align the style feature block with the segmentation feature block to achieve inter-block similarity constrained face segmentation. This loss is achieved by spatially segmenting the image at scale s and dividing it into M blocks. Right now Define the scale s as 32, which can better obtain style features and is more suitable for matching segmentation patches and style patches. Then, calculate the Gram matrix from the segmentation patches and style patches. The Gram matrix can obtain the correlation between patch features. After that, minimize the L2 distance between the segmentation matrix and the style matrix, as shown below:

[0078]

[0079] Among them, φ seg ((·)) represents the activation map after the segmentation skeleton network feature extraction and activation by the RelU layer. φ vgg ((·)) represents the style activation map after style feature extraction and activation by the ReLU layer. Represents the inter-block Gram matrix, which can capture the negative correlation of channel scales. Based on the block style inter-block similarity loss, the L2 distance of the Gram matrix between the style block and the segmentation feature block is calculated to measure the segmentation consistency of the local pattern. This loss function can capture the details and style type of the local pattern, making the segmentation result more refined and reasonable.

[0080] Step 4: Experiment:

[0081] 4.1 Experimental Setup

[0082] Datasets: In all experiments on image segmentation methods based on consistent contrast learning, three types of datasets are used, including classic image segmentation datasets, face datasets, and medical imaging datasets. For classic image segmentation datasets, fair comparisons are made based on the published training sets, validation sets, and test sets. For segmentation datasets without standard divisions, the experiment will be divided into training sets and test sets at a ratio of 70% and 30%.

[0083] Implementation details: In the network model setting, the pre-trained SegFormer is used as the basic segmentation module, and the experiment uses the AdamW algorithm as the optimizer. AdamW is an improvement of the Adam algorithm, which has been optimized for weight decay. The AdamW optimizer can better handle the weight decay problem and improve the performance and stability of the model during training. The parameters of the optimizer in this experiment are: β 1 =0.9,β 2 =0.999. The learning rate adjustment strategy is the polynomial learning (Poly LearningRate) strategy, combined with the linear warm-up strategy (Linear Warm Up). The formula for the learning rate adjustment strategy is as follows:

[0084]

[0085] The former is the warm-up phase formula, and the latter is the learning rate update phase formula. Among them, warmup_iters is the total number of rounds in the warm-up phase, warmup_ratio is the ratio of the initial learning rate in the warm-up phase to the base learning rate, base_lr is the base learning rate, cur_iter is the number of iteration steps in the current round, and power is the exponent of the polynomial learning rate. The parameters of the learning rate update strategy are set to: warmup_iters = 1500, warmup_ratio = 1e-6, power = 1.0. The model was trained for 160,000 iterations, the learning rate was 3e-4, and the batch size was set to 8. All experiments for this model were run on an Ubuntu computer and trained using an RTX3090 GPU.

[0086] Loss function: In image segmentation tasks, cross-entropy loss (CE Loss) is a commonly used loss function, which usually treats each pixel as an independent sample and divides it into different categories. For each pixel, the model generates a probability distribution indicating the probability that the pixel belongs to each category K. The true segmentation label marks each pixel as a specific category. The cross-entropy loss measures the prediction accuracy of the model by comparing the difference between the predicted probability distribution of the model and the true label. Specifically, for each pixel, the cross-entropy loss calculates the cross entropy between the predicted probability distribution p and the true label y, and then sums or averages the cross entropy of all pixels N. The corresponding formula is as follows:

[0087]

[0088] Finally, the cross entropy loss is combined with the pixel-level contrastive learning loss and the inter-block similarity loss to train the entire segmentation model. The final loss function is:

[0089]

[0090] Among them, λ 1 ,λ 2 is the weight parameter. In this experiment, 1 Set to 094, λ 2 Set to 092. The goal of this algorithm is to iteratively train the segmentation model so that minimize.

[0091] Evaluation indicators: This experiment uses commonly used evaluation indicators for semantic segmentation to measure the quality of segmented image generation, including mIoU and Dice Coefficient (F1-score). The experiment uses two evaluation indicators in combination.

[0092] Mean Intersection over Union (mIoU): mIoU is a commonly used evaluation metric for image semantic segmentation tasks, which is used to measure the model's ability to accurately segment objects of different categories at the pixel level. mIoU evaluates the performance of the model by calculating the intersection over Union (IoU) between the predicted results and the true labels. For each category, IoU is defined as the intersection area of ​​the predicted results and the true label divided by their union area. mIoU is the average of the IoUs of all categories. The calculation formula for IoU is as follows. mIoU is the average of the IoUs of multiple categories:

[0093]

[0094] Among them, TP(class) indicates the number of pixels correctly predicted as this category, FP(class) indicates the number of pixels incorrectly predicted as this category, and TN(class) indicates the number of pixels of this category that are missed in the prediction. The value range of mIoU is between 0 and 1. The higher the value, the higher the degree of consistency between the segmentation result of the model and the true label.

[0095] Dice Coefficient (also known as F1-score): F1-score is a commonly used evaluation metric used to measure the similarity or overlap between two sets. In image segmentation tasks, it is often used to evaluate the degree of consistency between the model's segmentation results and the true label. The calculation formula of Dice is as follows:

[0096]

[0097] The value range of Dice Coefficient is between 0 and 1. The closer the value is to 1, the higher the overlap between the model's segmentation result and the true label, that is, the segmentation result is more similar to the true label. In image segmentation tasks, F1-score can be used to measure the model's ability to accurately segment objects of different categories. It takes into account the overlap between the predicted result and the true label, so it is particularly suitable for segmentation tasks with blurred boundaries or overlapping objects.

[0098] 492 face dataset experiment

[0099] Quantitative experimental analysis: To verify the performance of the image segmentation method proposed in this paper, experimental analysis is conducted on two public face datasets, LaPa and CelebAMask-HQ. Training, testing, and evaluation are performed according to the experimental settings mentioned above, and the experimental indicators are compared with various advanced methods, including UNet, DANet, CE2P, EHANet, BiSeNetv2, SegNeXt, and Segformer. The evaluation indicators used include mIoU and F1-score.

[0100] Among them, Segformer * It is the basic segmentation module of this algorithm, and it shows advantages in multiple face parts. ACCL is a refined segmentation method proposed by this institute. Because the dataset is annotated as coarse-grained, this method achieves fine-grained segmentation, so it is reasonable that the indicators may be reduced.

[0101] As can be seen from Table 1, the basic segmentation algorithm of the present invention shows outstanding advantages. The two indicators obtained by verification in various data sets have achieved high values, which are better than other advanced image segmentation methods, proving its high precision and accuracy in face segmentation.

[0102] Table 1 IoU data evaluation of CelebAMask-HQ dataset

[0103]

[0104]

[0105] Quantitative experimental analysis: In order to verify the performance of the image segmentation method based on consistent contrast learning proposed in this paper, the face image segmentation results of the comparison model are mainly compared. By comparing the segmentation results generated by various advanced segmentation models, it can be proved that the image segmentation method based on consistent contrast learning can better segment face photos. The segmentation results on the CelebAMask-HQ dataset are shown in Figure 2. Figure 5 As shown in the figure. (a) Input image, (b) Segmentation label, (c) Segmentation result of SegNeXt, (d) Segmentation result of Segformer, (e) Segmentation result of basic segmentation module, (f) Result of ACCL method of the present invention. ACCL refined segmentation method can achieve refined segmentation for refined areas such as necklaces and hair while achieving high-precision segmentation, and can eliminate the irrationality of some dataset annotations.

[0106] Ablation experiment: It is mainly to optimize the precision and rationality of face segmentation by using the superpixel-guided image consistency contrast learning module and the local style-guided image consistency contrast learning module. The basic segmentation module method is used as the basic control group. As shown in Table 2, Segformer * It is the basic segmentation model of the present invention, which shows quite good performance in the recognition of various parts. * +ACCL sup The method is a superpixel-guided appearance consistency contrast learning method. Compared with the baseline model, this method has a slight decline in the recognition of most parts, but still maintains high segmentation indicators. * +ACCL sty ACCL is a local style guided representation consistency contrast learning method, which has achieved significant improvements in the recognition of multiple parts. In particular, the improvement in the recognition of the "hat" and "neck" parts is particularly significant. sty It can capture and identify the overall parts with block style very well. In addition, ACCL sty The method also showed certain advantages in the recognition of "left ear", "right ear" and "mouth", proving that it has strong recognition ability for small parts with strong style characteristics in face photos.

[0107] Table 2 Ablation experiments on CelebAMask-HQ

[0108]

[0109] The present invention proposes an image fine segmentation method based on appearance consistent contrastive learning (ACCL), and realizes image fine segmentation by using style consistency constraint and contrastive learning. The local semantic features are aligned with the style features, so as to learn the refined face image segmentation results based on coarse-grained labeled data. The contrastive learning of the label domain is introduced, and the superpixel feature points of the input image are extracted by the model, and the refined segmentation effect is improved by constraining the distance between superpixel points with the same label. The input image is divided into blocks, and the block style features are obtained by using the style feature extraction model, and the refined segmentation is realized by constraining the similarity between the segmentation feature blocks to be consistent with the similarity between the style feature blocks. It is used to solve the existing segmentation algorithm based on deep learning, because the labels of most data sets are coarse-grained or even have no real labels, which makes the model and training unstable, resulting in unclear recognition of the category subject, imprecise edge segmentation and low segmentation accuracy.

Claims

1. A method for fine-grained image segmentation based on consistent contrast learning, characterized in that: The steps include: Step 1: Build a basic segmentation module and pass the image through the basic segmentation module to obtain a feature map; Step 2: construct a superpixel-guided image consistency contrast learning module, and use the feature map as the input of the superpixel-guided image consistency contrast learning module; Step 3: construct a local style guided representation consistency contrast learning module; use the feature map as the input of the local style guided representation consistency contrast learning module; perform inter-block similarity loss calculation and pixel-level contrast loss calculation in the two modules; Step 4: Combine the pixel-level contrastive learning loss and the inter-patch similarity loss for joint training to minimize the loss.

2. The image segmentation method based on consistent contrast learning according to claim 1, characterized in that: The step 1 is specifically as follows: The basic segmentation module adopts an encoder-decoder structure, with SegFormer as the basic segmentation module. The encoder adopts the Transformer architecture and the decoder adopts the Uper structure. The encoder is used to extract the feature representation of the image, and the decoder is used to map these features back to the pixel-level semantic segmentation results. The encoder introduces the self-attention mechanism of the Transformer module. The self-attention mechanism enables the model to model the relationship between pixels globally, capture contextual information and improve the accuracy of semantic segmentation. The decoder is used to receive the multi-layer feature information of the encoder and input it into the Pyramid Pooling Module (PPM); The structure of the PPM module is used to aggregate context information of different scales to obtain global context information.

3. The image segmentation method based on consistent contrast learning according to claim 2, characterized in that: The specific operation of the PPM module is as follows: through convolution and upsampling operations, multiple layers of features are fused and features suitable for segmentation are generated. The results obtained by PPM are mapped to the features of the target number of channels through the segmentation head to obtain the segmentation map and label image, and the loss function is calculated.

4. The image segmentation method based on consistent contrast learning according to claim 2, characterized in that: The input of the basic segmentation module is a picture of size H×W×3; First, the image is divided into blocks of size 4×4, and the blocks are passed as input to the multi-layer Transformer encoder to obtain multi-layer features of the original image size {1 / 4, 1 / 8, 1 / 16, 1 / 32}, where i∈{1,2,3,4} and C i+1 Greater than C i ; Next, the multi-layer features are input into the decoder part; the highest layer features are input into the pyramid pooling module (PPM) to obtain the pooled features; Through upsampling and convolution operations, the features of other layers are fused with the features after pooling to obtain a size of Segmentation of high-dimensional features; Finally, through a segmentation convolution head, the segmentation high-dimensional features are mapped to a size of The segmentation result of cls Indicates the number of segmentation types.

5. The image segmentation method based on consistent contrast learning according to claim 4 is characterized in that: The step 2 is specifically as follows: Input an image of size H×W×3, and after processing by the basic segmentation module, obtain an image of size Segmentation of high-dimensional features; Among them, one branch inputs the segmentation high-dimensional features into the segmentation head to generate a size of The segmentation map output; The other branch inputs the segmented high-dimensional features into the feature mapping head and maps them into a size of The feature is taken and L2 regularized. This feature will be used for the calculation of contrastive learning loss later. L2 Regularization: x i represents the i-th element of vector x, and y represents the normalized vector.

6. The image segmentation method based on consistent contrast learning according to claim 5, characterized in that: Will The feature input superpixel guided appearance consistency contrast learning module calculates the pixel-level contrast learning loss. The pixel-level contrast learning loss is for a pixel i and its corresponding semantic label. Positive samples belong to the category Other pixels of , negative samples are pixels belonging to other categories, and the pixel-level contrast loss is defined as: Where P i and N i Represent the embedding sets of positive and negative samples of pixel i respectively, and the positive and negative samples and anchor point i are not limited to come from the same image.

7. The image segmentation method based on consistent contrast learning according to claim 1, characterized in that: The step 3 is specifically as follows: The local style guided representation consistency contrast learning module includes a segmentation feature extraction skeleton, a style feature extraction network, and an inter-block feature similarity calculation module; The segmentation feature extraction skeleton adopts the same feature extraction method as the basic segmentation module, and uses the feature mapping head to obtain image segmentation features and perform block operations on the features; The style feature extraction network extracts style features from the image divided into blocks to calculate the style similarity between blocks; The inter-block feature similarity calculation module obtains the segmentation feature similarity matrix and style feature similarity matrix between image blocks by calculating the Gram matrix. The Gram matrix obtains the correlation between different blocks by performing the inner product of the blocks. Then, the style feature similarity matrix is ​​used to constrain the distribution of the segmentation feature similarity matrix.

8. The image segmentation method based on consistent contrast learning according to claim 7, characterized in that: The specific process is as follows: an image of size H×W×3 is input. After being processed by the basic segmentation module, the algorithm obtains a Segmentation of high-dimensional features; Then, the segmented high-dimensional features are processed by the feature mapping head to obtain a size of The image scale segmentation feature is obtained by L2 regularization and the segmentation feature is divided into blocks; L2 Regularization: x i Represents the i-th element of vector x, y represents the normalized vector. At the same time, the algorithm crops the input image with a size of H×W×3 to obtain H×W×3×P image blocks, where P represents the number of blocks. Then, the corresponding style feature block is obtained from the image block to obtain a feature representation with a size of H×W×C×P. The style feature block is processed by L2 regularization, and the image segmentation feature block obtained in the segmentation stage is corresponded to the style feature block. By calculating the Gram matrix, the inter-block similarity matrix with a size of P×P×C is obtained, and the consistency of the two similarity matrices is constrained by the inter-block similarity loss.

9. The image segmentation method based on consistent contrast learning according to claim 8, characterized in that: The inter-block similarity loss aligns the style feature block with the segmentation feature block to achieve inter-block similarity constrained face segmentation. The inter-block similarity loss divides the image into M blocks by spatially segmenting it at scale s. Right now Then, the Gram matrix from the segmentation patch and the style patch is calculated, and then the L2 distance between the segmentation matrix and the style matrix is ​​minimized as follows: Among them, φ seg ((·)) represents the activation map after the segmentation skeleton network feature extraction and activation by the RelU layer, φ vgg ((·)) represents the style activation map after style feature extraction and activation by the ReLU layer. Represents the inter-block Gram matrix, capturing the negative correlation of channel scales. Based on the inter-block similarity loss of the style patch, the L2 distance of the Gram matrix between the style patch and the segmentation feature patch is calculated to measure the segmentation consistency effect of the local pattern.

10. The image segmentation method based on consistent contrast learning according to claim 1, characterized in that: In step 4, the pixel-level contrast learning loss and the inter-block similarity loss are combined, and the most basic cross entropy loss in the segmentation task is added to train the entire segmentation model, jointly guiding the optimization of the model parameters.

Images