Image refinement segmentation method based on representation consistent contrastive learning

Abstract

The image fine segmentation method based on appearance consistent contrast learning comprises the following steps: step 1: constructing a basic segmentation module, and obtaining a feature map after the picture passes through the basic segmentation module; step 2: constructing an appearance consistent contrast learning module guided by superpixels, and taking the feature map as the input of the appearance consistent contrast learning module guided by superpixels; step 3: constructing a local style guided appearance consistent contrast learning module; taking the feature map as the input of the local style guided appearance consistent contrast learning module; block similarity loss calculation and pixel-level contrast loss calculation are performed in the two modules; step 4: the pixel-level contrast learning loss and the block similarity loss are combined for joint training, and the loss is minimized. The present application effectively reduces the influence of false labels on the segmentation model, realizes fine segmentation results, and can perform more fine segmentation on images.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, specifically relating to a refined image segmentation method based on representational consistency contrastive learning. Background Technology

[0002] Image segmentation, a fundamental computer vision task aiming to assign a semantic category to each pixel, has garnered significant attention in recent years. Semantic segmentation typically employs deep learning methods, starting with pioneering fully convolutional networks (FCNs) and the Deep Lab series; CNN-based models have become the primary design choice for dense prediction tasks. Following the tremendous success of natural language processing, interest in introducing Transformers into visual tasks has surged, with Transformer-based models such as Segformer and SETR demonstrating immense potential. Semantic segmentation models have undergone significant changes in network architecture, and the rapid development of deep learning and natural language processing has greatly pushed the performance boundaries of semantic segmentation.

[0003] In the era of deep learning, the architecture of segmentation models can be broadly divided into two parts: the encoder and the decoder. The encoder stage primarily performs convolution and pooling operations to extract high-dimensional features containing semantic information. Encoders typically employ popular classification networks, such as VGG, ResNet, and DenseNet. Self-attention mechanisms are also commonly used in encoder networks. These mechanisms represent the correlations between different regions (pixels) or channels in the feature map. They calculate a weighted sum of pairwise correlations for a single sample across all locations to update the features at each location.

[0004] In the decoder stage, operations such as interpolation, deconvolution, or unpooling are typically used to map the multi-layered high-dimensional feature vectors extracted by the encoder back to the original image. Decoders are usually used in conjunction with encoders to achieve better results. Different types of decoders exist for different objectives, including those that implement multi-scale receptive fields, collect multi-scale semantics, expand the receptive field, enhance edge features, and capture global context.

[0005] Contrastive learning is an unsupervised learning method used to learn representations of data without label information. Its goal is to learn discriminative representations by comparing similarities and differences. In contrastive learning, data is organized into positive and negative pairs. Positive pairs contain similar data samples, while negative pairs contain dissimilar data samples. By comparing the differences between positive and negative pairs, the model can learn feature representations of the data, making similar samples closer together in the representation space and dissimilar samples more dispersed.

[0006] While classic segmentation networks have achieved impressive performance on classic datasets, existing datasets generally employ coarse-grained semantic labeling, resulting in noisy labels in the training data and limiting the model's effectiveness. Furthermore, some facial sketches and portraits lack true labels, leading to errors in the generated pseudo-labels at boundaries, further limiting the model's capabilities. Therefore, designing and implementing efficient and refined image segmentation methods to achieve fine-grained segmentation of images, facial images, and medical images, while reducing the segmentation error rate, is imperative.

[0007] The patent application CN118053051A, entitled "A Hyperspectral Remote Sensing Image Classification Method Based on Superpixel Self-Attention Mechanism," discloses a hyperspectral remote sensing image classification method based on a superpixel self-attention mechanism. The shortcoming of this method is that it does not delve into the relationships between superpixel points when extracting superpixel feature points. Simply using superpixels cannot enable the network to learn more representative features. However, by using superpixels and contrastive learning, by bringing superpixel features in the same region closer together, it is possible to form dense clusters in the feature space and make the boundaries between different categories clearer.

[0008] SETR, represented by Transformer, uses ViT as its backbone and combines multiple CNN decoders to expand feature resolution. While performing well, it has limitations: ViT outputs single-scale, low-resolution features instead of multi-scale features, and its computational cost is high on large images. SegNeXt, represented by CNNs, captures multi-scale features using convolutional attention operations and employs lightweight decoders to obtain multi-scale context from local to global, achieving adaptability in spatial and channel dimensions and information aggregation from low to high levels. Although SegNeXt reduces the number of parameters to some extent, it typically requires a large amount of data. While classic segmentation networks have achieved impressive performance on classic datasets, existing datasets generally use coarse-grained semantic labeling, resulting in noisy labels in the training data, limiting the model's effectiveness. Furthermore, some artistic portraits of faces lack true labels, and the generated pseudo-labels contain errors, further limiting the model's capabilities. Summary of the Invention

[0009] To overcome the shortcomings of existing technologies, this invention aims to provide a refined image segmentation method based on representational consistency contrastive learning. This method learns a refined image segmentation model from coarse-grained labeled data. By utilizing contrastive learning to constrain the distance between superpixels with the same label, it narrows the segmentation features of similar regions, enhances the differences in segmentation features between different regions, and effectively reduces the impact of mislabels on the segmentation model, achieving refined segmentation results. Furthermore, by utilizing style consistency to constrain similar segmentation regions to have the same style, it minimizes misclassification and enables more precise image segmentation.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A refined image segmentation method based on representational consistency contrastive learning includes the following steps;

[0012] Step 1: Construct a basic segmentation module to obtain feature maps after passing the image through the basic segmentation module;

[0013] Step 2: Construct a superpixel-guided representation consistency contrastive learning module, using the feature map as the input to the superpixel-guided representation consistency contrastive learning module;

[0014] Step 3: Construct a local style-guided representational consistency contrastive learning module; use the feature map as input to the local style-guided representational consistency contrastive learning module;

[0015] Inter-block similarity loss calculation and pixel-level contrast loss calculation are performed in two modules, which helps to improve the accuracy and fineness of segmentation.

[0016] Step 4: Combine the pixel-level contrastive learning loss and the inter-block similarity loss for joint training to minimize the loss.

[0017] Step 1 specifically involves:

[0018] The basic segmentation module adopts an encoder-decoder structure, using SegFormer as the basic segmentation module. The encoder adopts a Transformer architecture, and the decoder adopts an Upper structure. The encoder is used to extract the feature representation of the image, and the decoder is used to map these features back to pixel-level semantic segmentation results.

[0019] Furthermore, the encoder introduces the self-attention mechanism of the Transformer module, which enables the model to model the relationship between pixels globally, thereby better capturing contextual information and improving the accuracy of semantic segmentation. The decoder is used to receive multi-layer feature information from the encoder and input it into the Pyramid Pooling Module (PPM).

[0020] The structure of the PPM module is used to aggregate contextual information at different scales to obtain global contextual information; this effectively increases the receptive field, enabling better subsequent segmentation.

[0021] The specific operation is as follows: through convolution and upsampling operations, multi-layer features are fused and features suitable for segmentation are generated. Finally, the results obtained by PPM are mapped to the features of the target number of channels through the segmentation head, thereby obtaining the segmentation map and label image, and the loss function is calculated.

[0022] Furthermore, the input to the basic segmentation module is an image with dimensions H×W×3;

[0023] First, the image is segmented into 4×4 blocks. These blocks are then fed into a multi-layer Transformer encoder to obtain multi-layer features of the original image size {1 / 4, 1 / 8, 1 / 16, 1 / 32}, where i∈{1,2,3,4} and the number of channels C. i+1 Greater than C i , i represents the corresponding different image scale features. Next, the multi-layer features are input into the decoder part; the highest layer features are input into the pyramid pooling module (PPM) to obtain the pooled features.

[0024] By upsampling and convolution operations, the features from other layers are fused with the pooled features to obtain a feature of size [size missing]. High-dimensional segmentation features;

[0025] Finally, a segmentation convolution head is used to map the high-dimensional segmentation features to a size of [size missing]. The segmentation map result, where N cls Indicates the number of types of segmentation.

[0026] Step 2 specifically involves:

[0027] Given an input image of size H×W×3, after processing by the basic segmentation module, the algorithm obtains an image of size H×W×3. High-dimensional segmentation features;

[0028] One branch inputs the high-dimensional segmentation features into the segmentation head, generating a segmentation head with a size of [size missing]. Output the segmentation map;

[0029] Another branch inputs the segmented high-dimensional features into the feature mapping head, and maps them into a feature map of size through a convolution operation. The features are extracted and L2 regularized, and these features will be used to calculate the contrastive learning loss later.

[0030] L2 regularization:

[0031]

[0032] x i Let represent the i-th element of vector x, and y represent the normalized vector;

[0033] Will The feature input is a superpixel-guided representational consistency contrastive learning module, which calculates the pixel-level contrastive learning loss. The pixel-level contrastive learning loss is calculated for a pixel i and its corresponding semantic label. Positive samples belong to the category Other pixels, negative samples are pixels belonging to other categories, and the pixel-level contrast loss is defined as follows:

[0034]

[0035] Where P i and N i Let i represent the embedding sets of positive and negative samples of pixel i, respectively, and the positive and negative samples and anchor i are not limited to coming from the same image.

[0036] Step 3 specifically involves:

[0037] The local style-guided representational consistency contrastive learning module includes a segmentation feature extraction skeleton, a style feature extraction network, and an inter-block feature similarity calculation module;

[0038] The segmentation feature extraction skeleton uses the same feature extraction method as the basic segmentation module, and uses a feature mapping head to obtain image segmentation features, and performs block operations on the image segmentation features;

[0039] The style feature extraction network extracts style features from the image divided into blocks to calculate the style similarity between blocks. The inter-block feature similarity calculation module calculates the Gram matrix to obtain the segmentation feature similarity matrix and style feature similarity matrix between image blocks. The Gram matrix obtains the correlation between different blocks by performing an inner product on the blocks. Then, the style feature similarity matrix is ​​used to constrain the distribution of the segmentation feature similarity matrix.

[0040] The specific process is as follows: An image of size H×W×3 is input. After processing by the basic segmentation module, the algorithm obtains an image of size H×W×3. High-dimensional segmentation features;

[0041] Then, the segmented high-dimensional features are processed by a feature mapping head to obtain a feature of size . The image scale segmentation features are obtained, L2 regularization is applied to these features, and the segmentation features are divided into blocks.

[0042] L2 regularization:

[0043]

[0044] x i Let represent the i-th element of vector x, and y represent the normalized vector. Simultaneously, the algorithm crops the input image to size H×W×3, obtaining H×W×3×P image blocks, where P represents the number of blocks. Next, the corresponding style feature blocks are extracted from these image blocks, resulting in a feature representation of size H×W×C×P. Similar to the segmentation features, the style feature blocks also require L2 regularization. Then, the algorithm maps the image segmentation feature blocks obtained in the segmentation stage to the style feature blocks, calculating the Gram matrix to obtain the inter-block similarity matrix (segmentation feature similarity matrix and style feature similarity matrix), with a size of P×P×C. Finally, the algorithm constrains the consistency of these two similarity matrices through inter-block similarity loss.

[0045] The inter-block similarity loss aligns style feature blocks with segmentation feature blocks, achieving face refinement segmentation constrained by inter-block similarity. This loss involves spatially segmenting the image at scale 's', defined as 32, a size that effectively captures style features and is well-suited for matching segmentation and style patches. The image is divided into M patches. Right now Then, the Gram matrix from the segmentation tile and the style tile is calculated, and then the L2 distance between the segmentation matrix and the style matrix is ​​minimized, as shown below:

[0046]

[0047] Where, φ seg ((·)) represents the activation map after feature extraction by the segmentation skeleton network and activation by the ReLU layer, φ vgg ((·)) represents the style activation map after style feature extraction, which is also activated by the ReLU layer. The inter-block Gram matrix is ​​used to capture the negative correlation of channel scale. Based on the inter-block similarity loss of tile style, the L2 distance of the Gram matrix between style tiles and segmentation feature tiles is calculated to measure the segmentation consistency of local patterns.

[0048] In step 4, the pixel-level contrastive learning loss and the inter-block similarity loss are combined, and the most basic cross-entropy loss in the segmentation task is added to train the entire segmentation model, which together guides the optimization of the model's parameters.

[0049] The beneficial effects of this invention are:

[0050] This invention proposes a novel image refinement segmentation method based on representational consistency contrastive learning. Utilizing a self-attention mechanism and an efficient feature fusion method, it achieves state-of-the-art results in face image segmentation, effectively addressing the problem of low segmentation accuracy caused by training models on coarse-grained labels. Furthermore, this method employs a superpixel-guided representational consistency contrastive learning module to construct an embedding space that brings similar categories closer together and distancing different categories. A local style-guided representational consistency contrastive learning module captures the details and style types of local patterns. Combining these two modules yields more accurate and detailed segmentation results.

[0051] (1) This invention proposes a refined face image segmentation strategy based on local style-guided representation consistency contrastive learning, which can perform more refined segmentation of face images and effectively reduce the impact of mislabels on the segmentation model. Moreover, this strategy uses a loss function and branch model to adjust the segmentation model without increasing the computational overhead of the inference stage.

[0052] (2) This invention proposes a superpixel-guided representation consistency contrastive learning strategy for fine-grained face image segmentation, which can also perform more refined segmentation of face images and effectively reduce the impact of mislabels on the segmentation model. Moreover, this strategy uses a loss function and branch model to adjust the segmentation model without increasing the computational overhead of the inference stage. Attached Figure Description

[0053] Figure 1 This is a diagram of the pyramid pooling module of the present invention.

[0054] Figure 2 This is a diagram of the basic segmentation module of the present invention.

[0055] Figure 3 This is a diagram of the superpixel-guided representation consistency comparison learning module of the present invention.

[0056] Figure 4 This is a diagram of the representational consistency contrast learning module guided by local style in this invention.

[0057] Figure 5 This is an example image showing the segmentation results of this invention on the CelebAMask dataset.

[0058] Figure 6 This is a flowchart of the present invention. Detailed Implementation

[0059] The present invention will now be described in further detail with reference to the accompanying drawings.

[0060] like Figure 6 As shown, the image refinement segmentation method based on representation consistency contrastive learning includes:

[0061] Step 1: Construct the basic segmentation module:

[0062] The basic segmentation module employs an encoder-decoder structure, with most encoders consisting of stacked N feature extraction modules. The input image X passes through each feature extraction module sequentially, reducing the image size and increasing the number of channels to obtain a segmentation feature map of size H×W×C. The basic segmentation module can be any segmentation network, such as CNN-based U-Net, SegNeXt, and ConvNeXt, or Transformer-based SegFormer and SETR. Most decoders acquire the multi-scale features generated by the encoder, fuse these features, and restore the resolution of the feature map through operations such as deconvolution or upsampling to generate the predicted segmentation result.

[0063] This invention uses SegFormer as the basic segmentation module, with a Transformer architecture for the encoder and an Upper architecture for the decoder. The encoder extracts feature representations from the image, while the decoder maps these features back to pixel-level semantic segmentation results. The encoder incorporates the self-attention mechanism of the Transformer module, enabling the model to model the relationships between pixels globally, thereby better capturing contextual information and improving the accuracy of semantic segmentation. The decoder receives multi-layer feature information from the encoder and inputs it into the Pyramid Pooling Module (PPM).

[0064] Figure 1 The structure of the PPM module is shown. Then, through convolution and upsampling operations, multi-layer features are fused to generate features suitable for segmentation. Finally, the features are mapped to the target number of channels using the segmentation head, resulting in a segmentation map and a label image, and the loss function is calculated.

[0065] The input to the basic segmentation module is an image of size H×W×3. First, the image is segmented into 4×4 blocks. Then, these blocks are fed as input into a multi-layer Transformer encoder to obtain multi-layer features of the original image size {1 / 4, 1 / 8, 1 / 16, 1 / 32}, where i∈{1,2,3,4} and the number of channels C. i+1 Greater than C i Next, these multi-layer features are input into the decoder part.

[0066] First, the highest-level features are input into the pyramid pooling module (PPM) to obtain pooled features. Then, the features from the other layers are fused with the pooled features through upsampling and convolution operations to obtain a feature of size [size missing]. The high-dimensional segmentation features are then mapped to a size of [size missing]. Finally, a segmentation convolutional head is used to map the high-dimensional segmentation features to a [size missing] dimensional [format missing]. The segmentation map result, where N cls This indicates the number of segmentation types. A schematic diagram of this module is shown below. Figure 2 As shown, for the input image, overlapping block embedding first divides the image into multiple smaller patches, then different transform blocks are used to obtain multi-level features at different scales. These multi-level features are then processed by PPM and a decoder. The segmentation diagram.

[0067] Step 2: Construct a superpixel-guided representational consistency contrastive learning module:

[0068] The superpixel-guided representational consistency contrastive learning module is constructed to use the segmentation features obtained from the basic segmentation module, after passing through a mapping head, for calculating the contrastive learning loss. By using the idea of ​​contrastive learning, segmentation features of similar regions are brought closer together, while the differences in segmentation features of different regions are enhanced, thereby achieving refined segmentation.

[0069] Given an input image of size H×W×3, after processing by the basic segmentation module, the algorithm obtains an image of size H×W×3. The high-dimensional segmentation feature is used. One branch inputs this feature into the segmentation head, generating a segmentation head with a size of [size missing]. The segmentation map is output. Another branch inputs this feature into a feature mapping head, and maps it to a map of size [size missing] through a convolution operation. The feature map head is used to calculate the contrastive learning loss. It's important to note that the feature map head is only used during training and is removed during prediction, thus not altering the original segmentation network or incurring additional computational overhead. The structure of this module is as follows: Figure 3 As shown. For a batch of training data, it first passes through the backbone network of the basic segmentation module to obtain... The segmentation head maps the high-dimensional segmentation features to... The mapping head maps the segmentation high-dimensional features to The feature map is further used to calculate pixel-level contrastive learning loss, which is used to narrow the feature distance of the same region.

[0070] Pixel-level contrastive learning loss is a loss method proposed in this module to achieve robust face segmentation and eliminate label noise. This loss refines the segmentation by bringing superpixel features in the same region closer together. This allows for the formation of dense clusters in the feature space and makes the boundaries between different categories clearer. Pixel-level contrastive learning loss is applied to a pixel i and its corresponding semantic label. Positive samples belong to the category Other pixels are negative samples, which are pixels belonging to other categories. Pixel-level contrast loss is defined as follows:

[0071]

[0072] Where P i and N i Let represent the embedding sets of positive and negative samples for pixel i, respectively. Furthermore, the positive and negative samples and anchor point i are not necessarily from the same image. This pixel-contrast-based loss learns an embedding space by bringing pixel samples of the same class closer together and pushing samples of different classes apart.

[0073] Step 3: Construct a local style-guided representational consistency contrast learning module:

[0074] Since the basic segmentation module and the superpixel-guided representational consistency contrastive learning module have certain limitations and cannot fully meet the requirements of fine-grained segmentation, the local style-guided representational consistency contrastive learning module makes up for these shortcomings. This module divides the input image into blocks, extracts the style features of each block, and further divides the segmentation features into blocks, using similarity constraints to maintain the consistency between style blocks and segmentation feature blocks. This constraint helps improve the accuracy and fineness of segmentation.

[0075] The structure of this module is as follows: Figure 4 As shown, the system includes a segmentation feature extraction skeleton, a style feature extraction network, and an inter-block feature similarity calculation module. It employs the same feature extraction method as the basic segmentation module, using a feature mapping head to acquire image segmentation features and then dividing these features into blocks. The style feature extraction network extracts style features from the block-based image to calculate the style similarity between blocks. The inter-block feature similarity calculation module calculates the Gram matrix to obtain the segmentation feature similarity matrix and the style feature similarity matrix between image blocks. The Gram matrix is ​​used to obtain the correlation between different blocks by performing inner products on the blocks. Finally, the style feature similarity matrix is ​​used to constrain the distribution of the segmentation feature similarity matrix.

[0076] The specific process is as follows: An image of size H×W×3 is input. After processing by the basic segmentation module, the algorithm obtains an image of size H×W×3. The high-dimensional feature of the segmentation is then processed by a feature mapping head to obtain a feature of size . The algorithm first extracts image-scale segmentation features and then divides these features into blocks. To improve stability and training efficiency, L2 regularization is applied to these features. Simultaneously, another branch is used to randomly crop the image, resulting in H×W×3×P image blocks, where P represents the number of blocks. Next, the algorithm uses a pre-trained style feature extraction network to extract corresponding style feature blocks from these image blocks, obtaining feature representations of size H×W×C×P. Similar to the segmentation features, these style feature blocks also undergo L2 regularization. Next, the algorithm maps the image segmentation feature blocks obtained in the segmentation stage to the style feature blocks. By calculating the Gram matrix, a block similarity matrix of size P×P×C is obtained. Finally, the algorithm constrains the consistency of these two similarity matrices using a loss function.

[0077] Inter-block similarity loss is a loss method proposed in this module to achieve robust face segmentation and eliminate image label noise. This loss leverages the style consistency between blocks of the same segmentation region, aligning style feature blocks with segmentation feature blocks to achieve refined face segmentation constrained by inter-block similarity. This loss divides the image into M patches at scale *s*. Right now We define the scale *s* as 32, which is optimal for capturing style features and is also suitable for matching segmentation patches and style patches. Then, we compute the Gram matrix from the segmentation and style patches. The Gram matrix reveals the correlation between patch features. Next, we minimize the L2 distance between the segmentation and style matrices, as shown below:

[0078]

[0079] Where, φ seg ((·)) represents the activation map after feature extraction by the segmentation skeleton network and activation by the RelU layer. φ vgg ((·)) represents the style activation map after style feature extraction, which is also activated by the ReLU layer. The inter-block Gram matrix represents the segmentation consistency of a local pattern and can capture negative correlations at the channel scale. The tile-style inter-block similarity loss measures the segmentation consistency by calculating the L2 distance of the Gram matrix between style tiles and segmentation feature tiles. This loss function captures the details and style type of local patterns, resulting in more refined and reasonable segmentation results.

[0080] Step 4: Experiment:

[0081] 4.1 Experimental Setup

[0082] Datasets: All experiments using the image refinement segmentation method based on representational consistency contrastive learning utilized three types of datasets: classic image segmentation datasets, face datasets, and medical image datasets. For classic image segmentation datasets, fair comparisons were performed using the published training, validation, and test sets. For segmentation datasets without a standardized partitioning, the experiments were divided into training and test sets at a ratio of 70% and 30%, respectively.

[0083] Implementation Details: In the network model setup, a pre-trained SegFormer is used as the basic segmentation module, and the AdamW algorithm is used as the optimizer. AdamW is an improvement on the Adam algorithm, with optimizations for weight decay. The AdamW optimizer can better handle the weight decay problem, improving the model's performance and stability during training. In this experiment, the optimizer parameters are: β1 = 0.9, β2 = 0.999. The learning rate adjustment strategy is a polynomial learning rate strategy combined with a linear warm-up strategy. The formula for the learning rate adjustment strategy is as follows:

[0084]

[0085] The former is the warm-up phase formula, and the latter is the learning rate update phase formula. Here, `warmup_iters` is the total number of warm-up phase rounds, `warmup_ratio` is the ratio of the initial learning rate to the base learning rate, `base_lr` is the base learning rate, `cur_iter` is the number of iterations in the current round, and `power` is the exponent of the multinomial learning rate. The parameters of the learning rate update strategy are set as follows: `warmup_iters = 1500`, `warmup_ratio = 1e-6`, `power = 1.0`. The model is trained for 160,000 iterations with a learning rate of 3e-4 and a batch size of 8. All experiments of this model were run on an Ubuntu computer and trained using an RTX 3090 GPU.

[0086] Loss Function: In image segmentation tasks, Cross-Entropy Loss (CE Loss) is a commonly used loss function. It typically treats each pixel as an independent sample and categorizes it into different classes. For each pixel, the model generates a probability distribution representing the probability that the pixel belongs to each class K. The true segmentation label assigns each pixel to a specific class. Cross-Entropy Loss measures the model's prediction accuracy by comparing the difference between the model's predicted probability distribution and the true label. Specifically, for each pixel, Cross-Entropy Loss calculates the cross-entropy between the predicted probability distribution p and the true label y, and then sums or averages the cross-entropy of all pixels N. The corresponding formula is as follows:

[0087]

[0088] Finally, the cross-entropy loss, pixel-level contrastive learning loss, and inter-block similarity loss are combined to train the entire segmentation model. The final loss function is:

[0089]

[0090] Where λ1 and λ2 are weight parameters. In this experiment, λ1 is set to 0.94 and λ2 is set to 0.92. The goal of this algorithm is to train the segmentation model iteratively, so that... minimize.

[0091] Evaluation metrics: This experiment uses commonly used evaluation metrics for semantic segmentation to measure the quality of the segmented images generated. These include mIoU and Dice Coefficient (F1-score), and the experiment uses both metrics together.

[0092] Mean Intersection over Union (mIoU): mIoU is a commonly used evaluation metric for image semantic segmentation tasks, used to measure the model's ability to accurately segment objects of different categories at the pixel level. mIoU evaluates model performance by calculating the Intersection over Union (IoU) between the predicted results and the ground truth labels. For each category, IoU is defined as the area of ​​the intersection of the predicted results and the ground truth labels divided by the area of ​​their union. mIoU is the average IoU across all categories. The formula for calculating IoU is as follows, where mIoU is the average of the IoUs across multiple categories:

[0093]

[0094] Where TP(class) represents the number of pixels correctly predicted as belonging to that class, FP(class) represents the number of pixels incorrectly predicted as belonging to that class, and TN(class) represents the number of pixels of that class that were missed in the prediction. The value of mIoU ranges from 0 to 1, and a higher value indicates a higher degree of agreement between the model's segmentation results and the true labels.

[0095] Dice coefficient (also known as F1-score): The F1-score is a commonly used evaluation metric to measure the similarity or overlap between two sets. In image segmentation tasks, it is often used to evaluate how well the model's segmentation results match the ground truth labels. The formula for calculating Dice is as follows:

[0096]

[0097] The Dice Coefficient ranges from 0 to 1. A value closer to 1 indicates a higher degree of overlap between the model's segmentation result and the ground truth label, meaning the segmentation result is more similar to the ground truth label. In image segmentation tasks, the F1-score can be used to measure the model's ability to accurately segment different categories of objects. It takes into account the overlap between the predicted result and the ground truth label, making it particularly suitable for segmenting objects with blurred boundaries or overlapping areas.

[0098] 492 Face Dataset Experiment

[0099] Quantitative Experimental Analysis: To verify the performance of the image refinement segmentation method based on representational consistency contrastive learning proposed in this invention, experiments were conducted on two publicly available face datasets, LaPa and CelebAMask-HQ. Training, testing, and evaluation were performed according to the experimental setup described above, and the method was compared with various state-of-the-art methods on experimental metrics, including UNet, DANet, CE2P, EHANet, BiSeNetv2, SegNeXt, and Segformer. The evaluation metrics used included mIoU and F1-score.

[0100] Among them, Segformer * This is the foundational segmentation module of our algorithm, demonstrating its advantages in multiple face regions. ACCL is a refined segmentation method proposed by our institute. Because the dataset is labeled at a coarse-grained level, this method achieves fine-grained segmentation, so a slight reduction in performance metrics is reasonable.

[0101] As shown in Table 1, the basic segmentation algorithm of this invention exhibits outstanding advantages. Both metrics obtained from validation on various datasets achieved high values, outperforming other advanced image segmentation methods, demonstrating its high precision and accuracy in face segmentation.

[0102] Table 1. Evaluation of IoU data in the CelebAMask-HQ dataset.

[0103]

[0104]

[0105] Quantitative Experimental Analysis: To verify the performance of the image refinement segmentation method based on representational consistency contrastive learning proposed in this invention, the face image segmentation results of the model were mainly compared. By comparing the generated segmentation maps with various advanced segmentation models, it can be demonstrated that the image refinement segmentation method based on representational consistency contrastive learning can segment face photos well. The segmentation results on the CelebAMask-HQ dataset are as follows: Figure 5 As shown, (a) is the input image, (b) is the segmentation label, (c) is the segmentation result of SegNeXt, (d) is the segmentation result of Segformer, (e) is the segmentation result of the basic segmentation module, and (f) is the result of the ACCL method of this invention. The ACCL refinement segmentation method can achieve fine segmentation for detailed regions such as necklaces and hair while maintaining high-precision segmentation, and can also eliminate some unreasonableness in dataset labeling.

[0106] Ablation experiments: These experiments primarily aimed to optimize and improve the precision and accuracy of face segmentation using the superpixel-guided and local style-guided representational consistency contrastive learning modules. The basic segmentation module's method was used as a baseline control group. As shown in Table 2, Segformer... * This is the basic segmentation model of this invention, which demonstrates quite good performance in recognizing various parts. Segformer * +ACCL sup The method employs a superpixel-guided representation consistency contrastive learning approach. Compared to the baseline model, this method shows a slight decrease in recognition performance across most areas, but still maintains high segmentation metrics. (Segformer) * +ACCL sty This is a local style-guided representation consistency contrastive learning method that achieves significant improvements in the recognition of multiple body parts, particularly the "hat" and "neck" parts. Proof is provided by ACCL. sty It can effectively capture and identify overall sections with a blocky style. Furthermore, ACCL... sty The method also demonstrated certain advantages in recognizing features such as the left ear, right ear, and mouth. This proves that it also has a strong ability to recognize small features with strong stylistic characteristics in facial photographs.

[0107] Table 2 Ablation experiments on CelebAMask-HQ

[0108]

[0109] This invention proposes a refined image segmentation method based on Appearance Consistent Contrastive Learning (ACCL), which utilizes style consistency constraints and contrastive learning to achieve refined image segmentation. Local semantic features are aligned with style features, thereby learning refined face image segmentation results based on coarse-grained labeled data. Label-domain contrastive learning is introduced, using a model to extract superpixel feature points from the input image, and the refinement effect is improved by constraining the distance between superpixels with the same label. The input image is divided into blocks, and a style feature extraction model is used to obtain block style features. Refined segmentation is achieved by constraining the similarity between segmentation feature blocks to maintain consistency with the similarity between style feature blocks. This addresses the problems of existing deep learning-based segmentation algorithms, where most datasets have coarse-grained labels or even no true labels, leading to model and training instability, resulting in unclear subject identification, imprecise edge segmentation, and low segmentation accuracy.

Claims

1. An image fine-grained segmentation method based on representational consistency contrastive learning, characterized in that, Includes the following steps; Step 1: Construct a basic segmentation module to obtain feature maps after passing the image through the basic segmentation module; Step 2: Construct a superpixel-guided representation consistency contrastive learning module, using the feature map as the input to the superpixel-guided representation consistency contrastive learning module; Step 3: Construct a local style-guided representational consistency contrastive learning module; use the feature map as input to the local style-guided representational consistency contrastive learning module; calculate inter-block similarity loss and pixel-level contrastive learning loss in the two modules; Step 4: Combine the pixel-level contrastive learning loss and the inter-block similarity loss for joint training to minimize the loss; The superpixel-guided representational consistency contrastive learning module is specifically used to: take an image of size H×W×3 as input, process it through the basic segmentation module, and obtain an image of size H×W×3. High-dimensional segmentation features; One branch inputs the high-dimensional segmentation features into the segmentation head, generating a segmentation head with a size of [size missing]. Output the segmentation map; Another branch inputs the segmented high-dimensional features into the feature mapping head, and maps them into a feature map of size through a convolution operation. The features are extracted and L2 regularized. These features will be used to calculate the subsequent pixel-level contrastive learning loss. L2 regularization: x i Let represent the i-th element of vector x, and y represent the normalized vector; The local style-guided representational consistency contrast learning module is specifically used to: take an image of size H×W×3 as input, process it through the basic segmentation module, and obtain an image of size H×W×3. High-dimensional segmentation features; Then, the segmented high-dimensional features are processed by a feature mapping head to obtain a feature of size . The image scale segmentation features are obtained by performing L2 regularization on the features and then dividing the segmentation features into blocks to obtain image segmentation feature blocks. Another branch randomly crops the input image of size H×W×3 to obtain H×W×3×P image blocks, where P represents the number of blocks. Then, a pre-trained style feature extraction network is used to extract the corresponding style feature blocks from the image blocks, resulting in a feature representation of size H×W×C×P. The style feature blocks are then subjected to L2 regularization. The image segmentation feature blocks obtained in the segmentation stage are mapped to the style feature blocks. By calculating the Gram matrix, an inter-block similarity matrix of size P×P×C is obtained. The consistency of these two similarity matrices is constrained by the inter-block similarity loss.

2. The image fine-grained segmentation method based on representational consistency contrastive learning according to claim 1, characterized in that, Step 1 specifically involves: The basic segmentation module adopts an encoder-decoder structure, with SegFormer as the basic segmentation module. The encoder adopts a Transformer architecture, and the decoder adopts an Upper structure. The encoder is used to extract the feature representation of the image, and the decoder is used to map these features back to the pixel-level semantic segmentation results. The encoder introduces the self-attention mechanism of the Transformer module. The self-attention mechanism enables the model to model the relationship between pixels in a global scope, capture contextual information and improve the accuracy of semantic segmentation. The decoder is used to receive multi-layer feature information from the encoder and input it into the pyramid pooling module. The structure of the pyramid pooling module is used to aggregate context information at different scales, thereby obtaining global context information.

3. The image fine-grained segmentation method based on representational consistency contrastive learning according to claim 2, characterized in that, The pyramid pooling module operates as follows: through convolution and upsampling operations, multi-layer features are fused and features suitable for segmentation are generated. The results obtained from pyramid pooling are mapped to the features of the target number of channels through the segmentation head, thereby obtaining the segmentation map and label image, and the cross-entropy loss function is calculated.

4. The image fine-grained segmentation method based on representational consistency contrastive learning according to claim 2, characterized in that, The input to the basic segmentation module is an image with dimensions H×W×3; First, the image is segmented into 4×4 blocks. These blocks are then fed into a multi-layer Transformer encoder to obtain multi-layer features of the original image size {1 / 4, 1 / 8, 1 / 16, 1 / 32}, with C channels. i+1 Greater than C i , where i∈{1,2,3,4}, i represents the corresponding different image scale features; Next, the multi-layer features are input into the decoder section; the highest-layer features are input into the pyramid pooling module to obtain the pooled features. By upsampling and convolution operations, the features from other layers are fused with the pooled features to obtain a feature of size [size missing]. High-dimensional segmentation features; Finally, a segmentation convolution head is used to map the high-dimensional segmentation features to a size of [size missing]. The segmentation map results, where N cls Indicates the number of types of segmentation.

5. The image fine-grained segmentation method based on representational consistency contrastive learning according to claim 1, characterized in that, The pixel-level contrastive learning loss in step 3 is specifically: for a pixel i and its corresponding semantic label... Positive samples belong to the category Other pixels, negative samples are pixels belonging to other categories, and the pixel-level contrast loss is defined as follows: Where P i and N i Let i represent the embedding sets of positive and negative samples of pixel i, respectively, and the positive and negative samples and anchor i are not limited to coming from the same image.

6. The image fine-grained segmentation method based on representational consistency contrastive learning according to claim 1, characterized in that, The inter-block similarity loss aligns style feature blocks with image segmentation feature blocks, achieving face refinement segmentation constrained by inter-block similarity. This inter-block similarity loss divides the image into M patches by spatially segmenting it at scale s. Right now Then, the Gram matrix from the image segmentation feature blocks and style feature blocks is calculated. After that, the L2 distance between the segmentation matrix and the style matrix is ​​minimized, as shown below: Where, φ seg ((·)) represents the activation map after feature extraction by the segmentation skeleton network and activation by the ReLU layer, φ vgg ((·)) represents the style activation map after style feature extraction, which is also activated by the ReLU layer. The inter-block Gram matrix is ​​used to capture the negative correlation of channel scale. Based on the inter-block similarity loss of tile style, the L2 distance of the Gram matrix between style feature blocks and segmentation feature blocks is calculated to measure the segmentation consistency of local patterns.

7. The image fine-grained segmentation method based on representational consistency contrastive learning according to claim 1, characterized in that, In step 4, the pixel-level contrastive learning loss and the inter-block similarity loss are combined, and the most basic cross-entropy loss in the segmentation task is added to train the entire segmentation model, which together guides the optimization of the model's parameters.

Images