Semi-supervised semantic segmentation method and device based on MIM and contrastive learning

Abstract

The present application relates to a semi-supervised semantic segmentation method and device based on MIM and contrastive learning. The method comprises: obtaining a training image set; constructing a MIM semi-supervised semantic segmentation network model, which includes a teacher model, a student model, a reconstruction probe, and a feature probe; sending a weakly enhanced image to the teacher model for semantic segmentation to generate a first predicted entropy map; inputting a strongly enhanced image into the student model decoder, extracting strong semantic features and a second pseudo-label through the feature probe; inputting a mask image of the weakly enhanced image into the student model encoder, generating reconstructed image semantic features and a fourth pseudo-label through the reconstruction probe; splicing strong semantic features, reconstructed image semantic features, the second pseudo-label, and the fourth pseudo-label to generate a second predicted entropy map; and obtaining a trained MIM semi-supervised semantic segmentation network model based on the comparative learning results of the first predicted entropy map and the second predicted entropy map. The present invention can improve the image feature extraction effect.

Description

Technical Field

[0001] The present application relates to the technical field of image data processing, and in particular to a semi-supervised semantic segmentation method and device based on MIM and contrastive learning. Background Art

[0002] With the development of image data processing technology, it is necessary to predict the data authenticity of images captured during target tracking. Semi-supervised semantic segmentation technology has emerged. It uses semi-supervised learning to utilize the information of unlabeled data in the image, and simulates the real data distribution of image labeled and unlabeled data to generate new image data or high-quality pseudo labels. Under the assumption of consistency regularization method, the pseudo-label method is used to iteratively enhance the performance of semi-supervised semantic feature extraction.

[0003] However, current semi-supervised semantic segmentation methods focus on extracting unlabeled information from labeled data, while ignoring the information interaction between different regions within the image, resulting in poor accuracy in image data processing and unlabeled data extraction, and in the case of target masking, the captured image information has a large deviation rate in the training results. Summary of the Invention

[0004] Based on this, it is necessary to provide a semi-supervised semantic segmentation method and device based on MIM and contrastive learning that can extract image data information between different slices to address the above technical problems.

[0005] A semi-supervised semantic segmentation method based on MIM and contrastive learning, the method comprising:

[0006] A training image set is obtained, where the training image set includes weakly enhanced images and strongly enhanced images.

[0007] Construct an MIM semi-supervised semantic segmentation network model, which includes: teacher model, student model, reconstruction detection head and feature detection head.

[0008] The weakly enhanced image is sent to the teacher model for semantic segmentation to obtain a first pseudo label of the unlabeled image in the weakly enhanced image, and a first predicted entropy map is generated according to the first pseudo label.

[0009] The strongly enhanced image is input into the student model decoder, and the strong semantic features and the second pseudo label are extracted from the student model decoder through the feature detection head.

[0010] The mask image of the weakly enhanced image is input into the student model encoder, and the mask semantic features and the third pseudo label extracted by the reconstruction detector from the student model encoder are used to generate the reconstructed image semantic features and the fourth pseudo label based on MIM.

[0011] A second predicted entropy map is generated by concatenating the strong semantic features, the reconstructed image semantic features, the second pseudo label, and the fourth pseudo label.

[0012] Reverse training is performed based on the comparative learning results of the first predicted entropy map and the second predicted entropy map to obtain a trained MIM semi-supervised semantic segmentation network model.

[0013] In one embodiment, the method further includes: sending the weakly enhanced image to a teacher model, extracting the first semantic feature of the weakly enhanced image through a first residual module, a first global-local attention mechanism module, a first weighted summation module, and a first feature enhancement module in the teacher model, assigning a first pseudo label to the unlabeled image in the weakly enhanced image, and splicing the first semantic feature and the first pseudo label to generate a first predicted entropy map.

[0014] In one embodiment, the teacher model includes a first residual module, a first global-local attention mechanism module, a first weighted summation module, and a first feature enhancement module.

[0015] The first residual module receives the weakly enhanced image and processes the semantic feature information of the weakly enhanced image layer by layer to obtain a first residual semantic feature.

[0016] The first global-local attention mechanism module includes a first global branch and a first local branch. The first global branch uses a context interaction module to extract the first global semantic features of the weakly enhanced image, and the first local branch uses a parallel convolutional layer to extract the first local semantic features of the weakly enhanced image.

[0017] The first weighted summation module is used to aggregate the first residual semantic feature, the first global semantic feature, and the first local semantic feature to obtain a first fused semantic feature of the weakly enhanced image.

[0018] The first feature enhancement module receives the first fused semantic feature, processes the first fused semantic feature in a dual-channel processing manner, and obtains a first semantic feature of the weakly enhanced image.

[0019] In one embodiment, the student model further includes a second residual module, a second global-local attention mechanism module, a second weighted sum module and a second feature enhancement module.

[0020] The strongly enhanced image is input into the student model decoder, and semantic features are extracted through the second residual module, the second global-local attention mechanism module and the second weighted summation module in the student model decoder to obtain the second semantic feature.

[0021] The feature detection head obtains the second semantic features from the student model decoder, converges according to the pre-built contrast loss function, and generates strong semantic features and second pseudo labels.

[0022] In one embodiment, the method further includes: inputting the mask image of the weakly enhanced image into the student model encoder, performing semantic feature extraction through the second residual module in the student model encoder, and obtaining mask semantic features.

[0023] The reconstruction probe head extracts the mask semantic features and the third pseudo label of the mask image of the weakly enhanced image from the student model encoding, and splices the mask semantic features and the third pseudo label based on MIM to obtain the reconstructed image.

[0024] The reconstructed image is converged according to a pre-constructed reconstruction loss function to generate a mask prediction image, and the mask prediction image is converged through a pre-constructed reconstruction semantic consistency loss function to obtain the semantic features of the reconstructed image and the fourth pseudo label.

[0025] In one embodiment, the method further includes: using a pre-constructed label loss function, performing difference convergence on the labels of the training image set and the second pseudo-label and the fourth pseudo-label to obtain optimized second pseudo-label and fourth pseudo-label, splicing the strong semantic features, the reconstructed image semantic features, the optimized second pseudo-label and the optimized fourth pseudo-label to generate a second predicted entropy map.

[0026] In one embodiment, the label loss function is defined as:

[0027]

[0028] Among them, λ i is the label weight of the i-th pixel, y i is the label or pseudo-label of the i-th pixel, is the prediction of the i-th pixel, and n is the number of pixels involved in the calculation.

[0029] In one of the embodiments, it also includes: constructing an entropy loss function based on the difference between the first predicted entropy map and the second predicted entropy map, converging the comparative learning results of the first predicted entropy map and the second predicted entropy map, and inputting the converged comparative learning results as a training image set into the MIM semi-supervised semantic segmentation network model to obtain a trained MIM semi-supervised semantic segmentation network model.

[0030] In one embodiment, the reconstruction loss function is defined as:

[0031]

[0032] Among them, n m is the number of masked pixels, is the reconstructed image of the student model, x weak is the original image with weak enhancement, is the indicator function, Indicates a shielded area.

[0033] A semi-supervised semantic segmentation device based on MIM and contrastive learning, the device comprising:

[0034] The sample acquisition module is used to acquire a training image set, which includes weakly enhanced images and strongly enhanced images.

[0035] The model building module is used to build the MIM semi-supervised semantic segmentation network model. The MIM semi-supervised semantic segmentation network model includes: a teacher model, a student model, a reconstruction detection head, and a feature detection head.

[0036] The first predicted entropy map generation module is used to send the weakly enhanced image to the teacher model for semantic segmentation, obtain a first pseudo label of the unlabeled image in the weakly enhanced image, and generate a first predicted entropy map according to the first pseudo label.

[0037] The strong feature extraction module is used to input the strongly enhanced image into the student model decoder, and extract strong semantic features and the second pseudo label from the student model decoder through the feature detection head.

[0038] The image reconstruction module is used to input the mask image of the weakly enhanced image into the student model encoder, and generate the reconstructed image semantic features and the fourth pseudo label based on MIM by reconstructing the mask semantic features and the third pseudo label extracted by the detection head from the student model encoder.

[0039] The second prediction entropy map generation module is used to generate a second prediction entropy map by splicing strong semantic features, reconstructing image semantic features, second pseudo labels and fourth pseudo labels.

[0040] The model training module is used to perform reverse training based on the comparative learning results of the first predicted entropy map and the second predicted entropy map to obtain a trained MIM semi-supervised semantic segmentation network model.

[0041] The above-mentioned semi-supervised semantic segmentation method and device based on MIM and contrastive learning, by constructing a MIM semi-supervised semantic segmentation network model, first uses the teacher model to process the data of the unlabeled image in the weakly enhanced image, and then extracts the first pseudo label. The first predicted entropy map is generated based on the first pseudo label, and the mask image of the weakly enhanced image is used as the processing data of the student model, which promotes the information interaction between the slices of the training image set and ensures the stability of the data during the subsequent model training process. Secondly, in order to focus on extracting the unlabeled semantic features in the training image set, the strongly enhanced image is used as another data source received by the student model. By adding a feature detector head and a reconstruction detector head, the image data in the student model is processed by region, slice, and label, thereby obtaining strong semantic features, reconstructed image semantic features, second pseudo labels, and fourth pseudo labels. By splicing and fusion, a second predicted entropy map is generated. Finally, the comparative learning results of the first predicted entropy map and the second predicted entropy map are used as the reverse training image set and re-input into the MIM semi-supervised semantic segmentation network model, continuously improving the accuracy of the training results, thereby obtaining a trained MIM semi-supervised semantic segmentation network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 1 is a flow chart of a semi-supervised semantic segmentation method based on MIM and contrastive learning in one embodiment;

[0043] Figure 2 Schematic diagram of a semi-supervised semantic segmentation network model in one embodiment;

[0044] Figure 3 is a structural schematic diagram of a feature detection head in one embodiment;

[0045] Figure 4 is a schematic structural diagram of a reconstruction detection head in one embodiment;

[0046] Figure 5 A schematic diagram of the UNetFormer network architecture in one embodiment;

[0047] Figure 6 A comparison diagram of some results of training the LoveDA dataset in one embodiment, including: (a) remote sensing image; (b) road label; (c) ReCo; (d) U2PL; (e) CAC; (f) CCT; (g) CPS; (h) PSMT; (i) SemiCMNet;

[0048] Figure 7 A comparison diagram of some results of training the DGLCC dataset in one embodiment, including: (a) remote sensing image; (b) road label; (c) ReCo; (d) U2PL; (e) CAC; (f) CCT; (g) CPS; (h) PSMT; (i) SemiCMNet;

[0049] Figure 8 A comparison diagram of some results of the Potsdam dataset in one embodiment, including: (a) remote sensing image; (b) road label; (c) ReCo; (d) U2PL; (e) CAC; (f) CCT; (g) CPS; (h) PSMT; (i) SemiCMNet;

[0050] Figure 9 Schematic diagram of the structure of a semi-supervised semantic segmentation device based on MIM and contrastive learning in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] It is worth noting that the descriptions such as “first” and “second” involved in the present invention only represent naming methods, without distinguishing between size and order, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features.

[0053] In one embodiment, Figure 1 As shown in FIG, a semi-supervised semantic segmentation method based on MIM and contrastive learning is provided, which includes the following steps:

[0054] Step 102: Obtain a training image set.

[0055] The training image set includes: weakly enhanced images and strongly enhanced images.

[0056] The training image set can be divided into labeled images x l and unlabeled image x u , where the dataset of labeled images is represented as It contains M labeled images and its corresponding label In addition, the dataset of unlabeled images is represented as It contains N unlabeled unlabeled images Specifically, the weakly enhanced image is x weak , can be divided into labeled weakly enhanced images and unlabeled weakly enhanced images, and the strongly enhanced image is x strong , can be divided into labeled strongly enhanced images and unlabeled strongly enhanced images.

[0057] Step 104: construct a MIM semi-supervised semantic segmentation network model.

[0058] By constructing the MIM semi-supervised semantic segmentation network model, better semantic segmentation can be performed on unlabeled images in the training image set, which increases the number of extracted semantic features and their corresponding labels, and is conducive to the subsequent restoration of image data.

[0059] The semi-supervised semantic segmentation network model includes: teacher model, student model, reconstruction detection head and feature detection head.

[0060] Both the teacher model and the student model use the UNetFormer model, where the encoder in the UNetFormer model is a ResNet50 encoder. In addition, the parameters of the teacher model are updated on the student model using EMA (Exponential Moving Average). The update of the teacher model weights can be expressed as:

[0061] θ t,i =(1-α θ )θ t,i-1 +α θ θ s

[0062] Among them, θ t，i is the i-th generation weight of the teacher model, θ s is the current generation weight of the student model, α θ is a hyperparameter of the smoothing coefficient.

[0063] Step 106: Send the weakly enhanced image to the teacher model for semantic segmentation to obtain a first pseudo label of the unlabeled image in the weakly enhanced image, and generate a first predicted entropy map according to the first pseudo label.

[0064] It is worth noting that the weakly enhanced image is sent to the teacher model for semantic segmentation to obtain the semantic features of the weakly enhanced image and its corresponding label. The UNetFormer model is used to assign a first pseudo label to the unlabeled image in the weakly enhanced image. The first predicted entropy map is generated by splicing the first pseudo label with the label of the weakly enhanced image.

[0065] Step 108: input the strongly enhanced image into the student model decoder, and extract strong semantic features and a second pseudo label from the student model decoder through a feature detection head.

[0066] The first pseudo label is self-updated based on the second pseudo label of the student model, expressed as:

[0067]

[0068] in, is the pseudo label of the i-th pixel of the unlabeled image xu, is x uThe teacher model predicts that yes The predicted label, δ p It is worth noting that the feature detection head extracts the semantic features of the strongly enhanced image and the labels of the unlabeled image pixels, further capturing the semantic information between slices and improving the accuracy of semantic feature extraction.

[0069] In step 110 , the mask image of the weakly enhanced image is input into the student model encoder, and the mask semantic features and the third pseudo label extracted from the student model encoder by the reconstruction probe are used to generate the reconstructed image semantic features and the fourth pseudo label based on the MIM.

[0070] The mask m is randomly generated according to the mask rate, and its shape is the same as the training image set x. It is worth noting that the mask image x of the weakly enhanced image is weak·m Input into the student model, by reconstructing the semantic consistency loss function l rsc , the predicted labels of the mask image of the weakly enhanced image are compared with the labels of the training image set, based on the semantic consistency loss function l rsc The connection between different patches is enhanced at the semantic level, thereby supplementing the missing semantic feature information of the masked area of ​​the training image set, and reconstructing the image using the residual module of the reconstruction probe and the student model. And extract the mask semantic features in the reconstructed image, assign a third pseudo label to the unlabeled image in the reconstructed image, and then calculate the reconstructed image through the mean square loss With the weakly enhanced image x weak The difference between the two, the mask m pixels only contains two different values, where 1 represents the unmasked pixel and 0 represents the masked pixel.

[0071] Specifically, the reconstruction loss function l r Defined as:

[0072]

[0073] Among them, n m is the number of masked pixels, is the reconstructed mask image of the student model, x weak is a weakly enhanced image, is the indicator function, Indicates a shielded area.

[0074] Reconstruct the semantic consistency loss function lrsc, expressed as:

[0075]

[0076] Where y is the weakly enhanced image x processed by the teacher model weak The first pseudo label for semantic segmentation prediction is Is the mask image x of the weakly enhanced image weak·m Predict the third pseudo label for semantic segmentation.

[0077] Step 112 : Generate a second predicted entropy map by concatenating the strong semantic features, the reconstructed image semantic features, the second pseudo labels, and the fourth pseudo labels.

[0078] Step 114 , performing reverse training based on the comparative learning results of the first predicted entropy map and the second predicted entropy map to obtain a trained MIM semi-supervised semantic segmentation network model.

[0079] It is worth noting that the contrast loss function l is introduced c , defined as:

[0080]

[0081] Among them, F q is the query feature set, is the negative feature repository for class c, is the positive keyword of class c, and τ is the temperature parameter that controls the smoothness of the feature distribution. The contrastive learning results of the first and second predicted entropy maps are converged using a contrastive loss function. The converged contrastive learning results are used as the input data of the MIM semi-supervised semantic segmentation network model. Through continuous iterative training, the final trained MIM semi-supervised semantic segmentation network model, namely the SemiCMNet model, is obtained.

[0082] Specifically, the average feature of class c (the class of the query feature) in the batch is used as the primary key ,In order to maintain the balance between the number of negative keys and memory resources, a dynamic queue is constructed, where the negative key queue Represents all data in the queue except class c (i.e., query class). Since the sample strategy is important for contrastive learning, the query F q Sampling is performed and the corresponding pixel confidence is found to be lower than the query threshold δ q ,Therefore, the negative key number of the current batch is determined based on the average feature similarity between the batch and the queue.

[0083] In the above-mentioned semi-supervised semantic segmentation method based on MIM and contrastive learning, the above-mentioned semi-supervised semantic segmentation method and device based on MIM and contrastive learning, by constructing a MIM semi-supervised semantic segmentation network model, first uses the teacher model to process the data of the unlabeled image in the weakly enhanced image, and then extracts the first pseudo label. The first predicted entropy map is generated based on the first pseudo label, and the mask image of the weakly enhanced image is used as the processing data of the student model, which promotes the information interaction between the slices of the training image set and ensures the stability of the data during the subsequent model training process. Secondly, in order to focus on extracting the unlabeled semantic features in the training image set, the strongly enhanced image is used as another data source received by the student model. By adding a feature detector head and a reconstruction detector head, the image data in the student model is processed by region, slice, and label, thereby obtaining strong semantic features, reconstructed image semantic features, second pseudo labels, and fourth pseudo labels. By splicing and fusion, a second predicted entropy map is generated. Finally, the comparative learning results of the first predicted entropy map and the second predicted entropy map are used as the reverse training image set and re-input into the MIM semi-supervised semantic segmentation network model, continuously improving the accuracy of the training results, thereby obtaining a trained MIM semi-supervised semantic segmentation network model.

[0084] In one embodiment, the weakly enhanced image is sent to a teacher model, and the first semantic feature of the weakly enhanced image is extracted through the first residual module, the first global-local attention mechanism module, the first weighted sum module, and the first feature enhancement module in the teacher model, and the unlabeled image in the weakly enhanced image is assigned a first pseudo label, and the first semantic feature and the first pseudo label are spliced ​​together to generate a first predicted entropy map.

[0085] It is worth noting that if Figure 5 As shown in Figure 2, the teacher model uses the UNetFormer model with ResNet50 as the encoder to obtain better feature extraction capabilities.

[0086] In one embodiment, the teacher model includes: a first residual module, a first global-local attention mechanism module, a first weighted summation module and a first feature enhancement module. The first residual module receives a weakly enhanced image and processes the semantic feature information of the weakly enhanced image layer by layer to obtain a first residual semantic feature. The first global-local attention mechanism module includes a first global branch and a first local branch. The first global branch uses a context interaction module to extract the first global semantic feature of the weakly enhanced image, and the first local branch uses a parallel convolutional layer to extract the first local semantic feature of the weakly enhanced image. The first weighted summation module is used to aggregate the first residual semantic feature, the first global semantic feature and the first local semantic feature to obtain a first fused semantic feature of the weakly enhanced image. The first feature enhancement module receives the first fused semantic feature, processes the first fused semantic feature using a dual-channel processing method, and obtains the first semantic feature of the weakly enhanced image.

[0087] In one embodiment, the student model includes a second residual module, a second global-local attention mechanism module, a second weighted summation module, and a second feature enhancement module. The strongly enhanced image is input into the student model decoder, and semantic features are extracted by the second residual module, the second global-local attention mechanism module, and the second weighted summation module in the student model decoder to obtain a second semantic feature. The feature detector obtains the second semantic feature from the student model decoder, converges according to a pre-constructed contrast loss function, and generates a strong semantic feature and a second pseudo label.

[0088] It is worth noting that if Figure 3 、 Figure 5 As shown in the figure, by dividing the UNetFormer model of the student model into modules to extract features and using the feature detection head to perform secondary analysis and processing on the semantic features in the decoder, the unlabeled pixels in the strong image pixels can be further captured, thereby improving the overall extraction accuracy of the semantic features.

[0089] In one embodiment, the mask image of the weakly enhanced image is input into the student model encoder, and semantic features are extracted by the second residual module in the student model encoder to obtain mask semantic features. The reconstruction probe extracts the mask semantic features and the third pseudo label of the mask image of the weakly enhanced image from the student model encoding, and splices the mask semantic features and the third pseudo label based on MIM to obtain a reconstructed image. The reconstructed image is converged according to a pre-constructed reconstruction loss function to generate a mask prediction image, and the mask prediction image is converged by a pre-constructed reconstruction semantic consistency loss function to obtain the semantic features of the reconstructed image and the fourth pseudo label.

[0090] It is worth noting that if Figure 4 As shown, MIM (Masked Image Modeling) can enhance the semantic information interaction between slices of the reconstructed image, so that the mask semantic features and the third pseudo-label extracted by the reconstruction detector are more accurate, and the predicted label of the high-confidence pixel corresponds to the third pseudo-label.

[0091] In one embodiment, a pre-constructed label loss function is used to perform difference convergence between the labels of the training image set and the second pseudo-label and the fourth pseudo-label to obtain optimized second pseudo-label and fourth pseudo-label, and the strong semantic features, reconstructed image semantic features, optimized second pseudo-label and optimized fourth pseudo-label are spliced ​​to generate a second predicted entropy map.

[0092] In one embodiment, the label loss function is defined as:

[0093]

[0094] Among them, λ i is the label weight of the i-th pixel, y i is the label or pseudo-label of the i-th pixel, is the prediction of the i-th pixel, and n is the number of pixels involved in the calculation.

[0095] It is worth noting that the label loss function l lab Using the label y l and pseudo-label y u information, and l lab The weights in are dynamically adjusted based on the recall of the corresponding labels using EMA (Exponential Moving Average). For labeled images, all pixels of the training image and the label are involved in the calculation, so n is equal to H × W. For unlabeled images, only high-confidence pseudo-label pixels and the corresponding training image pixels are involved in the calculation, so n is equal to the number of pseudo-label pixels. Specifically, EMA updates the weight λ to maintain a balance between stability and dynamism. Considering that pseudo-labels are not reliable enough compared to true labels, the update basis is the recall of the labeled data, rather than the total data in the batch. Therefore, the weight λ is updated in the following way:

[0096] λ c,i =(1-α λ )λ c,i-1 +α λ (1-Rec l )

[0097] Among them, λ c，i is the weight of class c in the i-th weight update iteration, α λ is the defined parameter, Rec l is the recall of labeled data in the batch.

[0098] In one of the embodiments, an entropy loss function is constructed based on the difference between the first predicted entropy map and the second predicted entropy map, and the comparative learning results of the first predicted entropy map and the second predicted entropy map are converged. The converged comparative learning results are input into the MIM semi-supervised semantic segmentation network model as a training image set to obtain a trained MIM semi-supervised semantic segmentation network model.

[0099] It is worth noting that the entropy loss function le is designed based on the difference between the predicted entropy maps of the student model and the teacher model, and is defined as:

[0100] l e (e s ,e t )=(e t -e s )2

[0101] Among them, e s and e t are the entropy mappings of the student model and the teacher model, respectively. Specifically, the entropy mapping value reflects the uncertainty range of the prediction. e Tends to reduce based on teacher entropy graph e t The student model prediction uncertainty and the student entropy graph e. The entropy value e i (Student Entropy Graph e s or teacher entropy graph e t ) is calculated by the following formula:

[0102]

[0103] Among them, y j Represents the prediction vector The probability of category j is obtained by comparing the first predicted entropy map with the second predicted entropy map to improve the accuracy of the learning results.

[0104] In one embodiment, the reconstruction loss function is defined as:

[0105]

[0106] Among them, n m is the number of masked pixels, is the reconstructed image of the student model, x weak is the original image with weak enhancement, is the indicator function, Indicates a shielded area.

[0107] In one embodiment, three public remote sensing image road datasets are used for experimental comparison, namely: LoveDA dataset, DeepGlobe Land Cover Classification dataset and Potsdam dataset.

[0108] Table 1 Number of images in each dataset

[0109]

[0110] In order to demonstrate the effectiveness of the method proposed in the present invention, several advanced semi-supervised semantic segmentation methods are used to perform quantitative and qualitative comparisons on the performance of the invented model.

[0111] In comparative experiments, we used three image quality evaluation metrics to compare SemiCMNet (i.e., the MIM semi-supervised semantic segmentation network model) with other models. Tables 2, 3, and 4 list the quantitative results of using each method to train the LoveDA, DGLCC, and Potsdam datasets, respectively. Figure 6 、 Figure 7 、 Figure 8 The qualitative results of each method on the dataset are listed separately. The highest score is in bold. In general, the proposed SemiCMNet performs the best, with the highest scores in the evaluation indicators and the best predicted image continuity.

[0112] Table 2 Experimental results of LoveDA dataset

[0113]

[0114] Table 3DGLCC dataset experimental results

[0115]

[0116] Table 4 Experimental results of Potsdam dataset

[0117]

[0118] It should be understood that although Figure 1-5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1-5 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0119] In one embodiment, Figure 9 As shown, a semi-supervised semantic segmentation device based on MIM and contrastive learning is provided, including: a sample acquisition module, a model construction module, a first prediction entropy map generation module, a strong feature extraction module, an image reconstruction module, a second prediction entropy map generation module and a model training module, wherein:

[0120] The sample acquisition module is used to acquire a training image set, which includes weakly enhanced images and strongly enhanced images.

[0121] The model building module is used to build the MIM semi-supervised semantic segmentation network model. The MIM semi-supervised semantic segmentation network model includes: a teacher model, a student model, a reconstruction detection head, and a feature detection head.

[0122] The first predicted entropy map generation module is used to send the weakly enhanced image to the teacher model for semantic segmentation, obtain a first pseudo label of the unlabeled image in the weakly enhanced image, and generate a first predicted entropy map according to the first pseudo label.

[0123] The strong feature extraction module is used to input the strongly enhanced image into the student model decoder, and extract strong semantic features and the second pseudo label from the student model decoder through the feature detection head.

[0124] The image reconstruction module is used to input the mask image of the weakly enhanced image into the student model encoder, and generate the reconstructed image semantic features and the fourth pseudo label based on MIM by reconstructing the mask semantic features and the third pseudo label extracted by the detection head from the student model encoder.

[0125] The second prediction entropy map generation module is used to generate a second prediction entropy map by splicing strong semantic features, reconstructing image semantic features, second pseudo labels and fourth pseudo labels.

[0126] The model training module is used to perform reverse training based on the comparative learning results of the first predicted entropy map and the second predicted entropy map to obtain a trained MIM semi-supervised semantic segmentation network model.

[0127] For the specific definition of the semi-supervised semantic segmentation device based on MIM and contrastive learning, please refer to the definition of the semi-supervised semantic segmentation method based on MIM and contrastive learning above, which will not be repeated here. The various modules in the above-mentioned semi-supervised semantic segmentation device based on MIM and contrastive learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0128] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0129] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A semi-supervised semantic segmentation method based on MIM and contrastive learning, characterized in that: The method comprises: Acquire a training image set, wherein the training image set includes: a weakly enhanced image and a strongly enhanced image; Constructing an MIM semi-supervised semantic segmentation network model, wherein the MIM semi-supervised semantic segmentation network model includes: a teacher model, a student model, a reconstruction detection head, and a feature detection head; Sending the weakly enhanced image to the teacher model for semantic segmentation to obtain a first pseudo label of an unlabeled image in the weakly enhanced image, and generating a first predicted entropy map according to the first pseudo label; Inputting the strongly enhanced image into the student model decoder, and extracting strong semantic features and a second pseudo label from the student model decoder by the feature detection head; Inputting the mask image of the weakly enhanced image into the student model encoder, and generating a reconstructed image semantic feature and a fourth pseudo label based on MIM using the mask semantic feature and the third pseudo label extracted from the student model encoder by the reconstruction probe; Generate a second predicted entropy map by concatenating the strong semantic features, the reconstructed image semantic features, the second pseudo label, and the fourth pseudo label; Reverse training is performed based on the comparative learning results of the first predicted entropy map and the second predicted entropy map to obtain a trained MIM semi-supervised semantic segmentation network model.

2. The method according to claim 1, characterized in that Sending the weakly enhanced image to the teacher model for semantic segmentation to obtain a first pseudo label of an unlabeled image in the weakly enhanced image, and generating a first predicted entropy map according to the first pseudo label, including: The weakly enhanced image is sent to the teacher model, and the first semantic feature of the weakly enhanced image is extracted through the first residual module, the first global-local attention mechanism module, the first weighted sum module and the first feature enhancement module in the teacher model, and a first pseudo label is assigned to the unlabeled image in the weakly enhanced image. The first semantic feature and the first pseudo label are spliced ​​together to generate a first predicted entropy map.

3. The method according to claim 2, characterized in that The teacher model includes: a first residual module, a first global-local attention mechanism module, a first weighted summation module and a first feature enhancement module; The first residual module receives the weakly enhanced image and processes the semantic feature information of the weakly enhanced image layer by layer to obtain a first residual semantic feature; The first global-local attention mechanism module includes a first global branch and a first local branch, the first global branch uses a context interaction module to extract a first global semantic feature of the weakly enhanced image, and the first local branch uses a parallel convolutional layer to extract a first local semantic feature of the weakly enhanced image; The first weighted summation module is used to aggregate the first residual semantic feature, the first global semantic feature, and the first local semantic feature to obtain a first fused semantic feature of the weakly enhanced image; The first feature enhancement module receives the first fused semantic feature, processes the first fused semantic feature in a dual-channel processing manner, and obtains a first semantic feature of the weakly enhanced image.

4. The method according to claim 3, characterized in that Inputting the strongly enhanced image into the student model decoder, and extracting strong semantic features and a second pseudo label from the student model decoder by the feature detection head, comprising: The student model includes a second residual module, a second global-local attention mechanism module, a second weighted sum module and a second feature enhancement module; Inputting the strongly enhanced image into the student model decoder, performing semantic feature extraction through the second residual module, the second global-local attention mechanism module, and the second weighted sum module in the student model decoder to obtain a second semantic feature; The feature detection head obtains the second semantic feature from the student model decoder, converges according to a pre-constructed contrast loss function, and generates a strong semantic feature and a second pseudo label.

5. The method according to claim 4, characterized in that Inputting the mask image of the weakly enhanced image into the student model encoder, and generating a reconstructed image semantic feature and a fourth pseudo label based on MIM using the mask semantic feature and the third pseudo label extracted from the student model encoder by the reconstruction probe, including: Inputting the mask image of the weakly enhanced image into the student model encoder, performing semantic feature extraction through the second residual module in the student model encoder to obtain mask semantic features; The reconstruction probe extracts the mask semantic features and the third pseudo label of the mask image of the weakly enhanced image from the student model encoding, and splices the mask semantic features and the third pseudo label based on MIM to obtain a reconstructed image; The reconstructed image is converged according to a pre-constructed reconstruction loss function to generate a mask prediction image, and the mask prediction image is converged through a pre-constructed reconstruction semantic consistency loss function to obtain the reconstructed image semantic features and the fourth pseudo label.

6. The method according to claim 5, characterized in that Generating a second predicted entropy map by concatenating the strong semantic feature, the reconstructed image semantic feature, the second pseudo label, and the fourth pseudo label, including: Through a pre-constructed label loss function, the labels of the training image set are respectively subjected to difference convergence with the second pseudo-label and the fourth pseudo-label to obtain optimized second pseudo-label and fourth pseudo-label, and the strong semantic features, the reconstructed image semantic features, the optimized second pseudo-label and the optimized fourth pseudo-label are spliced ​​to generate a second predicted entropy map.

7. The method according to claim 6, characterized in that The label loss function is defined as: in, It is The label weight of each pixel, It is labels or pseudo-labels for pixels, It is The prediction of pixels, is the number of pixels involved in the calculation.

8. The method according to claim 7, characterized in that Reverse training is performed based on the comparative learning results of the first predicted entropy map and the second predicted entropy map to obtain a trained MIM semi-supervised semantic segmentation network model, including: An entropy loss function is constructed based on the difference between the first predicted entropy map and the second predicted entropy map, and the comparative learning results of the first predicted entropy map and the second predicted entropy map are converged. The converged comparative learning results are input into the MIM semi-supervised semantic segmentation network model as a training image set to obtain a trained MIM semi-supervised semantic segmentation network model.

9. The method according to claim 8, characterized in that The reconstruction loss function is defined as: in, is the number of masked pixels, is the reconstructed image of the student model, is the original image with weak enhancement, is the indicator function, Indicates a shielded area.

10. A semi-supervised semantic segmentation device based on MIM and contrastive learning, characterized in that: The device comprises: A sample acquisition module is used to acquire a training image set, wherein the training image set includes: a weakly enhanced image and a strongly enhanced image; A model construction module is used to construct an MIM semi-supervised semantic segmentation network model, wherein the MIM semi-supervised semantic segmentation network model includes: a teacher model, a student model, a reconstruction detection head, and a feature detection head; A first predicted entropy map generating module is configured to send the weakly enhanced image to the teacher model for semantic segmentation, obtain a first pseudo label of an unlabeled image in the weakly enhanced image, and generate a first predicted entropy map according to the first pseudo label; a strong feature extraction module, configured to input the strongly enhanced image into the student model decoder, and extract strong semantic features and a second pseudo label from the student model decoder via the feature detection head; an image reconstruction module, configured to input the mask image of the weakly enhanced image into the student model encoder, and generate a reconstructed image semantic feature and a fourth pseudo label based on MIM using the mask semantic feature and the third pseudo label extracted from the student model encoder by the reconstruction probe; A second predicted entropy map generating module, configured to generate a second predicted entropy map by concatenating the strong semantic features, the reconstructed image semantic features, the second pseudo labels, and the fourth pseudo labels; The model training module is used to perform reverse training based on the comparative learning results of the first predicted entropy map and the second predicted entropy map to obtain a trained MIM semi-supervised semantic segmentation network model.

Images